Magnolia 5.5 reached end of life on November 15, 2019. This branch is no longer supported, see End-of-life policy.
This page provides information about advanced configuration scenarios and troubleshooting.
Public instance backups should be done when the SyncState message id is identical for all public nodes. You can then choose one public node and to do a dump. Once the dump is done, create a new historical queue and empty the old queue by consuming all messages. The instance will then be ready to receive new activation messages which post-date the last dump.
Whenever you need to create a new instance, always use the latest dump. The new instance will be registered to the new historical queue. This will mean that the message id is in SyncState
with the latest activation message. New activation messages will be sent to the new historical queue before they register in the old historical queue. The new historical queue will start getting all messages after the last message in old historical queue.
This series of diagrams shows the best setup for scaling up when adding new public instances to the load balancer.
Once the remaining messages are consumed a backup is created. This serves as the new state for new instances. This is also the time to add the new public instance to the load balancer.
When to do a backup?
Each message has a version number or tag in its header. When a new public is added it can tell the author instance or the exchange about its presence by sending its identity before consuming starts. This enables the exchange or author to create the new queue for the public instance.
From this point, all activation messages contain a new version or tag and the consumer starts consuming messages in the queue. When the consumer comes across a message with the newest version number, it does the backup.
Standard backups
Regular backups are needed to prevent the queue becoming too big. Whenever you do a backup on the instances the spare queue can be emptied.
This example shows you how to add a new queue to the example testfanoutAck
exchange:
Add the queue:
curl -i -u guest:guest -H "content-type:application/json" \ -XPUT -d'{"auto_delete":false,"durable":true,"arguments":{},"node":"rabbit@localhost"}' \ http://localhost:55672/api/queues/%2f/fan2
Declare it to be bound to correct exchange:
curl -i -u guest:guest -H "content-type:application/json" \ -d'{"routing_key":"","arguments":[]}' \ http://localhost:55672/api/bindings/%2f/fan2/testfanoutAck
See RabbitMQ Management HTTP API for more.
When a consumer encounters an issue because of an activation exception, the consumer stops consuming. The message is sent back to the queue with a non-ack, and an exception is sent over the ACK exchange.
To illustrate this we deleted a node on the public instance and then tried to deactivate it on the author instance.
The deletion and attempted deactivation resulted in the following error:
ERROR lia.rabbitmq.activation.jobs.ActivationConsumerJob: /bla
The exception is visible in the Public Monitoring app. You can see from the PathNotFoundException
that the consumer did not find the path that was deleted directly of the public instance.
In the RabbitMQ console the:
When more nodes are published into the queue they become blocked behind the unacked message.
There are two ways to solve this:
Here's a step-by-step solution using option 2 (because it demonstrates how to delete an unacked message):
Restart the consumer on the public instance.
curl http://localhost:8081/.rest/rbmqClients/restartAll