Page tree
Skip to end of metadata
Go to start of metadata

This page provides information about advanced configuration scenarios and troubleshooting.

Node Activation Policy

The node order is respected when a node is created, but it is not respected on update.

To ensure that the node order is respected on update you can add the PUB_POLICY property to the parent of the nodes that need a specific order.  

The PUB_POLICY property has two valid values that result in the following behaviors:

Value

Behavior

respect_order

Nodes are always deleted on update. (warning) You need to use the Publish incl.subpages/nodes action to activate or the subnodes will not be created.

remove_orphan_nodes

Nodes whose parent was not found in the correct place are ignored. (Not implemented yet.)

Continuous deployment strategy

How to synchronize new public instances

Public instance backups should be done when the SyncState message id is identical for all public nodes. You can then choose one public node and to do a dump. Once the dump is done, create a new historical queue and empty the old queue by consuming all messages. The instance will then be ready to receive new activation messages which post-date the last dump. 

Whenever you need to create a new instance, always use the latest dump. The new instance will be registered to the new historical queue. This will mean that the message id is in SyncState with the latest activation message. New activation messages will be sent to the new historical queue before they register in the old historical queue. The new historical queue will start getting all messages after the last message in old historical queue.

Adding new instances

This series of diagrams shows the best setup for scaling up when adding new public instances to the load balancer.

  1. The basic idea is to always keep a spare queue (which does not have a consumer) connected to the exchange. 
  2. The spare queue stores all activation messages until a new consumer connects. The newly created instance is created with the same initial data as Public 1.
  3. Public 2 is created with the same data set as Public 1. Before Public 2 starts consuming a new queue (without a consumer) is created.
  4. Once the remaining messages are consumed a backup is created. This serves as the new state for new instances. This is also the time to add the new public instance to the load balancer.

    When to do a backup?

    Each message has a version number or tag in its header. When a new public is added it can tell the author instance or the exchange about its presence by sending its identity before consuming starts. This enables the exchange or author to create the new queue for the public instance.

    From this point, all activation messages contain a new version or tag and the consumer starts consuming messages in the queue. When the consumer comes across a message with the newest version number, it does the backup.

  5. Activation continues and the spare queue (queue_pub_3) is filled with new messages that represent the difference between Public 2 and newest content.

Standard backups

Regular backups are needed to prevent the queue becoming too big. Whenever you do a backup on the instances the spare queue can be emptied.

Adding a queue after backup

This example shows you how to add a new queue to the example testfanoutAck exchange:

  1. Add the queue:

    curl -i -u guest:guest -H "content-type:application/json" \
      -XPUT -d'{"auto_delete":false,"durable":true,"arguments":{},"node":"rabbit@localhost"}' \
      http://localhost:55672/api/queues/%2f/fan2
  2. Declare it to be bound to correct exchange:

    curl -i -u guest:guest -H "content-type:application/json" \
       -d'{"routing_key":"","arguments":[]}' \
      http://localhost:55672/api/bindings/%2f/fan2/testfanoutAck

See RabbitMQ Management HTTP API for more.

ACK queue blocked by unacked node

When a consumer encounters an issue because of an activation exception, the consumer stops consuming. The message is sent back to the queue with a non-ack, and an exception is sent over the ACK exchange. 

To illustrate this we deleted a node on the public instance and then tried to deactivate it on the author instance. 

The deletion and attempted deactivation resulted in the following error:

ERROR lia.rabbitmq.activation.jobs.ActivationConsumerJob: /bla

The exception is visible in the Public Monitoring app. You can see from the PathNotFoundException that the consumer did not find the path that was deleted directly of the public instance.

In the RabbitMQ console the:

  • Message is not consumed in the queue.
  • Message is marked unacked.
  • Consumer is gone,

When more nodes are published into the queue they become blocked behind the unacked message.

There are two ways to solve this:

  1. Recreate the node on the public instance and restart the consumer.
    • (plus) The sequence number count remains correct. 
    • (minus) You have to connect to the public instance and do "manual manipulation".
  2. Remove the faulty activation message from the queue and restart the consumer.
    • (plus) This can be done remotely and there is no need to connect "manually" to the public instance. 
    • (minus) Because there is one "missing" message the sequence number is lost.

Here's a step-by-step solution using option 2 (because it demonstrates how to delete an unacked message):

  1. Close the channel in the RabbitMQ console. This acks all unacked messages and is needed because messages can no longer be consumed.
     
  2. Force close the connection
     
  3. All messages are now acked in the queue.
     
  4. Consume only the "first" message.

  5. Restart the consumer on the public instance.

    curl http://localhost:8081/.rest/rbmqClients/restartAll
  6. The instance is getting activated again.
     
  • No labels