In this page we explain how to configure RabbitMQ with Magnolia to publish content and synchronize Magnolia instances.

In the scenario below:

  • A simple fan out pattern is used from one Magnolia author instance to N public instances.
  • Acknowledgements are delegated to RabbitMQ; no notification of activation success or failure is sent back to the author instance.
  • The AMQP 0-9-1 (Advanced Message Queuing Protocol) messaging protocol is used.



Configuring the RabbitMQ connector

RabbitMQ exchanges and consumers are configured in the connector module. The exchange is configured on the author or producing instance and the consumer on the public or receiving instance.

Configuring the RabbitMQ client

The magnolia-rabbitmq-connector module ships with an example client configuration in /modules/rabbitmq-connector/rabbitmq-client/sampleClient.

Node name

Value

 
rabbitmq-connector


 
rabbitmq-client


 
sampleClient


 
exchangeConfig


 
consumerDefinitions


 
clientName

sample-client 

 
enabled

false 

 
hostName

localhost 

 
password

guest 

 
portNumber

5672 

 
userName

guest 

 
virtualHost

Properties:

rabbitmq-client

required

RabbitMQ client folder.

<client name>

required

Client configuration node. One for each client.

exchangeConfig

required

See Exchange configuration.

consumerDefinitions

required

See Consumer definition.

clientName

required

Name of the client.

Note that the clientName can be different from the <client name> node name.

The clientName specified here must match the clientName property in the activate and deactivate commands for RabbitMQ, for example:

/modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@clientName

If you change the client name in one property, make sure you change it in the other one too.

enabled

required

Enables and disables the client. Toggling this property restarts the client.

hostName

required

Address of the broker.

password

required

Password of the user account to connect with.

portNumber

required

Port number to connect the AMQP service.

username

required

Username of the user account to connect with.

virtualHost

required

Virtualhost to connect to.

Exchange configuration

The exchange is the kind of router that the queues connect to. The producer instance pushes messages to the exchange and the exchange then decides what to do with them.

You must confugre the exchange configuration on the author or producing instance.

The example exchange configuration is in /modules/rabbitmq-connector/rabbitmq-client/sampleClient/exchangeConfig.

Node name

Value

 
sampleClient


 
exchangeConfig


 
queueConfigList


 
queue


 
autodelete

false 

 
exclusive

false 

 
name

fan1

 
routingKey

 -

 
name

testfanoutAck 

 
type

fanout

Properties:

exchangeConfig

required

Exchange configuration node. See

$webResourceManager.requireResource("info.magnolia.sys.confluence.artifact-info-plugin:javadoc-resource-macro-resources") RabbitMQExchangeConfig
.

queueConfigList

required

List of queue definitions to bind with the exchange.

queue

required

Queue node. See AMQP 0-9-1 Model Explained for more.

autodelete

optional

Automatically deletes queue when last consumer unsubscribes.

exclusive

optional, default is false

Used for only one connection. Queue is deleted when that connection closes.

name

required

Name of the queue.

routingKey

optional

Key used to route messages. Queue binds to the exchange with the routing key. Applicable when direct mode is enabled in the exchange. See Configuring monitoring.

name

required

Name of the exchange.

The name specified here must match the exchangeName property in the activate and deactivate commands for RabbitMQ, for example:

/modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@exchangeName

If you change the exchange name in one property, make sure you change it in the other one too.

type

required

Exchange type.

Options: fanout, direct, topic, headers See AMQP 0-9-1 Model Explained for more.

Testing the connection

You can test the connection in the RabbitMQ console.

When the client is enabled, the exchanges and queues should be visible in the RabbitMQ console. You can test this at  http://localhost:5672 , for example. 

Make sure the consumer is disabled on the author instance or it will re-consume what it had just tried to publish.


You should be able to see the queue in Queues

You should be able to see the exchange in Exchanges

RabbitMQ activation commands

The RabbitMQ Activation module bootstraps preconfigured activation commands in the standard Activation module.

You can select commands/catalogs to match your purpose for using RabbitMQ and use them in your apps. You can also customize them to suit your needs. 

The commands are configured in /modules/activation/commands/<catalog name>.

Node name

 
activation

 
commands

 
default

 
versioned

 
withContentSyncing

 
withContentSyncingVersioned

 
rbmqVersioned

Catalogs:

withContentSyncing
Allows you to hook into transmission over RabbitMQ once you have done a standard activation.
withContentSyncingVersioned
Adds versioning to withContentSyncing.
rbmqVersioned
Replaces standard activation with RabbitMQ versioned activation.

The activate and deactivate commands defined in the above catalogs depend on your client configuration and exchange configuration. The activate and deactivate commands each contain a sub-command that contain the client name and and exchange name as properties. 

For example:

Node name

Value

 
activation


 
commands


 
default


 
versioned


 
withContentSyncing


 
withContentSyncingVersioned


 
rbmqVersioned


 
activate


 
version


 
activate


 
rabbitmq-activate


 
clientName

<configured client name>

 
exchangeName

<configured exchange name>

 
delegateAck

true

 
stackSize

100

 
blocktime

100

 
blockActivation

true

 
minNbrOfPublics

2


Properties
:

clientName

required

Configured client name.

The clientName property must be the name of a configured RabbitMQ client (for example: @name). /modules/rabbitmq-connector/rabbitmq-client/<configured client>

exchangeName

required

Configured exchange name.

The exchangeName property must be the exchange name of a configured Rabbit MQ client (for example: /modules/rabbitmq-connector/rabbitmq-client/<configured client>/exchangeConfig@name).

delegateAck

required, default is true

When set to true message acknowlegements are delegated to RabbitMQ. Set to false if you want to use the rabbitmq-monitoring module instead.

stackSize

required, default is 1

Stack size allows you to send more nodes per activation message, making messages bigger but reducing the number of messages in the queue.

blocktime

optional, default is 100

Blocktime in seconds. Works only if delegateAck is false and if the ack exchange is configured correctly.

blockActivation

optional

Blockactivation. Works only if delegateAck is false and if the ack exchange is configured correctly.

minNbrOfPublics

optional

Minimum number of public instances to wait ack for. Works only if delegateAck is false and if the ack exchange is configured correctly.

Adding commands to apps

You can use the RabbitMQ commands in your apps, for example, to sync your content with another environment after an activation.

Example: Content syncing in the Pages app in /modules/pages/commands/website.

Node name

Value

 
pages


 
commands


 
website


 
activate


 
class

info.magnolia.commands.delegateCommand 

 
commandName

withContentSyncingVersioned-activate

 
activateDeletion


 
class

info.magnolia.commands.delegateCommand  

 
commandName

withContentSyncingVersioned-activateDeletion

 
deactivate


 
class

info.magnolia.commands.delegateCommand 

 
commandName

withContentSyncingVersioned-deactivate

Testing activation

To test the new commands activate content and then open the RabbitMQ console. You should see messages in the queue (fan1) . 

 

Configuring the activation consumer

Consuming messages on a public instance is called activation or publication. On other instances, for example another author instance in a staging environment, it is called syncing.

When the messages have successfully reached the queue:

  1. Start the receiving instance.
  2. Configure the activation consumer on the receiving instance to consume to the correct queue (queueName property).
  3. Enable the consumer.

(warning) Make sure the sampleClient configuration is enabled on both instances.

The example consumer definition is in /modules/rabbitmq-connector/rabbitmq-client/sampleClient/consumerDefinitions.

Node name

Value

 
sampleClient


 
consumerDefinitions


 
activationConsumer


 
ackExchangeName

 -

 
clientName

sample-client

 
consumerClass

info.magnolia.rabbitmq.activation.jobs.ActivationConsumerJob 

 
enabled

false

 
name

activationConsumer

 
queueName

fan1 

 
updatePolicy

update_strict=default or update_merge

Properties:

consumerDefinitions

required

Consumer definitions node. See

$webResourceManager.requireResource("info.magnolia.sys.confluence.artifact-info-plugin:javadoc-resource-macro-resources") ConsumerDefinition
.

activationConsumer

required

Activation consumer node.

ackExchangeName

optional

The name of the ACK exchange to use. An ACK exchange is bound to a queue on which activation confirmation messages are sent. It is used for monitoring. Leave empty if you do not need this.

clientName

required

Name of the client to use.

consumerClass

required

Consumer class to use.

$webResourceManager.requireResource("info.magnolia.sys.confluence.artifact-info-plugin:javadoc-resource-macro-resources") ActivationConsumerJob
: Activation consumer job. Custom classes must implement
$webResourceManager.requireResource("info.magnolia.sys.confluence.artifact-info-plugin:javadoc-resource-macro-resources") AbstractMQConsumerJob
.

enabled

optional

Enables and disables the consumer. Toggling this property shuts down and restarts the consumer.

name

required

Name of the consumer.

queueName

required

Name of the queue to consume on.

verifyAuthorSignature

optional, default is true

When set to true the activation public key is used to verify that the activation messages were transmitted by the author or instance containing the activation private key.

(warning) Add the correct public activation key to the receiving instance. Alternatively, disable public key verification by setting the property to false. This bypasses the signature check.

updatePolicy

optional, default is update_strict

Values possible are:

  • update_strict - behaves like traditional activation and will delete subnodes that are not of parent type and properties of nodes on each activation.
  • update_merge - only updates existing nodes, properties, i.e will merge nodes and properties, this is useful for independent language activation and synchronisation between staging environments for instance.

Message format, new txid

Activation IDs

Each activation is prefixed with a BOA (Beginning Of Activation) message and postfixed with a EOA (End Of Activation) message. When the consumer receives a message on a public instance, the message is stamped with the identity of the receiver and send back to the ack queue. When the author receives the message, the author knows exactly if the message started consuming the activation. In addition, identifying the activation in queue with BOA and EOA allow identification of TX inside the the queue.

Testing message consumption

When the activation consumer is configured, the receiving instance should start consuming messages.

Open the RabbitMQ console to verify that the specific queue (fan1) is being emptied and that the nodes appear on the receiving instance. 

 

Content Syncing

While transactions are possible with RabbitMQ, they are complicated and slow down publishing. Ideally, the same activation message would be received by all instances at the same time, but this is technically very difficult to achieve and does not add much advantage in any event.

What is important however is that a Magnolia instance knows when a node is out of sync with another node, and that the load balancer knows which node is late. This is achieved by recording the time stamp and latest message state

Whenever a message has been consumed and the content activated on the instance, the instance stores the message id (which is a long that is incremented) and the date the message was activated.

Example:

InstanceMessage IdTimestamp
Pubic 141441631639
Public 241441631640
Public 331441631600

This information is either sent to the load balancer or made available over a servlet. The goal is that the load balancer knows which instance is behind and by how much. In the example above the load balancer controlling tool knows that Public 1 and Public 2 are synced. It can decide to wait for Public 3 to sync, and if this does not happen, to alert the systems administrator or trigger the creation of a new instance.

SyncState and SyncStore mechanism

The Magnolia RabbitMQ implementation introduces the concepts of SyncState and SyncStore to achieve content synchronicity between public instances.

The mechanism is basically a counter that is incremented when each activation message is sent. The counter is sent in the message header. When the receiving instance gets the message, the instance increments its own activation counter. This means that two public instances which have different SyncStates are out of sync until the counters once again have the same number.

The load balancer calls the REST service of the public nodes and in this way knows which public node is most up to date and which one is out of sync.

Examples

StateReasonAction
REST service is not responding.Public node down.Remove from the pool.
Rest service's sync state is 0.Public Node is up but completely out of sync.

average(syncstates)-syncstate(pubnode)>=x

Public node is working but under heavy load.Remove public node temporarily from the pool until <x.

The loadbalancer also knows when the public instances are getting activated. SyncStates are persisted to the JCR.

Guaranteeing synchronicity 

The Dynamically Weighted Least Connections Algorithm on your load balancer guarantees synchronicity and ensures that public instances do not get too far out of sync. See Least-Connection Algorithm based on variable weight for multimedia transmission for more.

The load balancer plays a big role in this. During a very big and long activation process, the load balancer can decide to redirect the traffic to the instances which have the highest and most equal ids. This increases traffic on these instances, which in turn slows down the activation process. Instances with initial low and unequal ids now have a higher chance of reaching the same state as the instances under load. 

The algorithm creates a kind of "auto-damping" system that slows down instances by giving them more traffic, while speeding up the activation process on the others . Having the load balancer actively take the activation mechanism into consideration stabilizes the whole public node group. 


Next: Add monitoring to your RabbitMQ configuration


#trackbackRdf ($trackbackUtils.getContentIdentifier($page) $page.title $trackbackUtils.getPingUrl($page))
  • No labels