The RabbitMQ modules have been moved to the Services incubator.

Refer to RMQ Publication in the incubator documentation for up-to-date information. This page is no longer maintained. 

The RabbitMQ module bundle provides an alternative way of publishing content and synchronizing instances in Magnolia.

RabbitMQ is open source message broker software (a.k.a message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). 

The Magnolia RabbitMQ modules are suitable for: 

  • Environments consisting of more than five public instances.  
  • Environments that need to synchronize with other environments.

The RabbitMQ alternative:

  • Gives you full control over activation and synchronization.
  • Reduces the load on your author environment.
  • Persists activations in queues, allowing for easier setup of continuous deployment environments.

This document guides you through the process of setting up RabbitMQ in your environment and installing and configuring the modules.

The RabbitMQ modules were built for (warning) Magnolia 5.5+. Working modules can be made available on request to (warning) Magnolia 5.4+ users, and 5.3+ users.


RabbitMQ bundle

The RabbitMQ bundle contains three modules:

  • RabbitMQ Connector module: Integrates RabbitMQ with Magnolia.
  • RabbitMQ Activation module: Allows you to synchronize and activate over RabbitMQ.
  • RabbitMQ Monitoring module. Allows you to monitor behavior on update for each instance that receives activation messages.

Installing Magnolia RabbitMQ modules

The modules are interdependent. Please install all three modules. 

Maven is the easiest way to install the modules. Add the following dependencies to your bundle:

Pre-built jars are also available for download. See Installing a module for help.

Installing RabbitMQ service

Follow the instructions at RabbitMQ to download and install the service. Instructions and packages are provided for all operating systems.

Note:

  • Magnolia modules are needed to set up RabbitMQ so install the three jars that are shipped with the parent POM first on your Magnolia instances. 
  • Connector module allows you to configure RabbitMQ directly from Magnolia. The module comes with a set of example consumers and producers that you can extend as necessary.
  • Activation module allows you to sync and activate content to other instances. Those instances can be public or author instances.
  • Monitoring module provides tools and a consumer for visualizing and monitoring the consuming process on all instances that receive activation messages.

Configuring the Rabbit MQ connector

RabbitMQ exchanges and consumers are configured in the connector module. The exchange is configured on the author or producing instance and the consumer on the public or receiving instance.

Configuring the RabbitMQ client

The module ships with an example client configuration in /modules/rabbitmq-connector/rabbitmq-client/sampleClient.

Node name

Value

 rabbitmq-connector


 rabbitmq-client


 sampleClient


 exchangeConfig


 consumerDefinitions


 clientName

sample-client 

 enabled

false 

 hostName

localhost 

 password

guest 

 portNumber

5672 

 userName

guest 

 virtualHost

Properties:

rabbitmq-client

required

RabbitMQ client folder.

<client name>

required

Client configuration node. One for each client.

exchangeConfig

required

Exchange configuration.

consumerDefinitions

required

Consumer definition.

clientName

required

Name of the client. Note, the clientName can be different from the <client name> node name.

Important: the clientName specified here must match the clientName property in the activate and deactivate commands for Rabbit MQ, e.g.

/modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@clientName

If you change the client name in one, be sure to change it in the other.

enabled

required

Enables and disables the client. Toggling this property restarts the client.

hostName

required

Address of the broker.

password

required

Password of the user account to connect with.

portNumber

required

Port number to connect the AMQP service.

username

required

Username of the user account to connect with.

virtualHost

required

Virtualhost to connect to.

Exchange configuration

The exchange configuration is configured on the author or producing instance.

The exchange is the kind of router that the queues connect to. The producer instance pushes messages to the exchange and the exchange then decides what to do with them.

The example exchange configuration is in /modules/rabbitmq-connector/rabbitmq-client/sampleClient/exchangeConfig.

Node name

Value

 sampleClient


 exchangeConfig


 queueConfigList


 queue


 autodelete

false 

 exclusive

false 

 name

fan1

 routingKey

 -

 name

testfanoutAck 

 type

fanout

Properties:

exchangeConfig

required

Exchange configuration node. See RabbitMQExchangeConfig .

queueConfigList

required

List of queue definitions to bind with the exchange.

queue

required

Queue node. See AMQP 0-9-1 Model Explained for more.

autodelete

optional

Automatically deletes queue when last consumer unsubscribes.

exclusive

optional, default is false

Used for only one connection. Queue is deleted when that connection closes.

name

required

Name of the queue.

routingKey

optional

Key used to route messages. Queue binds to the exchange with the routing key. Applicable when direct mode is enabled in the exchange. See Configuring monitoring.

name

required

Name of the exchange.

Important: the name specified here must match the exchangeName property in the activate and deactivate commands for Rabbit MQ, e.g.

/modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@exchangeName

If you change the exchange name in one, be sure to change it in the other.

type

required

Exchange type.

Options: fanout, direct, topic, headers See AMQP 0-9-1 Model Explained for more.

Testing the connection

You can test the connection in the RabbitMQ console.

When the client is enabled, the exchanges and queues should be visible in the RabbitMQ console. You can test this at http://localhost:5672, for example. (warning) Make sure the consumer is disabled on the author instance or it will re-consume what it had just tried to publish.

You should be able to see the queue in Queues

You should be able to see the exchange in Exchanges

RabbitMQ activation commands

The RabbitMQ Activation module bootstraps preconfigured activation commands in the standard Activation module.

You can select commands/catalogs to match your purpose for using RabbitMQ and use them in your apps. You can also customize them to suit your needs. 

The commands are configured in /modules/rabbitmq-activation(activation)/commands/<catalog name>.

Node name

 rabbitmq-activation/activation

 commands

 default

 versioned

 withContentSyncing

 withContentSyncingVersioned

 rbmqVersioned

Catalogs:

withContentSyncing
Allows you to hook into transmission over RabbitMQ once you have done a standard activation.
withContentSyncingVersioned
Adds versioning to withContentSyncing.
rbmqVersioned
Replaces standard activation with RabbitMQ versioned activation.

The activate and deactivate commands defined in the above catalogs depend on your client configuration and exchange configuration. The activate and deactivate commands each contain a sub-command that contain the client name and and exchange name as properties. 

For example:


Node name

Value

 rabbitmq-activation/activation


 commands


 default


 versioned


 withContentSyncing


 withContentSyncingVersioned


 rbmqVersioned


 activate


 version


 activate


 rabbitmq-activate


 clientName

<configured client name>

 exchangeName

<configured exchange name>

The clientName property must be the name of a configured Rabbit MQ client (for example: /modules/rabbitmq-connector/rabbitmq-client/<configured client>@name). 

The exchangeName property must be the exchange name of a configured Rabbit MQ client (for example: /modules/rabbitmq-connector/rabbitmq-client/<configured client>/exchangeConfig@name).

Adding commands to apps

You can use the RabbitMQ commands in your apps, for example, to sync your content with another environment after an activation.

Example: Content syncing in the Pages app in /modules/pages/commands/website.

Node name

Value

 pages


 commands


 website


 activate


 class

info.magnolia.commands.delegateCommand 

 commandName

withContentSyncingVersioned-activate

 activateDeletion


 class

info.magnolia.commands.delegateCommand  

 commandName

withContentSyncingVersioned-activateDeletion

 deactivate


 class

info.magnolia.commands.delegateCommand 

 commandName

withContentSyncingVersioned-deactivate

Testing activation

To test the new commands activate content and then open the RabbitMQ console. You should see messages in the queue (fan1) . 

 

Configuring the activation consumer

Consuming messages on a public instance is called activation or publication. On other instances, for example another author instance in a staging environment, it is called syncing.

When the messages have successfully reached the queue:

  1. Start the receiving instance.
  2. Configure the activation consumer on the receiving instance to consume to the correct queue (queueName property).
  3. Enable the consumer.

(warning) Make sure the sampleClient configuration is enabled on both instances.

The example consumer definition is in /modules/rabbitmq-connector/rabbitmq-client/sampleClient/consumerDefinitions.

Node name

Value

 sampleClient


 consumerDefinitions


 activationConsumer


 ackExchangeName

 -

 clientName

sample-client

 consumerClass

info.magnolia.rabbitmq.activation.jobs.ActivationConsumerJob 

 enabled

false

 name

activationConsumer

 queueName

fan1 

Properties:

consumerDefinitions

required

Consumer definitions node. See ConsumerDefinition .

activationConsumer

required

Activation consumer node.

ackExchangeName

optional

The name of the ACT exchange to use. An ACT exchange is a queue on which activation confirmation messages are sent. It is used for monitoring. Leave empty if you do not need this.

clientName

required

Name of the client to use.

consumerClass

required

Consumer class to use.

ActivationConsumerJob : Activation consumer job. Custom classes must implement AbstractMQConsumerJob .

enabled

optional

Enables and disables the consumer. Toggling this property shuts down and restarts the consumer.

name

required

Name of the consumer.

queueName

required

Name of the queue to consume on.

verifyAuthorSignature

optional, default is true

When set to true the activation public key is used to verify that the activation messages were transmitted by the author or instance containing the activation private key.

(warning) Add the correct public activation key to the receiving instance. Alternatively, disable public key verification by setting the property to false. This bypasses the signature check.

Testing message consumption

When the activation consumer is configured, the receiving instance should start consuming messages.

Open the RabbitMQ console to verify that the specific queue (fan1) is being emptied and that the nodes appear on the receiving instance. 

 

Content Syncing

While transactions are possible with RabbitMQ, they are complicated and slow down publishing. The ultimate would be to have the same activation message received by all instances at the same time, but this is technically very difficult to achieve and does not add much advantage in any event.

What is important however is that a Magnolia instance knows when a node is out of sync with another node, and that the load balancer knows which node is late. This is achieved by recording the time stamp and latest message state

Whenever a message has been consumed and the content activated on the instance, the instance stores the message id (which is a long that is incremented) and the date the message was activated.

Example:

InstanceMessage IdTimestamp
Pubic 14
1441631639
Public 24
1441631640
Public 33
1441631600

This information is either sent to the load balancer or made available over a servlet. The goal is that the load balancer knows which instance is behind and by how much. In the example above the load balancer controlling tool knows that Public 1 and Public 2 are synced. It can decide to wait for Public 3 to sync, and if this does not happen, to alert the systems administrator or trigger the creation of a new instance.

SyncState and SyncStore mechanism

The Magnolia RabbitMQ implementation introduces the concepts of SyncState and SyncStore to achieve content synchronicity between public instances.

The mechanism is basically a counter that is incremented when each activation message is sent. The counter is sent in the message header. When the receiving instance gets the message, the instance increments its own activation counter. This means that two public instances which have different SyncStates are out of sync until the counters once again have the same number.

The load balancer calls the REST service of the public nodes and in this way knows which public node is most up to date and which one is out of sync.

Examples

StateReasonAction
REST service is not responding.Public node down.Remove from the pool.
Rest service's sync state is 0.Public Node is up but completely out of sync.

average(syncstates)-syncstate(pubnode)>=x

Public node is working but under heavy load.Remove public node temporarily from the pool until <x.

The loadbalancer also knows when the public instances are getting activated. SyncStates are persisted to the JCR.

Guaranteeing synchronicity 

The Dynamically Weighted Least Connections Algorithm on your load balancer guarantees synchronicity and ensures that public instances do not get too far out of sync. See Least-Connection Algorithm based on variable weight for multimedia transmission for more.

The load balancer plays a big role in this. During a very big and long activation process, the load balancer can decide to redirect the traffic to the instances which have the highest and most equal ids. This increases traffic on these instances, which in turn slows down the activation process. Instances with initial low and unequal ids now have a higher chance of reaching the same state as the instances under load. 

The algorithm creates a kind of "auto-damping" system that slows down instances by giving them more traffic, while speeding up the activation process on the others . Having the load balancer actively take the activation mechanism into consideration stabilizes the whole public node group. 

Public Monitoring app

The RabbitMQ Monitoring module includes the Public Monitoring app and the Public monitoring REST service

In the Public Monitoring app can can follow the state of any instance that is consuming on a queue. The app will return information messages containing the status of activations, provided the ACK exchange is configured correctly, 

If an activation fails, the message is put back into the queue and an exception is pushed into the ackQueue. The consumer is then stopped allowing you to fix the problem and to remove the faulty instance from the load balancer. There are techniques to fix the problem which are discussed in ACK queue blocked by unacked node

Configuring monitoring

For monitoring to function correctly the ACK client needs to be configured and enabled and the monitoring module installed. 

Monitoring is configured on the instance responsible for monitoring, typically the author or producing instance.

Configuration involves three steps:

  1. Configuring the ACK client .
  2. Adding the ACK exchange name to the activation consumer definition
  3. Configuring REST endpoints.

Configuring the ACK client

The ACK client is configured in the connector module.

The example ACK client configuration is in /modules/rabbitmq-connector/rabbitmq-client/ackClient/consumerDefinitions/ackConsumer.

Node name

Value

 rabbitmq-connector


 rabbitmq-client


 ackClient


 exchangeConfig


 queueConfigList


 name

actExchange

 type

direct 

 consumerDefinitions


 ackConsumer


 ackExchangeName

 -

 clientName

ack-client

 consumerClass

info.magnolia.rabbitmq.consumers.ConfirmationConsumerJob 

 enabled

false

 name

confirmConsumer

 queueName

ackQueue

 clientName

ack-client

 enabled

false 

 hostName

localhost 

 password

guest 

 portNumber

5672 

 userName

guest 

 virtualHost

Properties: 

rabbitmq-client

required

RabbitMQ client folder.

ackClient

optional

ACK client node.

exchangeConfig

required

Exchange configuration node.

queueConfigList

required

Queue configuration list node.

name

required

Name of the ACT exchange.

type

required

Exchange type. See AMQP 0-9-1 Model Explained for more.

consumerDefinitions

required

Consumer definitions node.

ackConsumer

required

ACK consumer node.

ackExchangeName

optional

The name of the ACK exchange to use.

clientName

required

Name of the client to use.

consumerClass

required

Consumer class to use.

ConfirmationConsumerJob : Confirmation consumer job.

enabled

optional

Enables and disables the ACK consumer. Toggling this property shuts down and restarts the consumer.

name

required

Name of the consumer.

queueName

required

Name of the queue to monitor.

<client properties>

required/optional

See Configuring the RabbitMQ connector for details.

Adding the ACK exchange name

The next step to configuring monitoring is to link the ACT client in the the activation consumer definition . 

To do this add the actExchangeName to the activation consumer definition in /modules/rabbitmq-connector/rabbitmq-client/sampleClient/consumerDefinitions/activationConsumer/ackExchangeName, for example.

Node name

Value

 sampleClient


 consumerDefinitions


 activationConsumer


 ackExchangeName

ackExchange

 clientName

sample-client

 consumerClass

info.magnolia.rabbitmq.activation.jobs.ActivationConsumerJob 

 enabled

true

 name

activationConsumer

 queueName

fan1 

 verifyAuthorSignature

false

Configuring REST endpoints

The final step to configuring monitoring is to configure REST endpoints and add the access rights to them. Permissions to issue REST requests are controlled by Magnolia's standard role-based security mechanism.

REST endpoints are used for getting information frommonitoring and controlling the activation status

SyncState REST service

The SyncState REST service allows you to get the current state of activation immediately (without checking the ACK return queue). 

The service is configured in the activation module in /modules/rabbitmq-activation/rest-endpoints/syncstate.

Node name

Value

 rabbitmq-activation


 syncNode


 seq_nbr

2,012

 stamp

1,467,280,192,928

 topo_tag

-

 rest-endpoints


 syncstate


 class

info.magnolia.rest.service.command.definition.ConfiguredCommandEndpointDefinition

 implementationClass

info.magnolia.rabbitmq.activation.rest.SyncStateRestService

Properties:

syncNode

required

Sync node node. (warning) The properties in this node are updated automatically. They are not for configuration purposes. Do not edit them unless you really know what you are doing.

seq_nbr

required

Sequence number.

stamp

required

Time stamp.

topo_tag

required

Topo tag.

rest-endpoints

optional

REST endpoints folder.

syncstate

optional

SyncState node.

class

required

REST endpoint class.

ConfiguredCommandEndpointDefinition : Command endpoint definition that adds a white-list for enabled commands.

implementationClass

required

REST endpoint implementation class.

SyncStateRestService : Provides sync status of activation by REST.

To enable the service:

  1. In the Security app add the following permissions to the rest role: 

    Access control lists:

    WorkspacePermissionScopePath
    ConfigRead onlySelected and sub nodes/rabbitmq-activation

    Web access:

    PermissionPath
    Get & Post.rest/syncstate/state
  2. Assign the rest role to a user or to anonymous if you need access without credentials. 

You can use the REST endpoint as follows:

curl http://publicaddress:publicport/.rest/syncstate/state

You should get the following (example):

{"seqNbr":2011,"stamp":1467280192928,"topoTag":""}

Public monitoring REST service

The Public monitoring REST service allows you to get the results of all consuming instances returning messages on the ACK queue. This service is used by the Public Monitoring app.

The service is configured in the monitoring module in /modules/magnolia-rabbitmq-monitoring/rest-endpoints/pubstate.

Node name

Value

 magnolia-rabbitmq-monitoring


 rest-endpoints


 pubstate


 class

info.magnolia.rest.service.command.definition.ConfiguredCommandEndpointDefinition

 implementationClass

info.magnolia.rabbitmq.rest.PublicMonitoringService

Properties:

rest-endpoints

optional

REST endpoints folder.

pubstate

optional

Pub state node.

class

required

REST endpoint class.

ConfiguredCommandEndpointDefinition : Command endpoint definition that adds a white-list for enabled commands.

implementationClass

required

REST endpoint implementation class.

PublicMonitoringService : Provides public monitoring status by REST.

To enable the service:

  1. In the Security app add the following permissions to the rest role: 

    Access control lists:

    WorkspacePermissionScopePath
    PblcMntrngRead onlySelected and sub nodes/

    Web access:

    PermissionPath
    Get & Post.rest/public-monitoring*
  2. Assign the rest role to a user or to anonymous if you need access without credentials.

You can use the REST endpoint as follows:

curl http://monitoringinstance:port/.rest/public-monitoring/all

This should return something like:

[{"workspace":"dam","address":"192.168.10.85","name":"administrators-MacBook-Pro.local","syncStamp":"1466422838236","diff":"0","uuid":"2acd7055-7168-4926-b66a-47400ab78d2e","seqNbr":"1977","exceptions":"{}"}]

Properties:

workspaceWorkspace of last activated node.

address

IP of consuming instance.
namehostName of consuming instance.
syncStampTime stamp of last activated node.
diffDifference on seqNbr with other instances who activated a similar node.

uuid

UUID of last activated node.
seqNbrSequence number of last activated node.
exceptionsErrors.

You can access the monitoring screen at http://localhost:8080/.resources/magnolia-rabbitmq-monitoring/webresources/visualisations/public_monitoring.html or in the Public Monitoring app.

Client control REST service

The client control REST service allows you to restart the consumer on the remote instance via a cURL command.

The service is configured in the connector module in /modules/rabbitmq-connector/rest-endpoints/consumer.

Node name

Value

 rabbitmq-connector


 rest-endpoints


 consumer


 class

info.magnolia.rest.service.command.definition.ConfiguredCommandEndpointDefinition

 implementationClass

info.magnolia.rabbitmq.activation.rest.ClientControlRestService

Properties:

rest-endpoints

optional

REST endpoints folder.

consumer

optional

Consumer node.

class

required

REST endpoint class.

ConfiguredCommandEndpointDefinition : Command endpoint definition that adds a white-list for enabled commands.

implementationClass

required

REST endpoint implementation class.

ClientControlRestService : Provides RabbitMQ client control by REST.

To enable the service:

  1. In the Security app add the following permission to the rest role: 

    Web access:

    PermissionPath
    Get & Post.rest/rbmqClients*
  2. Assign the rest role to a user or to anonymous if you need access without credentials.

You can use the REST endpoint as follows:

curl http://publicaddress:publicport/.rest/rbmqClients/restartAll

The command will restart all consumers and clients on that instance.

Continuous deployment strategy

How to synchronize new public instances

Public instance backups should be done when the SyncState message id is identical for all public nodes. You can then choose one public node and to do a dump. Once the dump is done, create a new historical queue and empty the old queue by consuming all messages. The instance will then be ready to receive new activation messages which post-date the last dump. 

Whenever you need to create a new instance, always use the latest dump. The new instance will be registered to the new historical queue. This will mean that the message id is in SyncState with the latest activation message. New activation messages will be sent to the new historical queue before they register in the old historical queue. The new historical queue will start getting all messages after the last message in old historical queue.

Adding new instances

This series of diagrams shows the best setup for scaling up when adding new public instances to the load balancer.

  1. The basic idea is to always keep a spare queue (which does not have a consumer) connected to the exchange. 
  2. The spare queue stores all activation messages until a new consumer connects. The newly created instance is created with the same initial data as Public 1.
  3. Public 2 is created with the same data set as Public 1. Before Public 2 starts consuming a new queue (without a consumer) is created.
  4. Once the remaining messages are consumed a backup is created. This serves as the new state for new instances. This is also the time to add the new public instance to the load balancer.

    When to do a backup?

    Each message has a version number or tag in its header. When a new public is added it can tell the author instance or the exchange about its presence by sending its identity before consuming starts. This enables the exchange or author to create the new queue for the public instance.

    From this point, all activation messages contain a new version or tag and the consumer starts consuming messages in the queue. When the consumer comes across a message with the newest version number, it does the backup.

  5. Activation continues and the spare queue (queue_pub_3) is filled with new messages that represent the difference between Public 2 and newest content.

Standard backups

Regular backups are needed to prevent the queue becoming too big. Whenever you do a backup on the instances the spare queue can be emptied.

Adding a queue after backup

This example shows you how to add a new queue to the example testfanoutAck exchange:

  1. Add the queue:

    curl -i -u guest:guest -H "content-type:application/json" \
      -XPUT -d'{"auto_delete":false,"durable":true,"arguments":{},"node":"rabbit@localhost"}' \
      http://localhost:55672/api/queues/%2f/fan2
  2. Declare it to be bound to correct exchange:

    curl -i -u guest:guest -H "content-type:application/json" \
       -d'{"routing_key":"","arguments":[]}' \
      http://localhost:55672/api/bindings/%2f/fan2/testfanoutAck

See RabbitMQ Management HTTP API for more.

Troubleshooting and special uses cases

Node Activation Policy

The node order is respected when a node is created, but it is not respected on update.

To ensure that the node order is respected on update you can add the PUB_POLICY property to the parent of the nodes that need a specific order.  

The PUB_POLICY property has two valid values that result in the following behaviors:

ValueBehavior
respect_orderNodes are always deleted on update. (warning) You need to use the Publish incl.subpages/nodes action to activate or the subnodes will not be created.
remove_orphan_nodes
Nodes whose parent was not found in the correct place are ignored. (Not implemented yet.)

ACK queue blocked by unacked node

When a consumer encounters an issue because of an activation exception, the consumer stops consuming. The message is sent back to the queue with a non-ack, and an exception is sent over the ACK exchange. 

To illustrate this we deleted a node on the public instance and then tried to deactivate it on the author instance. 

The deletion and attempted deactivation resulted in the following error:

ERROR lia.rabbitmq.activation.jobs.ActivationConsumerJob: /bla

The exception is visible in the Public Monitoring app. You can see from the PathNotFoundException that the consumer did not find the path that was deleted directly of the public instance.

In the RabbitMQ console the:

  • Message is not consumed in the queue.
  • Message is marked unacked.
  • Consumer is gone,

When more nodes are published into the queue they become blocked behind the unacked message.

There are two ways to solve this:

  1. Recreate the node on the public instance and restart the consumer.
    • (plus) The sequence number count remains correct.
    • (minus) You have to connect to the public instance and do "manual manipulation".
  2. Remove the faulty activation message from the queue and restart the consumer.
    • (plus) This can be done remotely and there is no need to connect "manually" to the public instance. 
    • (minus) Because there is one "missing" message the sequence number is lost.

Here's a step-by-step solution using option 2 (because it demonstrates how to delete an unacked message):

  1. Close the channel in the RabbitMQ console. This acks all unacked messages and is needed because messages can no longer be consumed.
     
  2. Force close the connection
     
  3. All messages are now acked in the queue.
     
  4. Consume only the "first" message.

  5. Restart the consumer on the public instance.

    curl http://localhost:8081/.rest/rbmqClients/restartAll
  6. The instance is getting activated again.
     
#trackbackRdf ($trackbackUtils.getContentIdentifier($page) $page.title $trackbackUtils.getPingUrl($page))