RabbitMQ & Magento 2 on Managed Hosting by Nexcess
Before explaining the process of enabling and configuring RabbitMQ/Magento 2, let us understand what containers and RabbitMQ are in term of cloud technology.
RabbitMQ & Magento 2
RabbitMQ is an open source message broker that helps websites to exchange data between processes, applications, and servers. This cloud technology item can help reduce load and delivery times by delegating tasks to a third party.
Nexcess provides the RabbitMQ cloud container service option for our Managed Applications (MA) Magento Plans. This article will cover how to optimally configure RabbitMQ for Magento 2 after enabling the corresponding option via my.nexcess.net.
Enabling the Magento 2 RabbitMQ Container
To learn how to enable the RabbitMQ container, check out the How to enable containers in Nexcess Cloud article. Once you enable the RabbitMQ container, you will receive details shown in the screenshot below related to the service.
Endpoint, Port, Username
Click on View Password option to find its related password. You will need that password information in the next step of configuring the service.
Configuring the Magento 2 RabbitMQ Container
To configure the service, you need to edit “/home/username/sitename/html/app/etc/env.php” file.
In that file you, will see below section of code:
'queue' => [
'consumers_wait_for_messages' => 1
],You have to edit the code as follows:
'queue' => [
'amqp' => [
'host' => 'Endpoint',
'port' => 'port-number',
'user' => 'username',
'password' => 'password',
'virtualhost' => '/'
],
'consumers_wait_for_messages' => 1
],Make sure you have entered the correct values in the host, port, user, and password areas. Then save that file and run the below command to apply the changes:
php bin/magento setup:upgradeThat’s it! You have successfully enabled and configured RabbitMQ Magento 2 on Nexcess!
Manage Message Queues
You can manage message queues from the command line using cron jobs to ensure that consumers retrieve messages.
Process Management
Cron jobs are the default mechanism to restart consumers. Processes started by cron consume the specified number of messages and then terminate. Rerunning cron restarts the consumer.
The following example shows the Magento crontab configuration for running consumers.
Path of cron job related to Message Queue:
vendor/magento/module-message-queue/etc/crontab.xml
$ cat vendor/magento/module-message-queue/etc/crontab.xml
<?xml version="1.0"?>
<!--
~ Copyright (c) Magento, Inc. All rights reserved.
~ See COPYING.txt for license details.
-->
<config xmlns_xsi="http://www.w3.org/2001/XMLSchema-instance" xsi_noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/crontab.xsd">
<group id="default">
<job name="messagequeue_clean_outdated_locks" instance="MagentoFrameworkMessageQueueLockWriterInterface" method="releaseOutdatedLocks">
<schedule>0 * * * *</schedule>
</job>
</group>
<group id="consumers">
<job name="consumers_runner" instance="MagentoMessageQueueModelCronConsumersRunner" method="run">
<schedule>* * * * *</schedule>
</job>
</group>
</config>How often you check message queues depends on your business logic and available system resources. In general, you’ll probably want to check for newly created customers and send welcome emails more frequently than a more resource-intensive process (for example, updating your catalog). It would be best if you defined cron schedules according to your business needs.
To learn how to configure cron jobs for Magento 2, check out the How to configure Magento 2 cron jobs article.
Configuration
The configuration behaviors by default are:
- The cron job consumers_runner is enabled.
- The cron job consumers_runner runs all defined consumers.
- Each consumer processes 10000 messages and then terminates.
Specific Configuration
Edit the /app/etc/env.php file to configure the cron job consumers_runner:
...
'cron_consumers_runner' => [
'cron_run' => false,
'max_messages' => 20000,
'consumers' => [
'consumer1',
'consumer2',
]
],
...Definitions:
- cron_run – A boolean value that enables or disables the consumers_runner cron job (default = true).
- max_messages – The maximum number of messages each consumer must process before terminating (default = 10000). Although we do not recommend it, you can use 0 to prevent the consumer from terminating. See consumers_wait_for_messages to configure how consumers process messages from the message queue.
- consumers – An array of strings specifying which consumer(s) to run. An empty array runs all consumers.
View a List of Available Message Queue Consumers
To view a list of all consumers, run the following command:
bin/magento queue:consumers:listStart Message Queue Consumers
To start message queue consumers, run the following command:
bin/magento queue:consumers:start [--max-messages=<value>] [--batch-size=<value>] [--single-thread] [--area-code=<value>] <consumer_name>
After consuming all available messages, the command terminates. You can run the command again manually or with a cron job. You can also run multiple instances of the magento queue:consumers:start command to process large message queues. For example, you can append & to the command to run it in the background, return to a prompt, and continue running commands:
bin/magento queue:consumers:start <consumer_name> &
Common RabbitMQ Mistakes & How to Avoid Them
Common Mistake #1: Don’t use too many connections or channels
Try to keep the connection/channel count low. For example, use separate connections to publish and consume. Ideally, you should have one connection per process; and use one channel per thread in your application.
How best to reuse connections:
- One connection for publishing
- One connection for consuming
Common Mistake #2: Don’t have queues that are too large or too long
Short queues are fastest; when a queue is empty, and it has consumers ready to receive messages, as soon as a message is received, it goes straight out to the consumer.
Common Mistake #3: Don’t have an unlimited prefetch value
A typical mistake is to have an unlimited prefetch value, where one client receives all messages. This configuration can lead to the client running out of memory and crashing, and then all messages are redelivered.
Common Mistake #4: Don’t ignore lazy queues
Use lazy queues to achieve predictable performance or when you have large queues. Lazy queues create a more stable cluster with more predictable performance. Your messages will not, without warning, get flushed to disk.
Common Mistake #5: Use multiple queues and consumers
Achieve better throughput on a multi-core system with multiple queues and consumers. You will achieve optimal throughput if you have as many queues as cores on the underlying node(s).
Conclusion
Successfully integrating RabbitMQ with Magento 2 on the Nexcess platform involves a straightforward process of enabling the container, retrieving the necessary credentials, and accurately updating the app/etc/env.php file. Beyond this initial setup, effective management of the message queues is critical, requiring proper configuration of the consumers_runner cron job and an understanding of the available CLI commands to control consumers. By following these configuration steps and adhering to best practices—such as limiting connections, managing prefetch values, and using lazy queues—developers can ensure a stable, high-performance system that leverages RabbitMQ to optimize their Magento 2 store’s asynchronous processing and scalability.