Queue jobs piling up or timing out are a common pain point for Craft CMS sites. But with a daemonised queue runner and job priorities appropriately set, you can alleviate this concern, guaranteeing a seamless content authoring experience. And with custom queues, you can configure multiple queues to run in parallel for an even smoother control panel workflow.
The Problem with Queues #
Most long-running tasks in Craft are handled in queue jobs, sets of tasks that can be time and process-intensive. By running these tasks in a background process, the control panel remains usable while those jobs are executed.
But the queue can also act like a bottleneck if too many jobs are added or if jobs take too long to complete. Say, for example, that a content editor changes an entry that triggers a large generate cache job in Blitz and several other related jobs. At the same time, another content editor decides to send a Campaign newsletter, which kicks off a sendout job.
Since jobs are run in the order in which they are added to the queue, all of the jobs must be completed before the sendout job can begin.
To make matters worse, by default in Craft, pending queue jobs are only run when someone visits the control panel. They are also run via web requests, which often have low execution time limits (in the region of 1 to 5 minutes) and conservative memory limits.
A queue job will fail if it times out or exceeds the memory limit, and the only way to resume running the other jobs in the queue is with a new web request. If no users are logged in to the control panel, then the queue jobs will remain in a pending state.
Queue Runners #
Both the Robust queue job handling in Craft CMS article and the Craft CMS docs on Queue Runners go into more detail and present potential options for using queue runners to deal with the challenges of queue jobs.
Running queue/listen via a daemonised queue runner is the clear winner. It continually checks (every three seconds, by default) whether there are any jobs on the queue, and if there are then it runs them. If the process ever stops running, the daemon will automatically restart it.
This requires setting up a daemon (or a worker), but with just one command, you no longer have to worry about queue jobs timing out or running out of memory.
/usr/bin/nice -n 10 /usr/bin/php /var/www/craft queue/listen
The -n 10 specifies the nice level, or the relative priority of the process. A process inherits its “niceness” from its parent by default (usually 0). Setting the nice level to 10 helps prevent the process from interfering with requests to the control panel or front-end.
You’ll need to add the correct path to the craft executable above and ensure that you set the runQueueAutomatically config setting to false.
I recommend limiting the number of processes to 1 when setting up a daemon. While it is tempting to run queue jobs concurrently (at the same time), it is much safer to run them sequentially (one after the other). This is because the result of running some jobs may affect other jobs in deterministic ways.
This is also the reason I don’t recommend using cron jobs to run the queue. If the jobs in the queue take longer to complete than the cron interval then multiple processes will end up running concurrently, unless you use a lock file to prevent this.
As per the docs, the queue should never be run as root!
Queue Runners in Local Dev #
While queue runners are less important in local development environments, it can be helpful to achieve parity with how things are set up in your production environment.
You can do this by manually running craft queue/listen or, if using DDEV, adding a post-start hook.
hooks:
post-start:
- exec: "php craft queue/listen"
Queue Job Priorities and TTR #
Each queue job is given a priority and a TTR (time to reserve), which is the amount of time in which it must be completed. Queue jobs in Craft get a priority of 1024 (the lower the number, the higher the priority) and a TTR of 300 seconds (5 minutes), by default.
Some plugins allow you to configure the priority and TTR of the queue jobs they provide, such as with Campaign’s config settings.
return [
'sendoutJobPriority' => 10,
'sendoutJobTtr' => 600,
];
By setting the value of sendoutJobPriority to 10, we’re making it a much higher priority than the default. Therefore, the sendout job will be run before other jobs with lower priority, even if they were added to the queue first.
If a plugin doesn’t allow you to configure the priority and TTR of the queue jobs via config settings then you can still achieve this with a custom module/plugin by listening for the Queue::EVENT_BEFORE_PUSH event.
use custom\plugin\jobs\CustomPluginJob;
use yii\base\Event;
use yii\queue\PushEvent;
use yii\queue\Queue;
Event::on(Queue::class, Queue::EVENT_BEFORE_PUSH,
function (PushEvent $event) {
if ($event->job instanceof CustomPluginJob) {
$event->priority = 10;
$event->ttr = 600;
}
}
);
Batched Jobs #
Some plugins allow you to split large queue jobs into batches. Campaign has batch sending settings, for example, that determine how many emails can be sent out at a time before a new queue job is kicked off to send the next batch.
return [
'maxBatchSize' => 1000,
];
Batching queue jobs makes them less likely to timeout or run out of memory. It also means that higher priority queue jobs can be run in between batch jobs, when necessary.
While the Campaign plugin implements its own job batching, the ability to batch jobs was added in Craft 4.4.0, so plugins can now support batched jobs using native functionality.
Configuring Custom Queues #
Running queue jobs via a daemonised process and setting an appropriate priority and TTR are the most important steps. But we can take it a step further.
Craft only has a single queue. That means that if, as I recommend above, you limit the number of processes to 1, only one queue job can ever run at a time.
But Craft also lets us configure and bootstrap a custom queue via the application configuration in /config/app.php.
return [
'bootstrap' => ['customQueue'],
'components' => [
'customQueue' => [
'class' => \craft\queue\Queue::class,
'channel' => 'custom',
],
],
];
We’re telling Craft to configure a customQueue component to Craft’s Queue class using a custom channel, and to bootstrap it. We could use any queue class that extends yii\queue\Queue here.
In this case, we’re using the Craft queue but a channel other than the default queue channel. This allows us to add jobs to a separate channel in the queue, which Craft will not run on its own. We must use a unique console command to run jobs in the custom channel, which is a kebab-cased version of the component name.
php craft custom-queue/run
We can create a second daemon to listen to queue jobs in this channel as follows.
/usr/bin/nice -n 10 /usr/bin/php /var/www/craft custom-queue/listen
Custom Queues in Plugins #
So, how does this help us? Well, some plugins with long-running processes, such as Blitz, Cache Igniter, Campaign and Feed Me, allow you to specify a custom queue to add jobs to.
Here’s how to configure Blitz to use the custom queue in /config/app.php.
return [
'bootstrap' => ['customQueue'],
'components' => [
'plugins' => [
'pluginConfigs' => [
'blitz' => [
'queue' => 'customQueue',
],
],
],
'customQueue' => [
'class' => \craft\queue\Queue::class,
'channel' => 'custom',
],
],
];
Now, each time Blitz creates a queue job, it will be added to the custom queue. And since we’ve set up a daemon to listen for queue jobs on the custom queue, those jobs will be run as expected.
The benefit of this is that long-running queue jobs generated by Blitz will no longer hold up queue jobs in Craft’s default queue!
The downside of using a custom queue, however, is that it will not appear in the Queue Manager utility in the control panel. This will hopefully become possible in a future Craft update.
Prioritising Queues #
We now have two queue runners that will never block each other’s progress!
Each queue runs in a separate PHP process. If your site runs on a web server with two cores (CPUs), then each process will be handled by each core. However, if the web server only has a single core, the processes will have to share the available resources.
We can prioritise one queue over the other by adjusting its nice level. For example, if we want to give the custom queue a lower priority, all we have to do is make it “nicer” than the default queue.
/usr/bin/nice -n 10 /usr/bin/php /var/www/craft queue/listen
/usr/bin/nice -n 11 /usr/bin/php /var/www/craft custom-queue/listen
And best of all, we can add as many unique custom queues as we want!
/usr/bin/nice -n 10 /usr/bin/php /var/www/craft queue/listen
/usr/bin/nice -n 11 /usr/bin/php /var/www/craft campaign-queue/listen
/usr/bin/nice -n 12 /usr/bin/php /var/www/craft blitz-queue/listen
/usr/bin/nice -n 13 /usr/bin/php /var/www/craft feed-me-queue/listen
Practically speaking, however, you shouldn’t go overboard with daemons. My general advice is to limit the number of daemons to your web server’s available CPUs. Also, if you want control panel users to have visibility into queue jobs, then, at least in the meantime, you should ensure they run in the default queue.
Interjection #
But hang on a second, Ben. You said that queue jobs should not be run concurrently. Doesn’t having multiple queues mean that jobs will end up running at the same time?
You’re absolutely right, dear reader! Which is why you should carefully choose which queue jobs should be allowed to run in their own custom queues.
The queue job that Cache Igniter creates, for example, often depends on Blitz queue jobs being completed first. So both of those plugins should be run using the same queue. Campaign’s sendout jobs and Feed Me’s import jobs, on the other hand, should be safe to run independently of Blitz and Cache Igniter’s queue jobs (and of each other), meaning they can be run using different queues.
This allows us to safely run multiple queue runners in parallel, ensuring that queue jobs are completed faster while not interfering with each other.
Conclusion #
A daemonised queue runner is essential for all but the simplest of Craft sites. Use the steps below to incrementally improve and tweak the queue running process.
- Set up a daemonised process for running queue jobs.
- Configure the priority and TTR of queue jobs accordingly.
- Add one or more custom queues for mutually exclusive jobs, depending on the web server’s available cores.