drush_queue_run_all

This module adds a Drush command that allows you to process all available queues in the website.

This originally started as a PR for Drush, but it was decided to (initially) let this command live in contrib.

Features

This module adds a drush queue:run-all command that does the same as drush queue:run, but for all available queues instead of a specific one. It takes the same options, but without the argument. The --time-limit and --items-limit options count across all queues instead of a specific queue.

The command has a --memory-limit option that takes either a number that ends with g/m/k (the same format as the memory_limit ini setting) or a percentage. The command will run until that amount of memory is consumed.

The command also has a --daemon option that keeps the command running until --items-limit, --lease-limit or --memory-limit is reached - if defined. Else, it keeps going indefinitely.

Usage with process managers

When used with a process manager like systemd, RoadRunner or Supervisord, it can be set up as a process that runs permanently and restarts automatically in case of failure. This is useful when you need queue items to be processed as fast as possible, instead of during every cron run.

Make sure to restart queue runners after deploying new code, otherwise the old code will keep running in memory until the queue runner is restarted. This can cause some very hard to debug issues. Trust me, I've been there.

Here are some example configurations to be used with different process managers. You will need to adjust things like the path to the PHP and Drush executables and the options passed to the command according to your setup and preferences.

systemd

[Unit]
Description=Queue runner
After=mysqld.service
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/bin/php %h/deploy/current/vendor/bin/drush queue:run-all --daemon --memory-limit=60%

[Install]
WantedBy=default.target

Supervisord

[program:queue_runner]
    command=/usr/bin/php vendor/bin/drush queue:run-all --daemon --memory-limit=200M
    redirect_stderr=true
    autostart=true
    autorestart=true
    user=<your user>
    numprocs=1
    directory=/home/deploy/current
    process_name=%(program_name)s_%(process_num)s

Known issues

Under heavy concurrency, the queue database table can suffer from SQLSTATE[40001]: Serialization failure: 1213 Deadlock found errors. This is not an issue with this module, but rather with Drupal core. More information and a working patch can be found in #3555806