Direct Queue

About

Direct Queue allows you to process queue items concurrently outside of the Drupal cron pipeline.
The module uses Drupal Console to process queue items.

Why

Sometimes tasks can take a very long time, you don't want queue items waiting for each other and you want to be able to process multiple queue items at the same time. By default, Drupal Queue Items are processed in the cron and run sequential, there is no concurrency. It's also common to only run the queue every x minutes (depends on configuration), so you have to wait for the next cron run.

Direct Queue provides a method to process queue items separated from the cron to make sure your Drupal site is able to handle many queue items, processes them almost immediately and process queues concurrently.

How

Direct Queue hooks into the regular Drupal (cron) queue process, it does not need any customization's to custom, contrib or core code to work.
Direct Queue has the following workflow:

1. Ask Drupal Console for DB connection details
2. Connect to the Database
3. Scan the queue table every 500 ms for new unclaimed items
4. Start new workers with an internal queue if it is the first time it saw this specific queue
5. Add items to the (new) internal queue
6. Claim the queue item in Drupal by setting the expired timestamp
7. Start up a Drupal Console command to process the item

Backends

Direct Queue does not work as is, you will need a backend that processes the queue.
We currently developed one backend in Go, it has releases for most OS'es.

• Go backend

Requirements

• A Direct Queue backend
• For 8.x-1.x Drupal Console
• For 8.x-2.x Drush

Roadmap

• More backends
• Integrated backend that can be ran from Crontab so it will work on shared webhosting environments
• Better way to give back response to the backend
• Support other DB engines in Go backend (Only MySQL is supported for now)

Contributing

Module

Create issues in the Drupal issue queue.
When you provide a patch, add tests for the case and add a patch that just contains the tests to show that you covered the issue properly.

Backend

Create issues and pull requests on Github.
Create new backends on any of the popular Git hosting sites (Github/BitBucket/Gitlab).
If you created a new backend and want it to be added to the list please create a new issue in the Drupal issue queue. Make sure the new backend contains some proper tests and has instructions on how to run the backend.