Queues in Drupal 8 and 9

A queue is simply a list of stuff that gets worked through one by one, one analogy could be a conveyor belt on a till in a supermarket, the cashier works through each item on the belt to scan them.

Queues are handy in Drupal for chunking up large operations, like sending emails to many people. By using a queue, you are trying to avoid overloading the servers resources which could cause the site to go offline until the resources on the server are free'd up.

First we need to create a new folder for our module in the modules/custom folder called for example custom_queue, then just like in the other tutorials previous to this one, we need to tell Drupal that we have added a new module by adding a file inside the custom_queue module called custom_queue.info.yml. The contents of this file should look a bit like the below, note I have set a dependency for the queue_ui module, so I wouldn't enable the module yet if you have downloaded that module

name: 'Custom queues'
type: module
description: 'The description.'
package: Custom
core: 8.x
core_version_requirement: ^8 || ^9
dependencies:
  - drupal:queue_ui

There are a few different ways of adding items to the queue. For example we can use nod insert/ update node hooks to add nodes to the queue on creation and update.

Below is some code that you can use to add items to the queue that can be dropped into your logic/ hook of choice. The code grabs the queue by the name that will be used in our queue worker.

Then create a new object and add your required properties and assign the values as needed then call the createItem function and pass the object you just created as a parameter to the function.

$queue = \Drupal::service('queue')->get('custom_queue');
$item = new \stdClass();
$item->nid = $node_id;
$item->update = $update;
$queue->createItem($item);

First of all, what are queue workers, they are designed to work through each item in the queue that the worker is assigned to and applies the supplied logic to each item as needed. The queue worker generally processes items on cron, but you can use drush to work through the queue items.

Now inside your module, create a src folder with a folder called Plugin inside, then inside that folder create another folder called QueueWorker. Now create a new php file that will contain your queue worker php class. In our example, the class would be called CustomQueue.php.

The full class is below, but I want to first draw attention to an annotation towards the top of the class, this tells Drupal that we are adding a queue worker with the item of custom_queue, the human name of Custom Queue and that on cron, the queue worker should process as many items as it can in 60 seconds. If there are still items to process, then when the cron job runs again, the queue worker will process some more and it will keep doing this until all items have been processed.

* @QueueWorker(
*   id = "custom_queue",
*   title = @Translation("Custom Queue"),
*   cron = {"time" = 60}
* )

In my below example queue worker, I am bringing in the Entity Type Manager and Database Connection functionality into the class via dependency injection. But you should bring in the functionality as needed.

Finally add a function for processItem, which takes one parameter which is the queue item. You can then access each property of the object and process the queue item as you need too. You don't need to worry about removing the item from the queue once finished, this will be done for you.

<?php

namespace Drupal\custom_queue\Plugin\QueueWorker;

use Drupal\Core\Annotation\QueueWorker;
use Drupal\Core\Database\Connection;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Queue\QueueWorkerBase;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
* Custom Queue Worker.
*
* @QueueWorker(
*   id = "custom_queue",
*   title = @Translation("Custom Queue"),
*   cron = {"time" = 60}
* )
*/
final class CustomQueue extends QueueWorkerBase implements ContainerFactoryPluginInterface {

/**
* The entity type manager.
*
* @var \Drupal\Core\Entity\EntityTypeManagerInterface
*/
protected $entityTypeManager;

/**
* The database connection.
*
* @var \Drupal\Core\Database\Connection
*/
protected $database;

/**
* Main constructor.
*
* @param array $configuration
*   Configuration array.
* @param mixed $plugin_id
*   The plugin id.
* @param mixed $plugin_definition
*   The plugin definition.
* @param \Drupal\Core\Entity\EntityTypeManagerInterface $entity_type_manager
*   The entity type manager.
* @param \Drupal\Core\Database\Connection $database
*   The connection to the database.
*/
public function __construct(array $configuration, $plugin_id, $plugin_definition, EntityTypeManagerInterface $entity_type_manager, Connection $database) {
parent::__construct($configuration, $plugin_id, $plugin_definition);
$this->entityTypeManager = $entity_type_manager;
$this->database = $database;
}

/**
* Used to grab functionality from the container.
*
* @param \Symfony\Component\DependencyInjection\ContainerInterface $container
*   The container.
* @param array $configuration
*   Configuration array.
* @param mixed $plugin_id
*   The plugin id.
* @param mixed $plugin_definition
*   The plugin definition.
*
* @return static
*/
public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
return new static(
$configuration,
$plugin_id,
$plugin_definition,
$container->get('entity_type.manager'),
$container->get('database'),
);
}

/**
* Processes an item in the queue.
*
* @param mixed $data
*   The queue item data.
*
* @throws \Drupal\Component\Plugin\Exception\InvalidPluginDefinitionException
* @throws \Drupal\Component\Plugin\Exception\PluginNotFoundException
* @throws \Drupal\Core\Entity\EntityStorageException
* @throws \Exception
*/
public function processItem($data) {
$nid = $data->nid;
$update = $data->update;

// Processing of queue items logic goes here.
}

}

There is a useful module to being able to see how many items are in the queue and be able to view the contents of each queue item. This module is usefully known as the Queue UI module. The project page is available at: https://www.drupal.org/project/queue_ui.

You can then use composer to add the module to your site, using a command along the lines of the below.

composer require 'drupal/queue_ui:^2.2'

Once the module has been added, you can go to enable your custom queue worker module, note that when you enable the module, it should also enable the queue ui module. The site will notify you that there are other modules to enable, you simply agree to this and Drupal will enable the queue ui module for you.

Once the module is enabled, you should then be able to access the Queue Manager page, as shown in the screen shot below. This page should be listing your custom queue and if you have added items, then you should see the number items represented.

You can then click the Inspect button for your queue, which will list the items on a screen that may look similar to the below screenshot. This page allows you to view the contents of each of the items in the queue.