Migrate Source Queue

Allows queue items to be used as a migrate source.

Advantages

• Increase performance of your custom API endpoints or import forms by creating queue items for entity create/update operations instead of doing them immediately.
• Automatically retry failed migrate rows, stop retrying after X tries using the Dead Letter Queue module.

How to use it?

In the migration file, add the plugin provided by this module as source plugin:

source:
  plugin: queue

  # The queue from which queue items will be processed.
  queue_name: users

  # The queue item array keys defined here will be made available in the migration.
  # Can also be an associative array with labels as values.
  fields:
    - id
    - firstName
    - lastName
    - address
    - email
    - telephoneNumber

  # The primary key as a list of keys. It's a list of source columns that are
  # composing the primary key. The list is keyed by column name and has the
  # field storage definition as value.
  keys:
    id:
      type: string

This plugin provides the following source fields:

• data: The same as what passed into createItem().
• item_id: The unique ID returned from createItem().
• created: Timestamp when the item was put into the queue.

All this plugin does is process queue items, you still have to create the queue items yourself. Note that the queue items have to be arrays.

Cron

This plugin defines a migration source, not a queue worker, because that would be less efficient. This means that the queue items will not automatically process during cron runs. If you need that functionality, please refer to the migrate_source_queue_cron_example example module.