migrate_batch

A Drupal module that provides batch migration processing with automatic offset tracking.

Features

This module extends Drupal's migration system by providing commands to process migration items in configurable batches with automatic progress tracking. Unlike standard drush migrate:import --limit, these commands maintain state between runs and can cycle through all source items continuously.

Usage

You can use the migrate_batch service directly in your custom modules, hooks, or other Drupal code:

/** @var \Drupal\migrate_batch\Service\MigrateBatchService $batch */
$batch = \Drupal::service('migrate_batch');

// Process a batch of default item amount (20).
$batch->next('my_migration');

// Process next batch of 50 items.
$batch->next('my_migration', 50);

// Process 50 items starting from offset 100.
$batch->next('my_migration', 50, 100);

// Check current offset.
$offset = $batch->getOffset('my_migration');
echo "Current offset: $offset";

// Set offset to a specific value.
$batch->setOffset('my_migration', 100);

// Reset offset back to 0.
$batch->resetOffset('my_migration');

Note: The service automatically tracks progress using Drupal's State API. Each call to next() processes the next batch and advances the offset.

Integration

This module works with any Drupal migration. For optimal performance with large datasets, source plugins should use the BatchableSourceTrait to support batch processing:

Source Plugin Integration

The BatchableSourceTrait provides three key methods for batch processing:

• isBatchRequest(): Returns TRUE when the migration is running in batch mode
• getBatchLimit(): Returns the number of items to process in this batch
• getBatchOffset(): Returns the starting position for this batch

Use these methods in your source plugin's initializeIterator() method to apply LIMIT and OFFSET to your data retrieval

For more details, see the examples in the README.

Drush Commands

drush migrate:batch-next (alias: mbn)
Main command for processing the next batch of migration items.

drush migrate:batch-offset (alias: mbo)
Check the current batch offset for a migration.

drush migrate:batch-offset:set (alias: mbos)
Set the batch offset for a migration to a specific value.

drush migrate:batch-offset:reset (alias: mbor)
Reset the batch offset for a migration back to 0.

Additional Requirements

• drupal:migrate - provides the Migrate API

Recommended modules/libraries

Similar projects

Supporting this Module

Community Documentation