opensolr_turbo_indexer

Overview

OpenSolr Turbo Indexer dramatically accelerates Solr indexing by using multiple parallel worker processes. Instead of indexing documents one batch at a time, it spawns independent workers that index
simultaneously, achieving 3-4x faster indexing speeds.

The module uses pcntl_fork() and pcntl_exec() to spawn completely independent PHP processes. Each worker bootstraps a fresh Drupal instance, processes a batch, and exits cleanly. The
master process monitors workers and respawns them as they complete.

Key Features

• True Parallel Processing - Multiple workers index simultaneously using forked processes
• Dynamic Worker Management - Automatically maintains your requested worker count, respawning as batches complete
• Background Mode - Run as a daemon that survives SSH disconnection
• Live Progress Display - Real-time progress bars with indexing rate and ETA
• Multi-Server Support - Target specific Search API servers when you have multiple Solr backends
• Server-Wide Reindex - Queue all items for reindexing across all indexes on a server with a single command
• Smart Stats - Stats command auto-detects which server the background indexer is running against
• Tracker Rebuild - Rebuild trackers for all indexes on a server when the tracker is out of sync

Commands

Command Alias Description opensolr:turbo ost Start parallel indexing opensolr:stats oss Show indexing statistics opensolr:stop osstop Stop all workers opensolr:reindex osri Queue all items for reindexing on a server opensolr:rebuild osrb Rebuild trackers for a server

Quick Start

    # Queue all items on a server for reindexing
    drush osri --server=my_solr_server

    # Start parallel indexing
    drush ost --server=my_solr_server --workers=8 --batch=500

    # Run in background (survives SSH disconnect)
    drush ost --server=my_solr_server --workers=8 --batch=500 --background

    # Check progress (auto-detects the server being indexed)
    drush oss

    # Stop indexing
    drush osstop
    

Typical Workflow

    # 1. Queue all items for reindexing
    drush osri --server=my_solr_server

    # 2. Start parallel indexing in background
    drush ost --server=my_solr_server --workers=10 --batch=500 --background

    # 3. Monitor progress
    drush oss

    # 4. Watch detailed log (optional)
    tail -f /tmp/opensolr_turbo.log
    

osri vs osrb

• drush osri (reindex) - Queues all tracked items as "needs indexing" without rebuilding the tracker. Fast, safe, and handles all indexes on a server in one command. Use this for routine reindexing.
• drush osrb (rebuild) - Drops and re-scans all content to rebuild the tracker from scratch. Only needed when the tracker is out of sync (incorrect counts, missing items, after migrations or backup restores).

Why Parallel Works

Solr indexing is idempotent - sending the same document twice simply overwrites it. This means occasional overlap between workers is harmless, allowing us to skip expensive locking mechanisms and run at full
speed.

Performance

Indexing speed varies by content complexity:

• Taxonomy terms: ~300 items/second
• Nodes: ~100-150 items/second

With 8 workers on a typical server, you can index hundreds of thousands of items in minutes rather than hours.

Requirements

• Drupal 10 or 11
• Search API
• Search API Solr
• PHP pcntl extension
• PHP posix extension
• Drush (CLI access)

Documentation

Full documentation available at: https://opensolr.com/opensolr-drush/