CRWLR turns a website into structured Drupal content through a configurable, observable administrative workflow.

Define where a crawl starts, decide which URLs should be followed and which pages should become content, map CSS or XPath selections to Drupal fields, and run the import through Drupal's Batch and Queue APIs. Created content remains ordinary Drupal nodes, ready for Views, Search API, revisions, moderation, and the rest of the Drupal ecosystem.

Features

Five-step administrative workflow: Define, Discover, Map, Run, and Review.
Entry URL and sitemap seeding with domain scope, depth, URL rules, and harvest rules.
CSS and XPath extraction into fields on existing Drupal content types.
Optional processors for dates, strings, URLs, HTML element removal, and absolute links inside saved HTML.
Browser-correct HTTP redirects, relative URL resolution, fragments, and canonical URL identity.
Default-on robots.txt compliance and configurable adaptive throttling.
Foreground Batch API processing with bounded handoff to cron for large crawls.
Per-run history, URL filtering, HTTP diagnostics, and discovery provenance.
Protection for locked or locally edited content, with explicit review decisions.
Target-scoped stop and batched hard-reset operations for recovery from a bad crawl.

How it works

Create or choose the destination content type and fields.
Define a crawl target, its seeds, scope, scheduling, and safety settings.
Use the bounded discovery preview to validate follow and harvest rules.
Map source selectors to Drupal fields and preview extracted values.
Run the crawl and inspect created, updated, skipped, or failed URLs.

Small crawls complete in the foreground. Larger crawls hand off to Drupal cron while retaining an active run record. CRWLR records the effective URL after redirects, the source page that discovered each URL, and concise HTTP diagnostics for investigation.

Requirements

Drupal 10.2 or Drupal 11.
PHP 8.3 or later with the DOM extension.
Working Drupal cron for background processing.
Composer installation of the dependencies declared by the module.

Responsible crawling

CRWLR enables robots.txt compliance by default and provides configurable request throttling. Site owners remain responsible for authorization, source-site terms, privacy, copyright, and applicable law. Administrative permissions should be granted only to trusted roles because crawl targets can create, update, and—in a confirmed hard reset—delete target-owned content.

Independent integration and upstream credit

CRWLR is an independent Drupal integration for the open-source crwlr/crawler PHP library created by Christian Olear and contributors. We are grateful for their work. This Drupal module is maintained separately and is not an official module from, affiliated with, sponsored by, or endorsed by the library author or project. The upstream library may be used directly without CRWLR.

Installation

composer require drupal/crwlr
drush en crwlr -y
drush cr

See the project README and the documentation in the docs/ directory for configuration, operations, and development guidance.

CRWLR