Document Loader: HTML Processor

What it does

Accepts an HTML string and runs it through the html_processor module's pipeline. Returns clean, narrowed HTML ready for AI systems, search indexes, or data migration.

• Extracts specific content regions using CSS selectors (e.g. main, article, .content-body)
• Strips unwanted markup via regex patterns (scripts, comments, tracking pixels)
• Removes advertising elements — Google AdSense, DoubleClick, Taboola, Outbrain, Media.net, or custom patterns
• Rewrites relative href and src attributes to absolute URLs using a caller-supplied base URL
• Sanitizes HTML with Symfony HtmlSanitizer — configurable element and attribute allowlists, or use the safe/static shorthands
• Wraps output in a full HTML document structure when needed
• Minifies whitespace to reduce token count for AI pipelines

The base module processes HTML you already have in memory. For fetching HTML directly from a URL, see the URL submodule below.

URL submodule (opt-in)

document_loader_html_processor_url ships inside this package and is disabled by default. Enable it when you need the server to fetch a URL and run the returned HTML through the pipeline.

• Fetches a URL with a single GET request via Drupal's http_client
• Auto-injects the fetched URL as base_url so relative links resolve without extra configuration
• Guards: scheme allowlist (http/https only), content-type check, 10 MB response cap, bounded streaming read, redirect protocol pinning, and credential-safe error messages

See the URL submodule's README.md for full security considerations and known SSRF limitations.

Use cases

• Prepare web content for ingestion into an AI or RAG pipeline
• Normalize HTML before search indexing
• Extract and sanitize article bodies from scraped or migrated HTML
• Strip ads and tracking from content before storage or republication

Requirements

• Document Loader ^2.0
• HTML Processor ^1.0
• Drupal 10.4 or 11+
• PHP 8.1+

This module's major version tracks Document Loader's major version — 2.x of this module requires Document Loader 2.x, 3.x will require Document Loader 3.x, and so on.

Recommended modules

• document_loader_html_processor_url — included in this package; enable for URL fetching

Installation

composer require drupal/html_processor drupal/document_loader_html_processor

Enable at Administration > Extend. To also enable URL fetching:

drush en document_loader_html_processor_url

Configuration

The plugin appears at Administration > Configuration > Media > Document Loader. There is no per-plugin admin form — all processing options are passed programmatically.

use Drupal\document_loader_html_processor\DocumentLoaderType\Input\HtmlContentInput;

$loader = \Drupal::service('plugin.manager.document_loader')
  ->createInstance('document_loader_html_processor.html_processor');

$input = new HtmlContentInput($html, [
  'container'  => 'article',
  'remove_ads' => TRUE,
  'sanitizer'  => 'safe',
  'base_url'   => 'https://example.com/page',
]);

$processed = $loader->load($input, 'html')->getContent();

See README.md for the full options reference.

Good to know

• No URL fetching in the base module. Pass resolved HTML via HtmlContentInput. Enable the URL submodule only when you want this module to make outbound HTTP requests.
• No admin UI. Configuration is entirely code-driven — there are no forms to fill in.
• SSRF limitation. The URL submodule does not filter private or internal IP ranges. Mitigate at the infrastructure layer (egress firewall, forward proxy) before exposing it to untrusted callers. See the URL submodule's README.md for details.
• Versioning follows Document Loader. The major version of this module always matches the required major version of Document Loader.
• Security advisory coverage. This project is not covered by Drupal's security advisory policy.

Similar projects

• Document Loader Webpage — Readability extraction and Markdown output, not just HTML cleaning
• HTML Processor — the standalone service module this bridge delegates to; use it directly if you do not need the Document Loader plugin layer