Pay-per-Crawl

Pay-per-Crawl answers HTTP 402 Payment Required to AI crawlers, SEO crawlers and scripts, and serves the page only after they pay in USDC on Base using the x402 protocol (or MPP). Human visitors and reverse-DNS-verified search engines (Googlebot, Bingbot, DuckDuckBot, Applebot…) are never affected, so SEO is untouched.

How it works
- A bot requests a page and receives a 402 carrying the price, the asset (USDC on Base) and your wallet, in the x402 v2 PAYMENT-REQUIRED header, a v1 JSON body, and an MPP WWW-Authenticate challenge.
- The bot signs a USDC transfer authorization and retries. The module sends it to an x402 facilitator (PayAI by default, no account or API key; any facilitator URL can be configured), which verifies and submits it on-chain. The page is served only when a transaction hash comes back.
- The owner keeps 100%: payments go straight from the payer to the configured wallet. The module never holds funds.

Features
- Monitor mode: log every bot and what it would have paid before charging anything.
- Policy per crawler class (search, social previews, uptime monitors, AI crawlers, SEO crawlers, scripts, other bots) with per-bot overrides and allow-listed paths.
- Report page: who is crawling, how often, from how many sources, which paths, what was paid, every settlement with its transaction hash.
- Discovery files served on your domain: /.well-known/x402.json, /llms.txt, /agents.txt.
- Gate runs as an HTTP middleware above the page cache, so cached pages are gated too.
- Optional hosted dashboard, weekly digest and paywall monitoring through a paid ForgeMesh Pro license; everything above works without it.

Requirements: Drupal 10.1+ or 11, PHP 8.1+. Depends on forgemesh/pay-per-crawl-core (Composer).

Install: composer require forgemesh/pay-per-crawl-drupal, enable "Pay-per-Crawl", then configure at /admin/config/services/pay-per-crawl.

Source: https://github.com/forgemeshlabs/pay-per-crawl-drupal (GPL-2.0-or-later). The same gate is available for WordPress, Laravel, Node.js, Cloudflare Workers, Go (Caddy, Traefik) and Python.

Note: if your site is behind Cloudflare, set AI Crawl Control to Allow; crawlers that Cloudflare blocks never get the chance to pay. If a proxy sets a client-IP header, configure the header the proxy itself writes (CF-Connecting-IP, X-Real-IP, True-Client-IP).