AI Agent Readiness

The AI Agent Readiness module transforms any Drupal 10 or 11 website into an AI-agent-friendly platform. It implements all emerging web agent discovery standards, LLM indexes, and content negotiation protocols validated by tools such as Cloudflare's Is Your Site Agent-Ready? scanner.

Overview

Modern Large Language Models (LLMs) and autonomous AI agents (such as OpenAI Operator, Anthropic Claude Computer Use, Google Gemini, and Perplexity) crawl the web to answer user queries, discover documentation, and execute actions. Traditional HTML pages with heavy JavaScript DOM, cookie banners, and complex layouts consume excessive LLM token context and often lead to extraction hallucinations.

AI Agent Readiness bridges this gap by providing zero-overhead, standardized markdown endpoints and machine-readable discovery protocols directly from your Drupal entity architecture.

Comparison with Existing Modules (llmstxt, llms_txt)

While existing modules such as llmstxt and llms_txt publish a static or manually typed llms.txt file, AI Agent Readiness provides a complete, dynamic ecosystem discovery engine for the modern AI agent web:

• Dynamic Entity & Paragraph Generation: Automatically generates markdown from live Drupal published nodes across multiple content types, extracting text from nested Paragraph entity reference trees and sub-components.
• Full Knowledge Guide (/llms-full.txt): Generates deep corpus documentation with memory-safe chunking and automatic cache freeing for large sites.
• Strict Node Grants & Field Access Control: Respects Drupal's access control layer (including domain_access, group, and field-level permissions) so restricted content is never leaked to anonymous agents.
• Complete Zero-Auth Discovery Protocol Suite: In addition to llms.txt, it automatically serves RFC 8288 API Catalogs, Agent Skills v0.2.0 with SHA-256 digests, SEP-2127 MCP Server Cards, RFC 9728/8414 OAuth Metadata, and WorkOS auth.md.
• Safe Content Negotiation: Implements strict RFC 9110 q-factor priority parsing for Accept: text/markdown requests and integrates with page cache kill switches to prevent HTML cache poisoning.
• Dual Monolithic & Decoupled Architecture: Features dedicated /api/ai-agent/* routes designed to bypass Nginx static file blocking rules in headless Next.js/Nuxt setups.
• Multilingual Support: Automatically resolves content translations based on the current site language.

Features & Standards Implemented

• /llms.txt Index: A concise, curated markdown index of site overview, contact information, and top content items, following the Answer.AI llms.txt standard.
• /llms-full.txt Knowledge Base: Complete, in-depth documentation guide formatted in plain markdown for comprehensive context windows, with support for configurable limits and memory-safe batch processing.
• Accept: text/markdown Content Negotiation: Safely intercepts requests from AI agents requesting markdown with higher preference than HTML without poisoning page cache.
• RFC 8288 API Catalog (/.well-known/api-catalog): Standardized linkset declaring machine-readable API and documentation endpoints.
• Agent Skills Discovery (/.well-known/agent-skills/index.json): Implements the Agent Skills v0.2.0 specification with cryptographic SHA-256 content digests for runtime verification.
• SEP-2127 MCP Server Card (/.well-known/mcp/server-card.json): Machine declaration for the Model Context Protocol (MCP) ecosystem.
• RFC 9728 OAuth Protected Resource Metadata (/.well-known/oauth-protected-resource): Standardized declaration of resource identity and public read scopes.
• RFC 8414 OAuth Authorization Server Metadata (/.well-known/oauth-authorization-server): Discovery document containing an agent_auth block for autonomous agent onboarding.
• WorkOS auth.md (/auth.md): Standard zero-auth declaration format explaining public read-only access and credentials requirements to AI crawlers.
• Dynamic Field & Paragraph Resolution: Extracts plain text from body fields, summaries, text components, and nested Paragraph entity reference trees.
• Drupal Cache Tag Invalidation: All endpoints integrate with Drupal's Dynamic Page Cache and cache tags (config:ai_agent_readiness.settings, node_list), invalidating instantly when content is updated.
• Decoupled Architecture Support: Delivers clean /api/ai-agent/* routes enabling headless frontends (Next.js, Nuxt, Remix, Astro) to proxy responses without Nginx static file conflicts.

Why Are Different Endpoints Needed?

Different AI agent frameworks look for different entrypoints during web traversal:

1. Search & Summarization Bots (Perplexity, ChatGPT, Claude) look for /llms.txt to understand site structure without downloading gigabytes of HTML.
2. Deep Research Agents look for /llms-full.txt to ingest the entire corpus into their context memory.
3. Protocol-Driven Tool Callers (MCP clients) probe /.well-known/mcp/server-card.json to determine available tools.
4. Skill-Aware Agents inspect /.well-known/agent-skills/index.json to load execution skills with cryptographic digest verification.
5. API Discoverers & Crawlers query /.well-known/api-catalog and /.well-known/oauth-protected-resource to verify authorization and entrypoint URLs.

Requirements

• Drupal 10.x or 11.x
• PHP 8.1 or higher

Installation

Install via Composer and enable via Drush:

composer require drupal/ai_agent_readiness
drush en ai_agent_readiness -y

Configuration Guide

Navigate to Administration > Configuration > Web Services > AI Agent Readiness (/admin/config/services/ai-agent-readiness).

1. General Settings

• Site Title & Description: Clear, factual summary of the organization or project for AI agents.
• Public Frontend Base URL: The canonical public URL (e.g. https://example.com). Used to generate absolute links across all discovery files.
• Contact Email: Primary contact address published in markdown headers.
• Custom Header / Footer Markdown: Optional arbitrary markdown snippets (e.g. key links, guidelines, or disclaimers) appended to llms.txt documents.

2. /llms.txt (Summary Index) Configuration

• Item Limit per Content Type: Max number of items per entity bundle (default: 10).
• Content Types Mapping: Select which bundles to include, custom URL path overrides, sort criteria (e.g. Changed date), and the summary field (supports text fields and Paragraph reference paths).

3. /llms-full.txt (Full Knowledge Guide) Configuration

• Item Limit per Content Type: Number of full records to render (default: 50). Processed in memory-safe chunks with entity cache clearance.
• Content Field Mapping: Map full text or paragraph component fields to render complete body text.

4. Discovery & Protocol Settings

• Agent Skills: Configure skill name, description, and target URL path.
• MCP Server Card: Set service name, version, and description.
• API Catalog: Configure the describedby target path.
• Agent Auth (auth.md): Declare registration requirements, authentication flags, and agent scope.
• OAuth & OpenID Discovery: Configure supported scopes (e.g. public:read) and grant types (e.g. implicit).
• Protocol Negotiation: Toggle Accept: text/markdown content negotiation.

External Requirements (Beyond Drupal)

While this module handles all Drupal backend routing, discovery documents, and content negotiation, achieving a 100% Agent-Ready score on scanners like isitagentready.com requires a few server and DNS settings:

1. DNS for AI Discovery (DNS-AID)

AI crawlers query DNS-over-HTTPS (DoH) looking for _agents entrypoint records before initiating HTTP connections. Add the following records in your DNS provider (e.g. Cloudflare):

• TXT Record: Name: _index._agents | Content: uri=https://example.com/llms.txt alpn=h2,h3
• HTTPS Record (Type 65): Name: _index._agents | Priority: 1 | Target: example.com | Params: alpn="h2,h3" port=443

2. robots.txt Content Signals

Include modern AI training and search content signals in your robots.txt:

User-agent: *
Allow: /
Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://example.com/sitemap.xml

3. HTTP Response Link Header (RFC 8288)

Ensure your web server or reverse proxy advertises discovery endpoints in the HTTP Link response header on GET /:

Link: </llms.txt>; rel="alternate"; type="text/markdown", </.well-known/api-catalog>; rel="api-catalog", </.well-known/agent-skills/index.json>; rel="agent-skills", </.well-known/mcp/server-card.json>; rel="mcp", </auth.md>; rel="author"

Monolithic vs. Decoupled Drupal Support

This module supports both architectures out of the box:

• Monolithic Drupal: Direct routes (/llms.txt, /llms-full.txt, /.well-known/*, /auth.md) serve requests directly from Drupal.
• Decoupled / Headless Frontends (Next.js, Remix, Astro): Default Nginx configurations often block direct requests for .txt or .json static files. The module provides dedicated /api/ai-agent/* routes so headless frontend reverse proxies can fetch backend markdown cleanly without requiring custom Nginx server rewrites.