Agent Surface

Agent Surface exposes what your site publishes to anonymous visitors in the format machine consumers prefer, and declares in machine-readable form how that content may be used.

Two halves, deliberately separable:

• Representation. A Markdown alternate for every public route (/recipes/oatmeal.md), with YAML front matter, stable validators and empty 304 Not Modified answers for repeat fetches. The HTML-to-Markdown conversion is Markdownify's; this module adds the route coverage, the selection rules, the caching discipline and the conditional requests.
• Declaration. AI usage preferences (training, inference, search) emitted through every discovery channel that exists today: response headers, front matter, robots.txt, llms.txt and, optionally, an RSL 1.0 licence.

What it does, and does not, promise

Google's Search Central documentation states that Search does not use Markdown alternates, llms.txt or any new markup. This module will not improve your rankings. What it delivers is measurable: a much smaller body for the agents that ask for one, a 304 instead of a re-download when nothing changed, and a correct, consistent statement of what those agents may do with what they fetch. The consumers that read these surfaces today are coding and IDE agents (Claude Code, Cursor, Copilot and others send Accept: text/markdown) and answer engines fetching pages live.

Features

• Markdown for every public page. Entities through Markdownify; Views pages, custom controllers and everything else through the agent_surface_fallback submodule, which replays the original route so a page that is 403 or 404 as HTML is 403 or 404 as Markdown. Fields hidden by the view mode stay hidden.
• Four ways in, with clear cache rules. A .md suffix and ?_format=markdown are cacheable. Accept negotiation and User-Agent matching are served private, no-store, so a Markdown body never sits in a shared cache under an HTML URL. Accept: */* never means Markdown. No Vary: User-Agent is ever emitted.
• Conditional requests that actually work. A strong ETag derived from the body survives cache rebuilds and deploys, so crawlers keep receiving 304s instead of re-downloading the site. Last-Modified comes from the entity's changed time. Estimated token counts are sent as X-Markdown-Tokens and X-Original-Tokens.
• Declarations in every channel. Content-Usage (IETF AIPREF) and Content-Signal (Content Signals) headers on HTML and Markdown responses; the same directives in robots.txt via the RobotsTxt module, path-scoped per bundle where you want them; the same values as front matter keys, because a document copied into a context window keeps its front matter and loses its headers.
• Vocabularies as plugins. The specifications are still moving (AIPREF has changed spelling three times). Each vocabulary is a serializer plugin; a revision is a new plugin, and both spellings can be emitted during a transition. Nothing outside a plugin references a specification's literal strings.
• Per-bundle configuration where site builders already are. Site-wide defaults, overridden from each content type, vocabulary or media type edit form. A read-only audit page shows every bundle, its effective signals and the exact strings being emitted.
• Discovery. <link rel="alternate" type="text/markdown"> and the matching HTTP Link header on every page that has a Markdown twin; translations cross-reference each other in front matter, the Markdown analogue of hreflang.
• A request log and report. Every Markdown request and every matched agent is recorded (no IP addresses, cookies or session identifiers) and aggregated by agent, by day and by selection rule, with the 304 ratio that tells you whether you are reducing load or merely moving it.
• Privacy by default. User profiles are never exposed as Markdown unless you switch them on: a text file at a predictable path removes the friction that makes harvesting /user/* inconvenient.

Read this before enabling it

The shipped defaults declare your content available for AI training, AI inference and search, on every page and in robots.txt. Permissive declarations have legal weight in several jurisdictions. The status report carries a warning until an administrator reviews and saves the settings form, which records the choice. "Unstated" is a valid choice for every category: under every current vocabulary, saying nothing is different from saying yes.

Submodules

• agent_surface_fallback (recommended): a Markdown twin for every route that renders HTML. An interim implementation of Markdownify #3551170.
• agent_surface_rsl: an RSL 1.0 licence at /license.xml derived from the declared signals, advertised in robots.txt, a Link header, the HTML head and front matter.
• agent_surface_llmstxt: appends per-content-type listings of Markdown alternates to the llms.txt module's file.
• agent_surface_filecache: precomputes Markdown responses to disk with Response Filesystem Cache, invalidated by cache tags.

Requirements

• Drupal 10.3 or 11, PHP 8.1 or later.
• Markdownify 1.2 or later. Enable its Path submodule for .md on aliases and its Metadata submodule for rich front matter.
• RobotsTxt to write directives to robots.txt. Without it, headers and front matter still carry the declarations.

composer require drupal/agent_surface, then enable agent_surface and the submodules you want. Configuration lives at Configuration › Web services › Agent Surface.

Hosting notes

On Pantheon, exclude robots.txt from the composer scaffold or the RobotsTxt module never gets to answer, and remember that platform domains serve Pantheon's own crawler-blocking robots.txt: under AIPREF, preferences apply only to crawlable paths, so test the declarations on Live with a custom domain.

Status

Under active development; no release yet. Security advisory coverage will be sought, which is why the main module depends only on covered projects and everything else is a submodule.

Maintainer

tbcs, FivePaths, LLC.