LLM Offramp

INTRODUCTION

AI crawlers walk faceted overviews combinatorially: every filter combination is another URL, every URL is another search query, and the same records keep coming back. This module gives them a cheaper route.

Pick a views display, and the module adds a JSON display next to it that returns the same records. That feed is then advertised through llms.txt, a Link response header and a rel="alternate" link element. Optionally, AI crawlers requesting a filtered URL are redirected to the feed.

Bear in mind that llms.txt is advisory and its adoption cannot be verified. The redirect is the part that actually saves server load.

REQUIREMENTS

Views, which ships with core.

The Facets module is optional. When it is installed, the faceted fields are added to every JSON record and a manifest of all filter values is published as well. Everything else works without it.

INSTALLATION

Install as you would normally install a contributed Drupal module. See Installing modules.

CONFIGURATION

1. Go to Configuration > Search and metadata > LLM offramp (/admin/config/search/llm-offramp) and add a source. Pick the views display that crawlers should be steered away from. The module adds an "LLM JSON feed" display to that view.
2. Check the generated display in the Views UI through the "Edit JSON display" operation, then export the configuration of that one view. The display is stored in views.view.[id], so it needs to be exported like any other views change.
3. On the Settings tab, choose how crawlers are handled:
   • Do nothing.
   • Advertise only (default): add the feed to the response headers.
   • Redirect filtered crawler requests to the feed (308).
   • Answer filtered crawler requests with 410 Gone.

Only anonymous requests that both match a crawler user agent and carry filter parameters are ever redirected or blocked. Start with "advertise only" and watch your access logs before enforcing anything.

The user agent patterns, the query keys that count as filtering, and the cache lifetime of the feed are configurable on the same tab.

THE JSON

{
  "title": "Vacancies",
  "description": "All open vacancies.",
  "generated": "2026-08-31T09:00:00+00:00",
  "page": 0,
  "per_page": 200,
  "total": 1284,
  "pages": 7,
  "next": "https://example.com/llms/vacancies.json?page=1",
  "html_version": "https://example.com/vacancies",
  "filterable_fields": ["job_group", "region"],
  "filter_values": "https://example.com/llm-offramp/jobs__overview/facets.json",
  "usage": {
    "paging": "Add ?page=N to this URL, counting from 0, or follow the \"next\" link until it is absent.",
    "filtering": "Every item carries its own job_group, region value, so filter this list locally.",
    "query_parameters": {"search": "Keyword (single value)"}
  },
  "items": [
    {
      "id": "42",
      "entity_type": "node",
      "label": "Nurse",
      "bundle": "job_offer",
      "langcode": "nl",
      "url": "https://example.com/vacancy/nurse-ghent",
      "job_group": "Care",
      "region": ["East Flanders"]
    }
  ]
}

The usage block is there because an LLM reading the feed has no other documentation, so the paging and filtering rules travel with the data. query_parameters lists the exposed filters of the display, which is the only server side filtering that works on the feed.

OTHER LLMS.TXT MODULES

Both llms_txt and llmstxt serve /llms.txt themselves, so this module stays out of the way when one of them is installed:

• Neither installed: this module serves /llms.txt.
• llmstxt: it serves the file, and the sections are added through hook_llmstxt() without any configuration.
• llms_txt: it serves the file. Paste [llm_offramp:sources] into its content field, or [llm_offramp:document] for a complete document.

This can be overridden per site on the Settings tab.

NOTES

• The generated display inherits its filters, sorts, arguments and fields from the same place the source display inherits them from, so editing the human display changes the feed as well. Where the source display overrides a section, the feed gets its own copy, and the sources listing reports when the two have drifted apart. The "Rebuild JSON display" operation repairs it.
• The faceted fields are read off the entity while rendering a row, instead of being added to the display's field list. That keeps the field list inheriting.
• A field with a broken or missing handler on the source display (usually a field that a Search API index stopped indexing) is inherited by the feed as well. Such fields are skipped instead of returning an empty value, and the sources listing, the source form and the status report all name the handler that needs fixing.
• This is not a REST export display. That would need the rest module, and its route requires a ?_format=json query argument, which is not a URL you want to hand to a crawler.
• Crawler handling runs after Drupal's internal page cache. That cache keys on the URL only and ignores cache contexts, so a response that depends on the user agent may never be stored in it: the redirect triggers the page cache kill switch. A crawler requesting a URL that happens to be cached already is therefore served that cached page instead of the redirect, which costs virtually nothing. The expensive requests are the cache misses.

EXTENDING

hook_llm_offramp_llms_txt_alter() alters the whole llms.txt structure before it is rendered. See llm_offramp.api.php.