llms_txt

Provides an /llms.txt endpoint for your Drupal site to help large language models and AI agents better understand your website content.

What is llms.txt?

The llms.txt standard is an emerging convention designed to help large language models better understand website content during inference. It works by providing clean, well-organized information in a standardized format. For comprehensive details about this standard, visit llmstxt.org.

In essence, the llms.txt file serves as your homepage for LLMs, AI agents, and other bots, directing them to the content required to understand and properly use your website.

Features

• Flexible content management: Configure content through both code-based config entities and environment-specific database entities
• Permission-based access: Users with "Administer /llms.txt configuration" permission can manage the endpoint content
• Custom tokens: Embed dedicated menus in machine-readable Markdown format within your llms.txt
• Optimized caching: Built-in render caching integration with proper invalidation logic
• Environment flexibility: Balance between generic llms.txt parts (stored in code) and environment-specific content (stored in database)

Understanding configuration vs. content

To maintain a clean and deployable llms.txt file, the module distinguishes between global configuration and environment-specific content:

• Configuration (/admin/content/llms-txt): Use this for the structural skeleton of your llms.txt file. Changes made here are exported as Drupal configuration, making them ideal for standardizing the file structure across all environments (Dev, Test, Prod) via your deployment pipeline.
  • Best practice: Keep the content here generic. Use tokens to inject dynamic values that change based on the environment.
• Content (/admin/content/llms-txt/sections): Use this for additions that are unique to a specific instance and should not be synchronized across environments. These are stored as content entities in the database.
  • Use case: Use this for site-specific additions that cannot be easily expressed through tokens.

Installation and setup

1. Install this module following standard Drupal module installation procedures
2. Configure your web server (see Web server configuration below)
3. Navigate to Administration » Content » llms.txt to configure and author your site's llms.txt content

Web server configuration

Important: Before using the module, ensure your web server is configured to serve the /llms.txt file directly. For some development stack-specific instructions, check the DEVELOPMENT.md file.

Nginx configuration

Add this to your Nginx configuration:

# Enable support for the llms_txt module
location = /llms.txt {
  access_log off;
  try_files $uri @drupal;
}

Apache configuration

For Apache users, refer to your server's documentation to enable direct access to /llms.txt.

Comparison with llmstxt module

This module differs from the llmstxt module (version 1.0.0-alpha1) in several key ways:

Feature llms_txt llmstxt Content storage ✅ Environment-specific content stored in database others in config ❌ Config-only approach (Config Ignore might be necessary) Render caching ✅ Maximized with proper invalidation ⚠️ Basic caching Test coverage ✅ Higher than 90% ❌ None Programming API ❌ No alteration API ✅ Provides alteration hooks