ai_llms_txt_generator

A Drupal module that automatically generates LLMs.txt files using AI based on your sitemap.xml content.

Features

• AI-Powered Generation: Uses the Drupal AI module to generate contextually relevant LLMs.txt content
• Sitemap Integration: Parses your sitemap.xml to understand site structure
• Multiple AI Providers: Support for OpenAI, Anthropic, and other AI providers via the AI module
• Manual Override: Ability to manually edit generated content
• LLMs.txt Spec Compliant: Follows the specification from https://llmstxt.org/
• Optimized Prompts: Generates concise, AI-friendly content with actual URLs
• AJAX Interface: Real-time content generation without page reloads

Requirements

• Drupal 10 or 11
• AI module (drupal/ai)
• Simple XML Sitemap module (drupal/simple_sitemap) or any sitemap generator
• Configured AI provider (OpenAI, Anthropic, etc.)

Installation

1. Install the module:

   composer require drupal/ai_llms_txt_generator
   drush en ai_llms_txt_generator
   

2. Configure your AI provider:

   • Go to /admin/config/ai
   • Configure your preferred AI provider (OpenAI, Anthropic, etc.)
   • Set a default provider for the 'chat' operation type
   • Add your API keys

3. Configure the LLMs.txt generator:

   • Go to /admin/config/search/ai-llms-txt
   • Enter your sitemap URL
   • Enter site name and description (optional)
   • Click "Generate LLMs.txt with AI"

Usage

Accessing the LLMs.txt file

The generated file will be available at:

https://your-site.com/llms.txt

Configuration Options

• Site Information: Name and description used in generation
• Sitemap URL: Full URL to your sitemap.xml
• Generated Content: AI-generated content (editable via AJAX)
• Manual Content: Additional content to append

Content Generation

1. Click "Generate LLMs.txt with AI" button
2. Wait for AI to process (loading indicator shown)
3. Content appears immediately in the textarea
4. Edit manually if needed
5. Click "Save configuration" to persist changes

API Usage

Programmatic Generation

$sitemap_parser = \Drupal::service('ai_llms_txt_generator.sitemap_parser');
$ai_generator = \Drupal::service('ai_llms_txt_generator.ai_generator');

// Parse sitemap.
$urls = $sitemap_parser->parseSitemap('https://example.com/sitemap.xml');
$sitemap_data = $sitemap_parser->formatForAi($urls);

// Generate content.
$content = $ai_generator->generateContent($sitemap_data);

Service Injection

use Drupal\ai_llms_txt_generator\Service\AiGeneratorService;
use Drupal\ai_llms_txt_generator\Service\SitemapParserService;

public function __construct(
  SitemapParserService $sitemap_parser,
  AiGeneratorService $ai_generator
) {
  $this->sitemapParser = $sitemap_parser;
  $this->aiGenerator = $ai_generator;
}

public static function create(ContainerInterface $container) {
  return new static(
    $container->get('ai_llms_txt_generator.sitemap_parser'),
    $container->get('ai_llms_txt_generator.ai_generator')
  );
}

How It Works

1. Sitemap Parsing: The module fetches and parses your sitemap.xml
2. Priority Grouping: URLs are organized by priority (high/medium/low)
3. AI Prompt Building: Creates an optimized prompt with sitemap data
4. AI Generation: Calls the configured AI provider to generate content
5. Content Cleaning: Removes code blocks and formats the output
6. Content Serving: The generated content is served at /llms.txt

Generated Content Format

The AI generates LLMs.txt content following this structure:

# Site Name
> One-sentence description

## Overview
Brief site overview (2-3 sentences)

## Key Pages
- [Page Title](URL) - Description
- [Another Page](URL) - Description

## Main Topics
### Category
Description of content area

## Site Structure
/section/ - Section description

Drupal Coding Standards

This module follows Drupal coding standards:

• Full dependency injection in all classes
• Proper PHPDoc comments
• Type hints for all parameters and return values
• Service definitions in services.yml
• Configuration schema definitions
• PSR-4 autoloading
• Proper error handling and logging

Troubleshooting

Content not generating

• Verify AI module is installed and configured
• Check that a default provider is set for 'chat' operation type
• Verify API keys are valid
• Ensure sitemap URL is accessible
• Review logs at /admin/reports/dblog

Sitemap not parsing

• Verify sitemap URL is correct and accessible
• Check sitemap XML is valid
• Ensure sitemap follows standard format
• Check for sitemap index vs regular sitemap

AJAX not updating

• Clear Drupal cache: drush cr
• Check browser console for JavaScript errors
• Verify AJAX wrapper ID matches in form

Technical Details

Services

• ai_llms_txt_generator.sitemap_parser: Parses sitemap.xml files
• ai_llms_txt_generator.ai_generator: Generates content using AI

Controllers

• LlmsTxtController: Serves the /llms.txt endpoint
• GenerateController: Handles POST requests for generation

Configuration

• Stored in: ai_llms_txt_generator.settings
• Schema defined in: config/schema/ai_llms_txt_generator.schema.yml

Support

For issues and feature requests, please use the issue queue on Drupal.org.

License

GPL-2.0+