robots_rerouter

Robots Rerouter dynamically serves robots.txt based on environment. Prevents search engine indexing of non-production sites with minimal config.

Features

Robots Rerouter serves a different robots.txt response depending on the current domain.

✅ Serves a custom robots.txt for production
🚫 Serves a disallow-all fallback for non-production environments (QA, Stage, Dev)
🔐 Prevents accidental indexing of staging/test sites by search engines
🛠 Admin config form allows you to:
- Define the production hostname (e.g. www.example.com)
- Specify file paths for both production and fallback robots.txt

🧩 Smart behavior:
- Automatically creates the folders and files under public:// if they don’t exist
- Ensures secure, sandboxed file handling even on hosted environments like Acquia

Use Cases:
- You're managing multiple environments and want only the live site indexed
- You've seen Google index staging/dev accidentally before and need a safety net
- You're using CI/CD or Composer-based deployments and want to avoid .htaccess hacks

Post-Installation

After enabling the module:

1. Visit:
/admin/config/search/robots-rerouter

2. Configure:
- Production Hostname (e.g., www.example.com)
- Production File Path: e.g., robots/good_robots.txt
- Fallback File Path: e.g., robots/nocrawl_robots.txt

3. On save:
- Any missing folders or files will be automatically created in public://

Then when a user visits /robots.txt:
- If the domain matches your production hostname, it serves the production file
- Else, it serves the disallow-all fallback file

Additional Requirements

- No contributed modules are required.
- Files are created automatically under public://, but you may optionally upload your own content (especially the production robots.txt).
- Example fallback content:
User-agent: *
Disallow: /