cookie_bot_protection

This module is designed to provide a protection from simple crawlers like AI or crawling scripts by testing redirection and set cookies capabilities.

Lots of AI or non-conventionnal crawling bots do not follows robots.txt and can produce significant performance issues on heavy computation pages like search pages.

It will not fully block traffic from bots or make them timeout, but it will significantly reduce impact by quickly provide à 302 and 403 response without computing the page which is not in Page Cache.

You'll be able to define url patterns that are concerned by the protection with PCRE regex patterns.
Don't forget to use non-capturing group in your patterns that will be checked on a preg_match functions.
A misused of the regex rules can lead to protect all the website and affect SEO crawlers.

I recommand to apply the same patterns as your robots.txt.

By default, a new attempt of loading the page will be provided with a refresh header to allow retries on legitimate users.
This option can be disabled by setting the refresh time to zero.

A challenge cookie with the name SESScookiebotprotection is created to validate the challenge. This name should bypass generic Varnish/Fastly configurations for Drupal.

This module intend also to provide a tool for people that are not able to install and protect their website with a challenge WAF like anubis or go-away reverse-proxies.

As the Cookie name is used on the project name, this module will not evolve to provide other protections like javascript Proof of Work or a Captcha protection.

You can exclude specific Ips or User-Agent from beeing challenged.