Crawl Control

Crawl Control stops search engines and simple bots from over-crawling Views listings without hiding the listed content from search. Less advanced crawlers can seem like they are attacking your site, particularly if they get stuck in a rabbit hole of indexing pages that list content where near-infinite possibly combinations of filters and pagination can result in huge numbers of crawls. This module helps provide guidance to crawlers around what to follow and index for your Views.

Problem solved

Plain pagination (?page=N) is real content. It stays crawlable, indexable and gets its own canonical URL, so every item in a listing keeps an internal link path. Any other query string variation of a listing (exposed filters, sorts, items per page, tracking parameters) is a variation of the same list. Its pager links get rel="nofollow" and the page is marked noindex, follow, with a canonical pointing at itself.
(Unwanted) crawling / indexing can otherwise become heavy on large views with many pages.

Features

Initially at this stage the module provides these options:

• Per view display settings in the Views UI Advanced column.
• Covers the full pager, mini pager and contrib pagers such as infinite scroll and load more.
• Works for embedded views in blocks, viewsreference fields, and general views pages
• Adjusts Metatag's robots and canonical tags in place instead of duplicating them. Metatag is optional.
• A cache context, so the dynamic page cache is not fragmented by unknown query keys.

This primarily affects Views where the URL changes as pagination and filters are applied so typically includes non-AJAX Views, Views with Drupal 11.3+ Core AJAX enabled, and Views with the Views AJAX History module enabled. Multiple Views per page are supported.

What this module is not doing

This module is not providing any flood control or mitigations against malicious attacks and crawling; that is the business of tools like Cloudflare and Protect Views Flood Control.