ai_search

AI Search is a semantic vector database search implementation for Drupal that integrates AI-powered vector embeddings with the Search API module. It transforms traditional keyword-based search into intelligent, context-aware semantic search by:

• Converting content into one or more chunks of vector embeddings per entity using AI language models (LLMs) to capture the semantic meaning of content of all links.
• Storing vectors in specialized vector databases such as Milvus, Pinecone, Qdrant, and others with vector support (see below).
• Performing similarity searches that understand meaning and context, not just keywords allowing users to ask natural questions or write in sentence form.
• Underpins RAG (Retrieval-Augmented Generation) workflows for AI assistants and chatbots which provide far better generative text when provided with accurate context.

Unlike traditional search, AI Search retrieves content based on semantic similarity, allowing it to find relevant content even when queries use different words. Traditional search also quickly fails since it counts keyword frequency to build scores, causing longer articles to have the highest relevance.

The AI Search module was formerly a sub-module of the AI core module and supported by the same community.

Key features

Core Search Capabilities

• Search API integration:
  Provides a Search API backend for vector databases to handle the distinct nature of vector databases while leveraging the tried and tested power of Search API.
   
• Vector database support:
  Pluggable architecture supporting multiple vector database providers via the AI module. Easily implement new vector database providers.
   
• Semantic search:
  Uses AI embeddings to understand query intent and content meaning.
   
• Embedding strategies for chunking content:
  Longer content is broken into chunks to ensure accurate semantic can be captured even across hundreds of pages. The pluggable embedding strategy system breaks apart the content, and allows site builders to ensure each chunk still contains the wider context. The AI Search tracker ensures Search API can generate vector embeddings at scale.
   
• Similarity search
  Provide existing vector embeddings to look up similar content using contributed modules like AI Related Content or implement your own.
   

Embedding Strategies for chunking built in

• Contextual Chunks (Recommended):
  Enriches each content chunk with contextual information for improved accuracy.
   
• Average Pool Embedding:
  Creates a single composite vector from multiple chunks using average pooling for better performance.
   
• Pluggable Strategy System:
  Extensible architecture for defining custom embedding approaches.
   

Consider extending an AI Search Embedding Strategy to set a strategy specific to your own content (e.g. splitting by chapter).

Advanced Features

• Chunking Support:
  Intelligently splits large content into manageable chunks for better vector representation.
   
• Entity Access Control:
  Post-query access checks ensure users only see content they are authorized to view.
   
• Score Threshold Filtering:
  Configurable minimum relevance score to filter out low-quality matches.
   
• RAG Integration:
  
  • RagTool function call plugin for AI agents
  • RagAction assistant action for chatbot integration
  • Support for chunk-level and entity-level result aggregation
• Hybrid Search:
  Combine with traditional Search API backends (Database, Solr) using boost processors, use the filtering support that some Vector Database providers have, or use a more native provider like OpenSearch.
   
• Grouping Support:
  Smart result grouping for compatible vector databases, avoiding duplicate entity results. For vector databases that do not support grouping, iteration is built in when a specific number of unique entities are requested as results.
   

Configuration and Management

• Custom Field Configuration Form:
  Enhanced UI for selecting which fields to use for generating the vector embeddings, which to use to provide context for each chunk, and further indexing options.
   
• Multiple Output Modes:
  Choose between raw chunks or aggregated rendered entities (so one result per entity).
   
• Views integration:
  Use Search API's native Views integration to build Views directly from your Vector Database (or combined via the Boost plugins).
   
• HTML to Markdown Conversion:
  Automatic content conversion for better LLM processing.
   
• Batch Processing:
  Utilities for indexing large content volumes efficiently.
   

Developer Features

• API Explorer Integration:
  Use the AI Explorer sub-module of AI Core to easily test your search relevance.
   
• Chunking preview:
  Find the chunking preview on the Search API Fields configuration page to see what chunks are to be passed to the AI to generate embeddings from.

Installation and configuration instructions

• The installation and configuration instructions for AI Search on its own are found in the readme file.
• To leverage AI Search within other AI features, explore options sign-posted on the AI core homepage.

History

AI Search was originally a sub-module of AI Core since mid 2024. Before that it started from a separate modules Search API AI and OpenAI Embeddings, both deprecated after joining forces and providing an abstraction layer so a far wider range of Vector Databases could be supported, and since then significant development has happened to refine and optimise the overall approach and make it easier for developers to effectively use.

VDB Providers

To deliver more accurate responses in AI Chatbots, Assistants, Overviews, and other AI-powered features, a vector database is required. Below are some of the most commonly used Vector Database (VDB) providers:

• Milvus
• Pinecone
• Postgres
• OpenSearch
• Azure
• SQLite
• Qdrant

Boost plugins

Use one of the boost plugins to additionally vastly improve the relevance of the following:

• Search API Database
• Search API Solr

Alternative modules

• Search API Solr Dense Vector Field
  Uses the Solr Dense Vector field type directly so results in a single chunk per entity. Useful for a simple hybrid search but starts to lose relevance with longer articles where the semantic meaning is far better captured in many chunks rather than a single chunk.
   

Drupal AI Search Demo

Try the Drupal AI Search demo for free and see how smarter, meaning-aware search delivers more relevant, context-driven results. Experience chat-based queries that understand intent, not just keywords, and help users find exactly what they need.

Explore the Drupal AI Search demo

Upgrading from AI Search sub-module

Developers extending AI Search previously as a sub-module (e.g. VDB Providers or other functionalities) should read the 2.0.x-dev release notes + the 2.0.0-alpha1 release notes. Site builders should be able to upgrade without issue; however, should see the release notes for new features they can optionally start to use.