SQLite VDB Provider

The SQLite VDB Provider is a database module that works together with the AI module, enabling vector searches with AI Search or other compatible modules. This module integrates with an SQLite database using sqlite-vec, a vector database extension, and the AI module.

Features

SQLite is a widely used, lightweight, and self-contained relational database management system known for its simplicity. A significant advantage is that it operates without requiring a separate database server process; the entire database is stored directly in a local file (or files). This characteristic makes it particularly well-suited for applications requiring local data storage and greatly simplifies deployment as there is no additional database service to set up or manage.

By leveraging the open-source sqlite-vec extension, SQLite can efficiently store and search upon vector embeddings.

This module currently offers features such as creating tables to store vectors and indexed fields, indexing data, deleting indexed data, vector searches with and without filters, and non-vector query searches.

Inspiration

The development of this module has largely been inspired by and referenced the Postgres VDB Provider module.

Installation/Requirement

   
• Install php-sqlite extension (usually built-in)
• Install sqlite-vec extension for sqlite. To do this:
  • Visit the sqlite-vec GitHub releases page.
  • Download the release package corresponding to your operating system and architecture (e.g., a .zip or .tar.gz file).
  • Extract the downloaded archive.
  • Locate the SQLite extension file. This is typically named vec0.so on Linux/macOS, or vec0.dll on Windows(Not yet tested).
  • Move this extension file to the directory configured in your PHP's sqlite3.extension_dir setting (see the next step). Ensure the web server has read access to this file.
• Configure extension-dir runtime config of php to allowed PHP loading extension of sqlite3. Check PHP documention
• Enable the module

Post-Installation

   
• Configure the /admin/config/ai/vdb_providers/sqlite page.
  • Based on the configuration form for the SQLite VDB Provider:
  • Database Directory Path: Enter the absolute path or path with Drupal support scheme where the SQLite database file(s) will be stored. This directory must be writable and readable by the web server process and should not be publicly accessible via the web. For example private://vdb, suggests using a Drupal private file system path, which is a recommended secure approach.
  • Extension File Path: Enter the filename of the sqlite-vec extension file you installed (e.g., vec0.so) without any path. This file should be placed within the directory specified by your PHP's sqlite3.extension_dir configuration.
• Create the Search API server at /admin/config/search/search-api/add-server
     
• Select AI Search as the Backend
• Configure the backend
  • Select SQLite vector DB as the Vector Database
  • Enter the database file path
  • Enter the collection name (main table name)
  • Select the similarity metric to use in vector searches
  • Configure the rest of the AI Search backend as desired (see AI Search documentation)
• When saved, this will attempt to create the collection main table.
• Setup an index that uses the new server and configure as desired (see AI Search documentation)

Additional Requirements 

   
• AI module
• AI Search module
• Key module
• Search API module

Similar Projects

Other VDB providers:

• Postgres VDB Provider
• Pinecone VDB Provider
• Milvus VDB Provider