Search API Japanese Tokenizer

Search API Japanese Tokenizer is a Drupal module that segments and indexes Japanese text at the word level. By default, Drupal's standard search and the Search API module use N-gram segmentation, which can be imprecise for Japanese. This module improves search performance using advanced natural language processing without requiring external search engines like Apache Solr or Elasticsearch.

Japanese differs from English and many other languages in that there are no spaces between words. For example, in English, This is a pen. clearly separates words with spaces, making tokenization straightforward. However, in Japanese, これはペンです。 does not include spaces, making it challenging to determine word boundaries. Therefore, a specialized natural language processing technique is required to segment text properly.

By default, Drupal's standard search and the Search API module use N-gram segmentation, which has the following issues:

• It is practically impossible to index and search for single-character words.
• Unintended articles may appear in search results.
• It only supports exact phrase matching.

The Search API Japanese Tokenizer module addresses these issues, enhancing Drupal’s search capabilities and improving the accuracy of Japanese search queries without relying on external search engines.

What's new in 2.x

Version 2.x delegates all natural language processing to the JNLP (Japanese Natural Language Processing) module, which now handles the actual text analysis. This release also adds a new pure-PHP morphological analyzer, Igo-php, so morphological analysis is now available without installing anything on the server.

If you are upgrading from 1.x, note that JNLP is now a required dependency and that the analyzer paths for MeCab and Sudachi have moved from the processor settings to settings.php. Please read the README for the full upgrade procedure before running database updates.

Features

• Enables indexing and searching at the single-character level
• Improves search accuracy by segmenting text into meaningful words with natural language processing
• Lets you choose from four tokenizers: TinySegmenter, Igo-php, MeCab, and Sudachi
• Offers pure-PHP tokenizers (TinySegmenter and Igo-php) that require no server-side installation
• Supports machine learning-based tokenization (TinySegmenter)
• Supports index exclusion based on character type (TinySegmenter only)
• Resolves variations in spelling by converting words to their base form when using morphological analysis (Igo-php, MeCab, and Sudachi)
• Allows exclusion of index entries based on part of speech when using morphological analysis

Post-Installation

1. Enable the Search API and JNLP modules
   This module requires the Search API module and the JNLP module. Install the JNLP submodule that corresponds to the tokenizer you want to use, and configure it (for MeCab and Sudachi, set the analyzer paths in settings.php).
2. Install the module Enable the search_api_japanese_tokenizer module and the submodule for your chosen tokenizer:
   • search_api_tinysegmenter
   • search_api_igo_php
   • search_api_mecab
   • search_api_sudachi
3. Configure the search server and index
   Go to /admin/config/search/search-api and configure the search server and search index.
4. Select the tokenizer
   In the Search API processor settings, choose one of the following tokenizers:
   • TinySegmenter tokenizer
   • Igo-php tokenizer
   • MeCab tokenizer
   • Sudachi tokenizer

   A tokenizer is hidden from the form when its analyzer is not available on the server. Do not enable more than one Japanese tokenizer on the same index.

5. Disable the default Tokenizer and Transliteration processors
   The default Tokenizer processor should be disabled, as enabling it may cause incorrect indexing. The Transliteration processor should also be disabled, because it converts Japanese characters into alphabetic representations and prevents indexed tokens from matching Japanese search keywords.
6. Set the minimum word length to 1
   Japanese has many meaningful single-character words (e.g. 猫, 犬, 木). To make them searchable, set the minimum word length to 1 in the tokenizer processor, in the Database server configuration, and in the "Minimum keyword length" of the Views filter on your search page.

Additional Requirements

This module requires the following components:

• Search API module
• JNLP module (handles all natural language processing)
• A tokenizer engine, provided through the JNLP submodules:
  • TinySegmenter (no additional installation required; pure PHP)
  • Igo-php (no additional installation required; pure PHP)
  • MeCab (must be installed on the server)
  • Sudachi (must be installed on the server; since it is written in Java, a JRE is required)

If using MeCab or Sudachi, they must be installed on the server beforehand.

Recommended modules/libraries

• Search API Japanese Normalizer: helps improve search accuracy by normalizing text, including unifying hiragana and katakana and converting between full-width and half-width characters.

Related articles (in Japanese)

• 本モジュールの日本語 README
• Drupalの日本語検索、ちゃんと動いてる？N-gramの課題と自然言語処理での改善策
• Drupalの日本語検索を改善！自然言語処理を活用したSearch API Japanese Tokenizerモジュールの紹介
• 日本語の検索性能を向上させるモジュール Search API Japanese Tokenizer 2 をリリースしました