AI Eval

AI Eval measures and improves the quality of your AI integrations in Drupal. Define evaluation datasets in YAML, config, or content entities (or author them in the browser), run them against your agents or any AI provider, and get scored results with pass/fail quality gates you can trust.

Most of this module's code was generated with an AI assistant under human direction, review, and testing, using Strikethroo & Kenkeep for improved Drupal code generation.

Two evaluation modes

Agent mode invokes ai_agents plugins end-to-end, testing the full loop: tool calls, reasoning, and response quality.

Chat mode sends prompts directly to any AI provider (Anthropic, OpenAI, Ollama). No agent framework needed. Useful for evaluating system prompts, RAG pipelines, Q&A bots, or classification tasks. Multi-turn conversations are supported: questions can carry conversation context, and the full transcript is stored and shown.

Run and review in the browser

A portfolio landing page gives you an at-a-glance view of every target, with grouped navigation for targets, datasets, traces, judges, results, and settings. Launch an evaluation from the UI and watch live progress, or cancel a run in flight. No Drush required.

Results dashboard

The results dashboard shows status cards with run summaries and week-over-week trends, plus a target health table with score meters, sparkline trends, and error counts.

Each result detail page shows collapsible per-question breakdowns with grader scores, tooltips with judge reasoning, and a weakest grader callout.

Production-trace review and labeling

Import real production traces and review them in a grouped queue. Label a trace by verdict and by dimension (holistic or per-dimension, with domain tags), then promote the good ones straight into a dataset. A promotion integrity guard refuses truncated or streamed-but-uncaptured traces, so your datasets are never seeded with incomplete records.

Pluggable graders

Ships with nine graders. Six are LLM-based judges:

• relevance_grader, completeness_grader, actionability_grader: rubric-based scoring across three response dimensions, on a 0-5 scale.
• accuracy_grader: rubric with explicit handling for must_not_contain matches (hard cap at 1) and evasive answers (score ≤ 2). Reads an optional expected_facts block from the dataset and weaves it into the judge prompt as ground truth.
• fact_match_grader: dedicated fact-recall judge that scores only against expected_facts. Skips cleanly when a row has none.
• groundedness_grader: answers one binary question, is every claim supported by the source material. Scores 0 or 1, which the scorer maps onto the composite 0-5 scale.

Three graders are deterministic and run locally with no API cost:

• format_grader: validates the response against the format the question asked for, and against a configurable character cap.
• tool_usage_grader: scores against an expected_tools list, naming missing or forbidden tools in the reason. Beyond {tool, should_run} it can assert the arguments a tool was called with and the order two tools ran in.
• rubric_checks: runs a rubric of deterministic checks against the response. Built-in check kinds include target_match, command, score_delta, and chrF text similarity.

Graders differ in what they can observe. Most read the response text; tool_usage_grader reads the agent's own record of the tools it ran, so it grades what the agent did rather than what it said about it. The optional ai_eval_droost submodule adds a tenth grader, droost_state, which inspects the Drupal state an agent actually built.

You can add your own graders as plugins in any module. Check executors are pluggable too: a tagged executor can contribute new rubric check kinds and the JSON Schema that validates them, so out-of-core bundles (for example, layout and design-token fidelity via headless Chromium) can add check kinds without patching core.

Judge validation and trust

An LLM judge that disagrees with humans is an unvalidated ruler. If you gate deployments or optimize prompts based on judge scores, you need to know how much to trust those scores in the first place.

Two Drush commands close that loop:

• drush ai-eval:sample-traces pulls stored judge traces for one grader and writes a YAML fixture stubbed for human labeling.
• drush ai-eval:validate-judge compares your labels to the judge's scores and reports true-positive and true-negative rates with a pass/fail verdict against Shankar's trust rule (TPR ≥ 0.9 AND TNR ≥ 0.9).

Once a judge is validated, its trust status is surfaced as advisory warnings across the CLI, the results detail page, and the dashboard, so an unvalidated or failing judge never quietly drives a gate decision. When a single trusted judge is the sole labeler, results also report a bias-corrected pass rate (Rogan-Gladen) with a confidence interval, correcting the observed rate for the judge's own known error rates.

A gate you can trust

The gate is only as trustworthy as the evaluation behind it. When LLM judges error out (a provider outage, an unparseable response) their questions are flagged degraded and excluded from the gate rather than silently dropped, and a target fails outright when scored coverage falls below half. An infrastructure failure can no longer masquerade as a passing evaluation.

Prompt optimization

When a target fails its gate, the optimizer analyzes the failure patterns and proposes an improved system prompt. Review proposals with a side-by-side prompt comparison before applying.

LLM judges are noisy, so the optimizer runs each candidate prompt multiple times and compares the average against an equally-repeated baseline. You can tune the number of runs per candidate, require a minimum dataset size before the optimizer will run, and set an improvement margin that stays above the judge's own variance floor.

Portable result envelopes

Every result can be exported as a portable, standards-aligned evaluation envelope (EEE) carrying the scores, the gate verdict, judge-trust metadata, and captured generation telemetry. Export moves results between tools and dashboards without lock-in.

Quality gates

Each target defines a threshold and a gate type. Hard gates fail the run when the score is below threshold, which is what you want in CI pipelines. Soft gates log a warning instead. Use hard gates to block deployments when AI quality drops.

Response char limit

Long agent responses can silently blow out judge prompts and cost. A global response_char_limit caps the length of the response text the judge sees, with a per-target override for targets that need more headroom. The cap is explicit rather than hidden inside grader code.

Admin UI

• A portfolio landing page and grouped navigation across targets, datasets, traces, judges, results, and settings
• Configure targets with select fields for agents, providers, and models
• Author datasets and questions directly in the browser
• Launch, watch, and cancel runs with live progress
• Review and apply or reject optimization proposals with side-by-side prompt comparison
• Settings for judge provider, scoring thresholds, rate limiting, response char limit, optimizer noise controls, and dataset paths

Drush commands

• drush ai-eval:run (alias aer): run every enabled target, store scored results, decide pass/fail against each target's gate. Scope with --target, --question, impersonate with --as-user, tag cron runs with --source=cron, or preview with --dry-run.
• drush ai-eval:optimize (alias aeo): rewrite prompts on targets that failed their last gate. Runs baseline and candidate multiple times to average over judge noise. Use --propose to queue candidates for human review instead of auto-applying.
• drush ai-eval:import-traces (alias aeit): import production traces (OTLP) for review, labeling, and promotion into datasets.
• drush ai-eval:export-envelope (alias aee): export results as a portable EEE evaluation envelope for exchange between tools.
• drush ai-eval:distill (alias aed): summarize recent results as markdown intel files (weakest graders, recurring failure patterns, trend deltas). Good for team channels or planning docs.
• drush ai-eval:sample-traces (alias aest): pull stored judge traces for one grader into a YAML fixture stubbed for human labeling. Stratified by default, windowed by --window-days, sized by --n.
• drush ai-eval:validate-judge (alias aevj): compare human labels to judge scores and report TPR / TNR with a pass/fail verdict against Shankar's trust rule (TPR ≥ 0.9 AND TNR ≥ 0.9).

Requirements

Drupal 11.3+, PHP 8.3+, AI module. Optional: AI Agents for agent mode.

Documentation

Read the handbook

Installation, configuration, every grader, judge validation, trace review, the optimizer, and a developer guide to writing your own plugins.

Open the documentation

• Installation and configuration to get a first run out of it
• Graders for what each one measures and what it can see
• Trusting your judges for validating an LLM judge against your own labels
• For developers to write your own graders, dataset sources, and check executors