AI Empathy Evaluation

AI Empathy Evaluation

The AI Empathy Evaluation module provides a framework for evaluating and improving how AI models handle ethical dilemmas with empathetic reasoning. It integrates with Drupal's AI module to benchmark LLM responses, and acts as a testing and quality-assurance layer for AI in Drupal.

Core evaluation

• Scores responses to ethical-dilemma scenarios on four research-backed metrics: decision accuracy, empathy alignment, explanation quality, and consistency.
• Ships with 20 default scenarios across Military, Medical, Emotion-based, Cultural — plus an Organizational category — each with a reference decision and difficulty level (1–5). Create your own through the admin UI or the AI-assisted scenario builder.
• Send scenarios to any configured AI provider (OpenAI, Anthropic, etc.) via the AI module and receive structured responses analysed for empathetic reasoning.

Analysis & tooling

• Interactive dashboard (Chart.js): metric gauges, provider/model comparison, category performance, and trends.
• Multi-provider comparison / benchmarking with rankings, per-scenario tables, historical trends, and CSV export.
• Training mode with cumulative feedback across a session.
• Human rating system with inter-rater reliability (Cohen's / Fleiss' kappa), blind comparison, and rating analysis.
• Scheduled cron-based evaluation with email alerts on threshold breaches and provider-model changes.

Ecosystem integrations (optional submodules)

• AI Empathy Guardrail — an Empathy Score Check guardrail for the AI guardrails system.
• AI Empathy Governance — adds Accountability and Trust dimensions (the Trust · Accountability · Empathy triad) with a triad visualization.
• AI Empathy Observability — exports metrics as OpenTelemetry spans with a Grafana dashboard template.
• AI Empathy Field Action — a one-click "Check Empathy" button on text fields, plus an optional content-moderation publish gate.
• Context Control Centre integration — injects brand / organisational context and adds a tone-alignment metric.
• Domain scenario packs (Finance, Healthcare, HR) with domain-specific scenarios and scoring.

All integrations are backward-compatible and opt-in — enable only what you need.

Research & references

This module is grounded in peer-reviewed research on evaluating empathetic decision-making in AI:

• Evaluating Empathetic Decision-Making in AI: A Comparative Study of Open-Source Models in High-Stakes Scenarios — International Journal For Multidisciplinary Research (IJFMR), 2025-12-13.
  DOI: 10.36948/ijfmr.2025.v07i06.63345
  (ISSN 2582-2160).
• Operationalising Empathy in AI Systems for High-Stakes Decision-Making — International Journal of Innovative Science and
  Research Technology (IJISRT), 2026-06-23.
  DOI: 10.38124/ijisrt/26jun931
  (ISSN 2456-2165).
• Scoring the Score: An Empathy, Trust and Accountability Lens for
  Auditing AI-Assisted Hiring Decisions — International Journal of Science and Research (IJSR), 2026-07-08.
  DOI: 10.21275/sr26705230110
  (ISSN 2319-7064).

See also the Drupal AI module documentation, which this module builds on.