Native Observability

Overview

Native Observability brings runtime observability into Drupal: request
tracing, execution inspection, performance metrics and telemetry export — directly inside
the application, with no core patches. Install, enable, look at what is actually happening
in your Drupal system, and ship the signal to your monitoring stack.

It is not a development toolbar

Tools like WebProfiler and
XHProf inspect a single local request
while you code, and you turn them off in production. Native Observability lives at the
runtime and operations layer: it assigns correlation IDs, builds spans,
aggregates latency/throughput/P95/P99 over time, and exports to OpenTelemetry, Prometheus
and Grafana. The question it answers is not "what happened in this request" but
"how does the system behave over time, and how do I feed that into my monitoring pipeline?"

What is implemented

Each item below is shipping today, exercised by the test module
(testing_native_observability) and validated end-to-end by
drush no:validate-pipeline at every release.

• Request tracing with correlation IDs across the full hop chain.
• Execution spans with timeline visualisation in the dashboard.
• Outbound HTTP tracing via a Guzzle middleware on the shared
  http_handler_stack: every call that flows through Drupal's
  http_client is captured automatically, including calls issued by libraries
  that wrap it (Typesense PHP SDK, custom REST clients, OTLP exporter). Per-call
  categorisation is opt-in via a request option (subtype + name).
• Performance metrics per route (request count, throughput,
  avg / P95 / P99 duration) aggregated into 5-minute buckets.
• Forensic Route Analysis — see the dedicated section below.
• Dashboard with four pages: Overview, Cache, Database, Forensic Route
  Analysis. Apache ECharts rendering; "Exclude from logs" / "Re-enable logs" row actions
  with deep-linking to exclusion-rule management.
• Reports generated as a self-contained printable HTML document
  (browser "Print → Save as PDF" is the supported export route; there is no server-side
  PDF renderer).
• Pull-based scrape: /native-observability/prometheus
  (Prometheus text format) and /native-observability/elastic (Elastic bulk
  JSON), both bearer-token-protected. Token managed by
  drush no:token:generate|store|revoke.
• Push-based telemetry over OpenTelemetry OTLP/HTTP to a collector
  (tested against Grafana Alloy → Tempo / Mimir / Loki).
• Request-log exclusion by HTTP method / route / path / query parameter
  (operators: exists, missing, equals, regex, gte, lte, between), cache-first runtime
  model.
• Cache observability across
  cache.backend.{null, database, memcache, apcu, redis}.
• Database query observability with slow-query inline pagination and
  AJAX paginator on the database tab.
• ECA integration: trace, AJAX, slow-request and error events exposed
  as ECA triggers; a demo workflow pack ships separately.
• Three install presets:
  drush no:preset:{raw, dashboard, integrations}, each idempotent and ending
  with a "Next steps" block.
• Sanity check (drush no:sanity-check) reporting the live
  state of every instrumentation point; advisory, never blocks installation.

Known limitations / not yet implemented

• Browser-side traffic is not captured server-side.
  search_api_typesense issues read queries directly from the browser via the
  Typesense InstantSearch JS adapter. That traffic never reaches Drupal's
  http_client and therefore produces no PHP-side span. Indexing, schema sync
  and any server-side query are captured normally.
• No server-side PDF renderer. Reports ship as a printable HTML
  document; "Save as PDF" goes through the browser's print dialog. A wired-up server-side
  renderer (for example via headless Chromium) is on the roadmap.
• Drush php-script does not flush the span buffer. Spans
  are buffered and flushed on kernel.terminate, which Drush command runners
  dispatch but php-script does not. Ad-hoc scripts must flush the buffer
  explicitly via
  \Drupal::service('native_observability_spans.span_insert_buffer')->flush().
• Authenticated Elastic scrape variant is not yet shipped; the current
  Elastic endpoint shares the same bearer token as the Prometheus one.
• Operator handbook is currently spread across per-module READMEs; a
  single consolidated guide is on the roadmap.
• Functional test coverage of the install presets is a roadmap item;
  the presets are exercised via integration runs today, not via Drupal functional tests.

Forensic Route Analysis

The Forensic Route Analysis page (under the dashboard at
/admin/reports/native-observability/dashboard/forensic-route-analysis) is
the module's investigative tool. Given a single route or path, it correlates every signal
the family has captured for it into one timeline-aware view:

• Recent traces for the selected route, with request id, status code,
  duration and a deep-link to the per-trace span breakdown.
• Execution spans reconstructed as a timeline, so a slow path shows
  immediately whether the cost is in PHP, in a database query, in an outbound HTTP call,
  or in cache invalidation.
• Outbound HTTP signals: every external request issued during the
  selected traces — host, method, status code, duration — pulled from the
  http.client spans, including calls captured under custom subtypes
  (http.client.typesense, http.client.typesense.facets, …).
• Slow queries recorded by the database observer for the same route,
  inline-paginated.
• Cache events from the cache observer (page-cache hits/misses,
  dynamic-page-cache, tag invalidations, orphan invalidations).
• Candidate root causes: the page surfaces the most likely culprits
  (slowest external dependency, hot tags, expensive queries) so an operator can shortcut
  the "where is the time going" question without leaving Drupal.
• Direct row actions to exclude noisy routes from observability or
  re-enable previously excluded ones, deep-linking to the exclusion-rule management form.

Forensic Route Analysis is designed for the moment "this one endpoint is misbehaving":
pick the route, see everything that happened on it correlated by request id, and decide
whether to fix code, tune cache, add an exclusion rule, or open a ticket against an
external dependency.

How it works

1. A request enters Drupal and gets a correlation ID.
2. Exclusion rules decide whether the request is observed.
3. If observed, the trace, its spans, and any outbound HTTP / SQL signal are captured.
4. Metrics aggregate into 5-minute buckets and feed the dashboard + reports.
5. Telemetry leaves the site by pull (Prometheus / Elastic scrape) and/or push (OTLP).
6. Runtime events can drive ECA workflows.

Persistence is deferred to kernel.terminate: most of the bookkeeping happens
after the response has been sent, so user-perceived latency is lower than the total
processing cost.

Reference surface (for integrators)

Database tables

• native_observability_trace — one row per request
• native_observability_spans — execution spans, including outbound HTTP
• native_observability_route_metric — 5-minute aggregates per route
• native_observability_cache_event — cache observer events
• native_observability_database_query — slow-query records
• native_observability_execution_history — execution context history
• native_observability_request_exclusion_rule — exclusion rule storage

Public service IDs

• native_observability.trace_storage
• native_observability_spans.span_storage
• native_observability_spans.span_insert_buffer — flush manually in
  drush php-script
• native_observability_metrics.route_metrics_aggregator
• native_observability_report.document_builder

Permissions

• access native observability dashboard
• administer native observability
• access native observability execution

ECA event channels

• Slow request detected
• AJAX request finished
• 500-error trace recorded
• Trace deleted

Install

composer require 'drupal/native_observability:^1.0'
drush en native_observability -y
drush no:preset:dashboard

The three presets:

• no:preset:raw — machine-only consumers (Prometheus, Elastic, CI).
• no:preset:dashboard — adds the in-Drupal dashboard and report.
• no:preset:integrations — adds OTLP push and the ECA bridge.

À-la-carte installs work too; Drupal resolves transitive deps automatically. The base
module README documents a seven-group install layout for fine-grained control.

Requirements

• Drupal 10.3+ or 11.x
• PHP 8.3+
• The dashboard pulls Apache ECharts via Composer merge plugin
  (wikimedia/composer-merge-plugin:^2.1, configured in the dashboard's
  composer.libraries.json)
• ECA bridge requires the eca module only if you want event-driven
  automation
• OTLP push requires a reachable OTLP/HTTP collector (Grafana Alloy works out of the
  box)

Performance

Approximate overhead is around 10% with full tracing enabled, measured on
anonymous traffic, Drupal 11.x, PHP 8.3, the dashboard preset, and a k6
baseline (p95 ≈ 1 s on the test scenarios). Overhead is noticeably lower on
cached / anonymous traffic. Configure retention and record-limit options before turning
full tracing on in production, and use exclusion rules to suppress synthetic traffic
(health checks, probes, the scrape endpoints themselves).

Security notes

The scrape endpoints expose operational data. They are bearer-token-protected; the token
is stored in Drupal State (never in exported config). Generate one with
drush no:token:generate before exposing the URL publicly. The OTel exporter
can sign requests via custom headers configured on the OTel settings form.

Status

Stable 1.0.10. Actively maintained.