native_observability

Overview

Native Observability brings real observability into Drupal: request tracing,
execution inspection, performance metrics, telemetry export, and runtime insight, directly
inside your application. No core patches required — install, enable, and start seeing what
actually happens inside your Drupal system.

Starting with 1.0.9 the module family ships three install presets
(drush no:preset:raw, :dashboard, :integrations) that
bring up the right set of sub-modules for a given consumer profile — a Prometheus scraper, an
in-Drupal operator browsing the dashboard, or a Grafana stack receiving OTLP — with one
command. The same module sets stay available as à-la-carte installs.

How it differs from similar tools

Native Observability is not a development toolbar. Tools like
WebProfiler and
XHProf are excellent for inspecting a single
request while developing locally, and are meant to be turned off in production.

Native Observability works at the runtime and operations level instead. It
assigns correlation IDs across requests, builds spans and aggregated metrics (latency,
throughput, P95/P99), and exports telemetry to external stacks such as OpenTelemetry, Prometheus,
and Grafana. The question it answers is not "what happened in this request while I code",
but "how does the system behave over time, and how do I feed that into my monitoring pipeline".

What you get

• Request tracing with correlation IDs
• Execution spans with timeline visualization
• External HTTP (Guzzle) tracing
• Performance metrics: latency, throughput, P95 and P99
• A dashboard covering runtime, cache, database, technical insights, and route-level actions
• Structured reports with printable HTML output
• Pull-based scrape endpoints (Prometheus text format, Elastic bulk JSON) served directly by Drupal
• Push-based telemetry export over OpenTelemetry OTLP
• Request-log exclusion rules to suppress noisy or unwanted traffic
• ECA integration to react to observability events inside Drupal

What's new in 1.0.9

Twelve commits on top of 1.0.8. Headline changes (see the 1.0.9 release notes on this page
for the full list):

• Three install presets. drush no:preset:raw,
  :dashboard, :integrations — one command per consumer profile,
  idempotent, with a "Next steps" block at the end.
• New sub-module native_observability_export. The pull-based
  scrape endpoints (Prometheus text format, Elastic bulk JSON) were split out of
  native_observability_otel into a dedicated sub-module so sites that only need
  a scrape do not have to install the OTLP push exporter.
• Sanity check is now advisory. The install-time gate introduced in 1.0.8
  has been removed. Service-integrity conflicts are still reported on the Status Report and
  via drush no:sanity-check; they no longer block module installation.
• Honest dependency tree. The dashboard now hard-deps on
  cache_observer and database_observer (those pages used to 500
  when the modules were missing). The ECA bridge dropped three runtime deps it never used.
• Automatic upgrade migration. drush updb moves the Prometheus
  bearer-token State key and the Prometheus configuration keys from the old _otel
  namespace to the new _export namespace. Public scrape URLs are unchanged.
• Dashboard dataset CSV exports fixed. The three downloadable datasets
  (slow endpoints, AJAX endpoints, external requests) used to return 500 because of an
  autowire issue; they now return 200 with CSV (or 204 when the range is empty).

Key capabilities

Request-log exclusion

Prevent noisy, synthetic, or infrastructure-driven requests from polluting observability data.
Rules are managed from the settings area and apply across the full request-correlated pipeline.

• Match requests by HTTP method
• Match routes and paths by exact, prefix, or regex pattern
• Match query parameters with operators such as exists, missing,
  equals, regex, gte, lte, and
  between
• Suppress excluded traffic from traces, spans, metrics, observers, and telemetry exports
• Cache-first runtime model, so checks do not query the database on every request

Dashboard and reports

• Latency, throughput, and error rate
• Database and cache observability
• External request analysis
• Span breakdown and technical insights
• Charts powered by Apache ECharts
• Direct row actions such as "Exclude from logs" and "Re-enable logs", with deep-linking to
  exclusion-rule management
• An extended descriptive report of the most relevant runtime, performance, cache, database,
  and external-request findings
• Downloadable dataset CSVs (slow endpoints, AJAX endpoints, external requests)

Scrape endpoints (pull-based export)

The native_observability_export sub-module serves
/native-observability/prometheus in Prometheus text format and
/native-observability/elastic in Elastic bulk JSON. The Prometheus endpoint
publishes per-route counters, latency histograms (P50/P95/P99), cache operation counters and
span counts; the Elastic endpoint streams a paginated bulk of recent traces. Both are pure
readers — they query the local native_observability_* tables, do not push
anywhere, and do not depend on the OTel sub-module.

OTLP push (Grafana / Tempo)

The native_observability_otel sub-module is a push-side OpenTelemetry exporter:
it ships traces over OTLP/HTTP to a collector such as Grafana Alloy. Endpoint, headers,
service name and timeout are configured from the OTel settings form. Installing it is
independent from installing the scrape endpoints above.

ECA integration

Observability events are exposed to Drupal's event-condition-action ecosystem, enabling
automation in response to slow or failed requests, AJAX activity, trace deletion, and other
runtime signals. The bridge only depends on the eca core module; an optional
demo module ships example workflows and pulls in eca_base and eca_log
for its own actions.

How it works

1. A request enters Drupal and is assigned a unique request ID.
2. Native Observability evaluates whether the request should be excluded.
3. If not excluded, execution is traced and stored, and spans capture internal operations.
4. Metrics are aggregated and visualized in the dashboard and reports.
5. Telemetry can be exported by pull (Prometheus / Elastic scrape) and/or by push (OTLP).
6. Runtime events can drive ECA workflows.

Modules

• native_observability — tracing core, request-log exclusion engine, and service-integrity sanity check
• native_observability_execution — execution monitoring
• native_observability_spans — span structure
• native_observability_metrics — performance analytics
• native_observability_cache_observer — cache observability
• native_observability_database_observer — database query observability
• native_observability_export — pull-based scrape endpoints (Prometheus, Elastic). New in 1.0.9.
• native_observability_otel — OpenTelemetry OTLP push exporter
• native_observability_dashboard — UI dashboard
• native_observability_report — reporting
• native_observability_status_block — Drupal block summarising what the family is currently capturing
• native_observability_eca_bridge — ECA integration
• native_observability_eca_bridge_demo — optional demo ECA workflows

Installation

The recommended path is the preset commands. Each preset wraps the sub-modules needed for a
specific consumer profile and is idempotent: re-running it on a provisioned site simply prints
"Already enabled" for each module already on, and exits zero.

composer require 'drupal/native_observability:^1.0'
drush en native_observability -y
drush no:preset:dashboard

The three presets:

• drush no:preset:raw — base, spans, metrics, cache_observer, export. For machine-only consumers (Prometheus, Elastic, CI checks).
• drush no:preset:dashboard — raw plus execution, database_observer, dashboard, report, status_block. For Drupal-admin operators using the in-site UI.
• drush no:preset:integrations — dashboard plus otel and eca_bridge. For sites pushing telemetry to an external Grafana stack and/or wiring observability events into ECA workflows.

Operators who want fine-grained control can install the sub-modules à la carte; the base module
README documents a seven-group install layout with one drush command per group. Drupal resolves
the transitive dependencies automatically — enabling
native_observability_dashboard on its own brings up the full capture stack with it.

Requirements

composer require wikimedia/composer-merge-plugin:^2.1

The dashboard and reports use Apache ECharts for chart rendering, included via a Composer merge
plugin:

"merge-plugin": {
      "include": [
          "[MODULE_ROOT_PATH]/native_observability/modules/native_observability_dashboard/composer.libraries.json"
      ]
  }

• Drupal 10.3+ or 11.x
• PHP 8.3 or newer
• The ECA bridge requires the eca contrib module only when you want event-driven automation
• The OTel sub-module requires an OTLP/HTTP collector reachable from the Drupal site (for example Grafana Alloy)
• Memcache and Redis cache backends require their respective contrib modules; the cache observer logs uniformly across cache.backend.null, cache.backend.database, cache.backend.memcache, cache.backend.apcu and cache.backend.redis (verified before each release)

Performance

Approximate overhead is around 10% when full tracing is enabled, and noticeably
lower on cached or anonymous traffic. Most persistence work is deferred until after the response
is sent, so the impact on user-perceived latency is smaller than the total processing cost.

Exclusion rules are stored in the database as the source of truth and cached so the application
does not query the database on every request; the loaded rule set is also reused in memory during
a request.

Usage notes

Native Observability is built for diagnostics, investigation, performance analysis, and controlled
monitoring windows. Because it can collect a large amount of runtime data, keeping full tracing
enabled indefinitely without limits is not recommended.

Configure the retention and record-limit options before using it in production or on busy
environments, so that stored traces, spans, metrics, and reports stay under control. Use
request-log exclusion to suppress health checks, synthetic traffic, probes, and obsolete endpoints.

When enabling the scrape endpoints or the OTLP push exporter, review security and access
configuration carefully: telemetry endpoints may expose sensitive operational data if they are
not protected. The Prometheus endpoint is guarded by a bearer token stored in Drupal State
(never in exported configuration); set it from the export settings page before exposing the
URL publicly. Setup guidance is provided in the native_observability_export and
native_observability_otel READMEs.

On sites that already run other modules touching the HTTP client, cache invalidation, stack
middleware, or kernel event subscribers, run the sanity check after installation to confirm
nothing is competing with the instrumentation. Since 1.0.9 the check is advisory and never
blocks installation, so the report is informational only.

Status

Stable 1.0.9 release. Actively maintained.

Roadmap

Development continues around stronger external-observability integrations, richer dashboard
insights, better operator workflows, and deeper ECA automation. Near-term focus areas:
expanding the scrape endpoints with additional metric families and an authenticated Elastic
variant, tighter coverage of the install presets with functional tests, and a documentation
pass that consolidates the per-module READMEs into a single operator handbook. Suggestions
and real-world monitoring use cases are welcome in the issue queue.