DB
Back homeUse case

Observability & logs

Turn raw event streams into live metrics and materialized views with continuous transforms — no external stream processor.

The problem

Why this is hard today

Observability pipelines are stream processors wearing a database costume: raw events flow into a queue, a separate processor rolls them into metrics, and yet another store serves dashboards. The rollups lag, the schemas drift, and every layer is a place for events to be double-counted or dropped.

The core operation — turn a firehose of raw events into a small set of continuously-updated metric tables — is exactly what a streaming database should do in one place.

The freshness has to come from event routing, not from a timer re-scanning the source; and the metric tables have to survive restarts without gaps.

Architecture

How NYXDB does it

Raw events append; a continuous transform rolls them into a live metric table that stays fresh through PSI routing, with gap-free recovery on restart.

  1. 01

    Ingest events

    Raw log/event lines append to a source table.

  2. 02

    Roll up (transform)

    CREATE TRANSFORM … INTO a metric table maintains counts and rates incrementally.

  3. 03

    Stay fresh (PSI)

    Only matching changes are routed to the transform — no periodic full re-scan.

  4. 04

    Serve

    Dashboards read the metric table; STREAM SELECT follows it live.

Real SQL

In practice

Metric target + rollup transform
CREATE TABLE service_errors (
  service String NOT NULL,
  errors  UInt64,
  PRIMARY KEY (service)
) SETTINGS mode='keyed';
CREATE TRANSFORM roll_errors
  INTO service_errors AS
SELECT service, count() AS errors
FROM logs
GROUP BY service;
Follow the metric live
STREAM SELECT service, errors FROM service_errors;
Inspect running transforms
SELECT name, state, lag, rows_emitted, last_error
FROM system.transforms;

Every statement follows the engine’s own test SQL shapes. See the SQL reference for full syntax.

Capabilities

What you get

In-database rollups

CREATE TRANSFORM turns raw events into live metric tables.

PSI freshness

Metrics update via event routing, not a re-scan timer.

Gap-free recovery

Transforms resume from their reflected position after restart.

Exact counts

No double-counting on the critical path.

Self-observable

system.queries, system.traces, and system.transforms expose the engine itself.

One engine

Queue, processor, and store collapse into one runtime.

Proof

Measured on the vetted benchmark lane

gap-free

transform recovery from reflected position

ADR-080

2.66M rows/s

engine-side durable ingest, single core

1M-row batches (~232B, 16-col)

group-fsync default

View benchmark952M rows/s

single-shard columnar scan

vetted table

View benchmark

Measured on Apple M4 Max (dev), macOS — server-class validation pending. Release build, median of 5, commit-pinned (d4a3885b, 2026-07-07). Ingest figures are engine-side. See the full benchmark suite.

Learn more

Related documentation

Transforms

Standing rollup pipelines.

System tables

Observe the engine itself.

Profiling

Tracy, Parca, Pyroscope.

Roll events into live metrics

Define a transform and watch the metric table update itself.

Get started Read the docs