Observability & logs

Turn raw event streams into metric tables that maintain themselves, with no stream processor beside the database.

The problem

Where the usual stack breaks

01
An observability pipeline is a stream processor wearing a database costume: raw events flow into a queue, a separate processor rolls them into metrics, and yet another store serves the dashboards. The rollups lag, the schemas drift, and every layer is a place to double-count or drop an event.
02
The core operation is turning a stream of raw events into a small set of metric tables that are always up to date. That is exactly what a streaming database should do in one place.
03
You want freshness that comes from change routing rather than a timer re-scanning the source, and metric tables that come back correct after a restart.

Architecture

How NYXDB does it

Raw events land in one table; a transformation or materialized view keeps each metric relation current from every committed write; system relations show you each transformation’s progress and lineage.

01
Ingest events
Raw log and event rows land in one source table.
02
Roll up
Counts and sums update incrementally, maintained by a materialized view or a grouped CREATE TRANSFORMATION.
03
Stay fresh (PSI)
Each committed change is delivered to the subscribers that read it, so metrics move with the source instead of waiting for a re-scan.
04
Resume
Metrics come back correct after a restart: transformations checkpoint their progress and resume from their cursors or reseed from a source snapshot, depending on shape.

Examples

In practice

Activity per balance, kept current by a transformation

CREATE TRANSFORMATION balance_activity INTO balances ATTRIBUTE GROUP activity AS
SELECT
  chain_id,
  recipient AS address,
  token,
  COUNT(*) AS transfer_count,
  MAX(transferred_at) AS last_transfer_at,
  MAX(_index) AS activity_index
FROM transfers
GROUP BY chain_id, recipient, token;

Live volume per token, as a query that stays open

STREAM SELECT chain_id, token, SUM(amount) AS volume, COUNT(*) AS transfers
FROM transfers
GROUP BY chain_id, token;

Read another instance’s balances through a source

-- on the instance that owns the tables
CREATE APPLICATION ACCESS analytics_reader;
GRANT read, subscribe
  ON TABLE balances
  WHERE chain_id = 1
  TO APPLICATION ACCESS analytics_reader;
ALTER APPLICATION ACCESS analytics_reader MINT SECRET;
-- on the reading instance
CREATE SOURCE ledger
  KIND nyxdb_instance
  OPTIONS (
    endpoint 'wss://ledger.example.com/client',
    credential 'analytics-reader'
  )
  EXPOSING (balances AS ledger_balances);

The examples use the same two tables on every page. See the SQL reference for full syntax.

Capabilities

What you get out of the box

In-database rollups

Metric relations are maintained by materialized views and transformations, inside the engine.

PSI freshness

Metrics update from committed changes, not from a re-scan timer.

Restart contract

Each transformation shape resumes from its cursor or reseeds from a snapshot.

Maintained aggregates

Counts, sums, minimums, maximums and averages stay current, maintained by transformations.

Self-observable

See the derived graph in system.transformations, system.transformation_revisions and system.derived_lineage.

One engine

Queue, processor and store collapse into one node process.

Learn more

Transformations

Continuous rollups.

Views

Materialized views and materializations.

System tables

Observe the engine itself.

Roll events into live metrics

Define a materialized view and watch it follow the source.

Get startedRead the docs