Metrics and events

Build local dashboards and alerts from gauges, counters, queries, and subscriptions.

system.metrics contains current signed gauges; system.events contains monotonic unsigned counters. Every row includes name, value, and description.

SELECT name, value, description
FROM system.metrics
ORDER BY name;

SELECT name, value, description
FROM system.events
ORDER BY name;

Discover names from the running revision instead of assuming a static list.

Collection rules

Poll metrics as current values.
Derive counter rates from successive samples; handle process restart by resetting the baseline.
Attach build revision, process start time, and collection timestamp.
Preserve descriptions with the schema mapping.
Use a collection interval no faster than the useful system-table refresh cadence.
Keep external history; system tables are local process telemetry.

High-value gauges

Area	Representative names
Process	`memory_bytes_total`, `process_virtual_bytes`
Query/stream	`live_queries`, `open_subscriptions`
Tail	`committed_tail_rows`, `committed_tail_bytes` and their configured maximums
Spill/seal	`committed_tail_spilled_bytes`, `committed_tail_spilled_chunks`, `storage_sealed_in_memory_bytes`
Parts/storage	manifest part/shard counts, active scan refs, pending retired parts
Compaction	configured/active jobs, queued shards, completed merges
Caches	current bytes/entries/open files and configured maximums
I/O backend	requested/actual backend, queue depth, io_uring availability

Read each row's live description for exact semantics.

High-value counters

Area	Representative names
Query volume	`query_parse_us_total`, `query_exec_us_total`, `process_cpu_us_total`, `rows_scanned`, `rows_returned`
WAL	`wal_records_appended`, `wal_bytes_appended`, `wal_fsyncs`
PSI	`psi_events_published`, `psi_events_delivered`, enqueue/delivery time, identity violations
Pruning	shards, parts, and granules considered/pruned
Storage I/O	read/write operations, bytes, time, sync time, backend fallbacks
Caches	hit/miss counters by metadata, decoded, bitmap, and materialized caches
Memory	`memory_governor_ramp_engagements` and `memory_governor_blocks`

Alert design

Condition	Signal	First response
Process unavailable	Failed protocol ping	Preserve logs; confirm process/listener/recovery state
Query overload	Rising `queue_us`, server-busy/queue-timeout errors	Identify long/high-memory queries; reduce intake or fix capacity
Memory pressure	Ramp/block counter deltas, high RSS/tail/sealed bytes	Capture memory profile; inspect tail, queries, streams, flush
Tail not draining	Tail rows/bytes rise without part/flush progress	Inspect disk, flush errors, policy, capacity, compaction
Stream consumer lag	Pending bytes/events, p95/p99 lag, overflow count	Fix downstream/backpressure; resubscribe only under app policy
I/O regression	Read/sync time and fallback counters rise	Check backend, kernel, filesystem, device latency
Compaction backlog	Queued shards/parts grow and merges stall	Check I/O caps, free space, errors, job sizing
Backup risk	Scheduled command failure, lag/fence growth, missing drill	Repair repository/source health and rerun create + restore drill

Use rate, duration, and multiple corroborating signals. A single counter value after a restart is not an incident.

Query-level evidence

For a latency alert, save the matching rows from system.queries and system.query_log, including:

queue, prepare, scan, pipeline, materialize, and serialization time;
per-operator time;
disk/cache/pruning counters;
rows/bytes and peak memory;
WAL/commit/PSI time for mutations; and
relation UUID/version where present.

This makes the alert actionable before escalating to an external profiler.