Skip to main content
DB

Operations

Storage operations

What an operator can observe about tablets, physical layout, codecs, policy bindings, WAL and checkpoints, and which knobs exist.

This page is the operator's view of the storage model: what to read to see the physical state of a node, and what can be configured.

Reading storage state

Storage state is exposed as read-only system relations. Reading system.storage requires the storage_read administrative action.

SELECT *
FROM system.storage;

STREAM SELECT *
FROM system.storage;

The STREAM SELECT form sends the current rows, then each change to the counters.

SELECT relation_id, tablet_id, owner_epoch, committed_frontier
FROM system.tablets;

Relations to watch

RelationOne row perWhat to look at
system.tabletstabletOwner epoch, committed frontier, routing revision, active feed subscribers, retained feed records, WAL records recovered and replayed at the last open, whether a torn WAL tail was repaired, accepting and pressure_stalled.
system.storagetabletMutable, frozen, cache and durable bytes; active state parts; admitted and rejected mutations; flushes; compactions and their input and output; checkpoint attempts, completions and failures; WAL groups, records, barriers, closed and reclaimed segments, queued records and bytes; index generations and index lag in commits.
system.table_storage_planstableThe storage profile and the physical plan it expanded into.
system.table_column_codecscolumnThe codec chain in force, and whether it was declared explicitly.
system.table_policy_bindingstable and roleThe built-in policy bound for each role.
system.wal_reclamation_blockersblockerWhy WAL segments cannot be reclaimed yet: the oldest retained segment, the frontier required and the frontier covered.
system.table_tabletstable and tabletWhich tablets belong to which table, and each tablet's lifecycle.

Every row carries source_revision, observed_at_millis and freshness_millis, so you can tell how current an observation is.

What to check

  • Write pressure: pressure_stalled and accepting in system.tablets, and rejected_mutations (writes the tablet deferred under pressure) in system.storage.
  • WAL growth: wal_queued_records and wal_live_bytes in system.storage. If live WAL bytes keep growing, look at system.wal_reclamation_blockers for the reason.
  • Checkpoints: checkpoint_failures, checkpoint_stage and checkpoint_failure_code in system.storage.
  • Recovery: after a restart, recovered_wal_records, replayed_wal_suffix_records and repaired_torn_wal_tail in system.tablets show what the open replayed.
  • Index lag: table_index_lag_commits in system.storage.

Physical layout

The physical layout of a table is fixed at CREATE TABLE by STORAGE PROFILE ROW | COLUMNAR | HYBRID | DEFAULT and by its column codecs (see Storage model). Every table is bound to placement, retention and compaction policies, and named policies attach with … POLICY <name>.

Configuration keys

WAL behaviour is set in the node configuration under node.storage.wal:

KeySets
node.storage.wal.shard_countThe number of WAL shards per database runtime.
node.storage.wal.group_commit_delay_microsHow long the WAL waits to gather a commit group.
node.storage.wal.seal_interval_millis, node.storage.wal.seal_bytesWhen an open WAL segment is sealed, by time and by size.
node.storage.wal.maximum_record_bytesThe largest WAL record one tablet batch may write.

node.storage.recovery_workers sets how many relations are recovered at once when the node opens.

See Configuration for the configuration file itself and Durability and recovery for the commit and restart contract.

On this page