Operations
Storage operations
What an operator can observe about tablets, physical layout, codecs, policy bindings, WAL and checkpoints, and which knobs exist.
This page is the operator's view of the storage model: what to read to see the physical state of a node, and what can be configured.
Reading storage state
Storage state is exposed as read-only system relations. Reading system.storage requires the
storage_read administrative action.
SELECT *
FROM system.storage;
STREAM SELECT *
FROM system.storage;The STREAM SELECT form sends the current rows, then each change to the counters.
SELECT relation_id, tablet_id, owner_epoch, committed_frontier
FROM system.tablets;Relations to watch
| Relation | One row per | What to look at |
|---|---|---|
system.tablets | tablet | Owner epoch, committed frontier, routing revision, active feed subscribers, retained feed records, WAL records recovered and replayed at the last open, whether a torn WAL tail was repaired, accepting and pressure_stalled. |
system.storage | tablet | Mutable, frozen, cache and durable bytes; active state parts; admitted and rejected mutations; flushes; compactions and their input and output; checkpoint attempts, completions and failures; WAL groups, records, barriers, closed and reclaimed segments, queued records and bytes; index generations and index lag in commits. |
system.table_storage_plans | table | The storage profile and the physical plan it expanded into. |
system.table_column_codecs | column | The codec chain in force, and whether it was declared explicitly. |
system.table_policy_bindings | table and role | The built-in policy bound for each role. |
system.wal_reclamation_blockers | blocker | Why WAL segments cannot be reclaimed yet: the oldest retained segment, the frontier required and the frontier covered. |
system.table_tablets | table and tablet | Which tablets belong to which table, and each tablet's lifecycle. |
Every row carries source_revision, observed_at_millis and freshness_millis, so you can tell
how current an observation is.
What to check
- Write pressure:
pressure_stalledandacceptinginsystem.tablets, andrejected_mutations(writes the tablet deferred under pressure) insystem.storage. - WAL growth:
wal_queued_recordsandwal_live_bytesinsystem.storage. If live WAL bytes keep growing, look atsystem.wal_reclamation_blockersfor the reason. - Checkpoints:
checkpoint_failures,checkpoint_stageandcheckpoint_failure_codeinsystem.storage. - Recovery: after a restart,
recovered_wal_records,replayed_wal_suffix_recordsandrepaired_torn_wal_tailinsystem.tabletsshow what the open replayed. - Index lag:
table_index_lag_commitsinsystem.storage.
Physical layout
The physical layout of a table is fixed at CREATE TABLE by STORAGE PROFILE ROW | COLUMNAR | HYBRID | DEFAULT and by its column codecs (see Storage model). Every
table is bound to placement, retention and compaction policies, and named policies attach with
… POLICY <name>.
Configuration keys
WAL behaviour is set in the node configuration under node.storage.wal:
| Key | Sets |
|---|---|
node.storage.wal.shard_count | The number of WAL shards per database runtime. |
node.storage.wal.group_commit_delay_micros | How long the WAL waits to gather a commit group. |
node.storage.wal.seal_interval_millis, node.storage.wal.seal_bytes | When an open WAL segment is sealed, by time and by size. |
node.storage.wal.maximum_record_bytes | The largest WAL record one tablet batch may write. |
node.storage.recovery_workers sets how many relations are recovered at once when the node
opens.
See Configuration for the configuration file itself and Durability and recovery for the commit and restart contract.