Operations
Durability and recovery
When a write is acknowledged, what survives a crash, how a node recovers on restart, and how a graceful stop drains.
The engine has one durability contract for every table: a write is acknowledged only after it is durable in the write-ahead log (WAL), on every table and every request.
The acknowledgement contract
- A tablet's owner orders the commit and assigns its commit sequence.
- The commit's records join a WAL group. The group is written to the local WAL and passes one persistence barrier.
- Only then is the write acknowledged to the client and published to consumers through PSI.
An acknowledged write therefore survives a crash of the process or of the machine, as long as the node's storage survives and honours the persistence barrier. A write that was not acknowledged may or may not be present after a crash.
The WAL is segmented and checksummed. Each database runtime spreads its tablets over a fixed set
of WAL shards (node.storage.wal.shard_count); each tablet belongs to one shard.
What an operator can observe
| Signal | Where |
|---|---|
| WAL groups, records, framed bytes and persistence barriers written | system.storage: wal_durable_groups, wal_durable_records, wal_durable_framed_bytes, wal_persistence_barriers |
| WAL waiting to be written | system.storage: wal_queued_records, wal_queued_bytes |
| WAL segments closed and reclaimed, and live WAL bytes | system.storage: wal_closed_segments, wal_reclaimed_segments, wal_live_bytes, wal_reclaimed_bytes |
| Why WAL cannot be reclaimed | system.wal_reclamation_blockers |
| Checkpoints | system.storage: checkpoint_attempts, checkpoint_completions, checkpoint_failures, checkpoint_frontier |
| What the last open replayed | system.tablets: recovered_wal_records, replayed_wal_suffix_records, repaired_torn_wal_tail |
| Tablet recovery progress | system.storage: tablet_recovery_stage, tablet_recovery_durable_frontier, tablet_recovery_applied_frontier |
See Storage operations for how to read these relations.
Recovery on restart
When the node opens, each tablet:
- loads its last checkpoint;
- replays the WAL suffix written after that checkpoint;
- repairs a torn final record, if the process stopped in the middle of writing one, and
records that it did (
repaired_torn_wal_tail); - starts accepting writes again.
node.storage.recovery_workers sets how many relations recover at once.
Each transformation resumes from its own durable cursor, rebuilds from current rows, or
reseeds, depending on its shape, and logs its choice as event=derived_runtime_resume_shape (see
Transformations). Materialized views reseed.
Process-resident relations such as system.query_history start empty.
Graceful stop
On SIGTERM the node drains before it exits, so that every transformation has caught up with
its sources when the process stops:
- New public writes answer
resource_exhausted(HTTP 429), so clients retry them on another node. Reads keep being answered. - The readiness probe
GET /auth/issuersanswers 503 while the node drains. - The node waits until every transformation's durable checkpoint has reached the committed heads
of its source tablets, and logs one
event=planned_stop_drainline per transformation withoutcome=drained,deadlineorunobserved.
node.lifecycle.stop_drain_seconds bounds the drain: default 120, at most 3,600, and 0 disables
it. A second SIGTERM or SIGINT ends the drain at once. A transformation still behind at the
deadline reseeds on the next start.
Replication and backups
The WAL of each tablet is replicated to its replicas on other nodes; when a node is lost, a replica takes over with every acknowledged write. Backups add point-in-time copies of a node's tables; see Backup and recovery.