# Durability and recovery

> When a write is acknowledged, what survives a crash, how a node recovers on restart, and how a graceful stop drains.

The engine has one durability contract for every table: a write is acknowledged only after it
is durable in the write-ahead log (WAL), on every table and every request.

## The acknowledgement contract

1. A tablet's owner orders the commit and assigns its commit sequence.
2. The commit's records join a WAL group. The group is written to the local WAL and passes one
   persistence barrier.
3. Only then is the write acknowledged to the client and published to consumers through
   [PSI](/docs/concepts/psi).

An acknowledged write therefore survives a crash of the process or of the machine, as long as
the node's storage survives and honours the persistence barrier. A write that was not
acknowledged may or may not be present after a crash.

The WAL is segmented and checksummed. Each database runtime spreads its tablets over a fixed set
of WAL shards (`node.storage.wal.shard_count`); each tablet belongs to one shard.

## What an operator can observe

| Signal | Where |
| --- | --- |
| WAL groups, records, framed bytes and persistence barriers written | `system.storage`: `wal_durable_groups`, `wal_durable_records`, `wal_durable_framed_bytes`, `wal_persistence_barriers` |
| WAL waiting to be written | `system.storage`: `wal_queued_records`, `wal_queued_bytes` |
| WAL segments closed and reclaimed, and live WAL bytes | `system.storage`: `wal_closed_segments`, `wal_reclaimed_segments`, `wal_live_bytes`, `wal_reclaimed_bytes` |
| Why WAL cannot be reclaimed | `system.wal_reclamation_blockers` |
| Checkpoints | `system.storage`: `checkpoint_attempts`, `checkpoint_completions`, `checkpoint_failures`, `checkpoint_frontier` |
| What the last open replayed | `system.tablets`: `recovered_wal_records`, `replayed_wal_suffix_records`, `repaired_torn_wal_tail` |
| Tablet recovery progress | `system.storage`: `tablet_recovery_stage`, `tablet_recovery_durable_frontier`, `tablet_recovery_applied_frontier` |

See [Storage operations](/docs/operations/storage) for how to read these relations.

## Recovery on restart

When the node opens, each tablet:

1. loads its last checkpoint;
2. replays the WAL suffix written after that checkpoint;
3. repairs a torn final record, if the process stopped in the middle of writing one, and
   records that it did (`repaired_torn_wal_tail`);
4. starts accepting writes again.

`node.storage.recovery_workers` sets how many relations recover at once.

Each transformation resumes from its own durable cursor, rebuilds from current rows, or
reseeds, depending on its shape, and logs its choice as `event=derived_runtime_resume_shape` (see
[Transformations](/docs/concepts/transformations#after-a-restart)). Materialized views reseed.
Process-resident relations such as `system.query_history` start empty.

## Graceful stop

On `SIGTERM` the node drains before it exits, so that every transformation has caught up with
its sources when the process stops:

- New public writes answer `resource_exhausted` (HTTP 429), so clients retry them on another
  node. Reads keep being answered.
- The readiness probe `GET /auth/issuers` answers 503 while the node drains.
- The node waits until every transformation's durable checkpoint has reached the committed heads
  of its source tablets, and logs one `event=planned_stop_drain` line per transformation with
  `outcome=drained`, `deadline` or `unobserved`.

`node.lifecycle.stop_drain_seconds` bounds the drain: default 120, at most 3,600, and `0` disables
it. A second `SIGTERM` or `SIGINT` ends the drain at once. A transformation still behind at the
deadline reseeds on the next start.

## Replication and backups

The WAL of each tablet is replicated to its replicas on other nodes; when a node is lost, a
replica takes over with every acknowledged write. Backups add point-in-time copies of a node's
tables; see [Backup and recovery](/docs/operations/backup-recovery).

---

Source: https://nyxdb.io/docs/operations/durability/