Durability and crash recovery
Choose the acknowledgement contract and understand local restart recovery.
Durability has two independent inputs:
- the table's named
storage_policydecides whether data is durable and where durable artifacts belong; and --wal-syncsets the default DML fsync posture, except where a policy such assync_diskrequires synchronous WAL.
Neither setting creates a replica.
WAL sync modes
| Mode | Acknowledgement and crash contract | Use |
|---|---|---|
always | Fsync every DML statement before response. An acknowledged write survives process, OS, and power crash on correctly functioning durable storage. | Strongest local acknowledgement boundary |
group | Fsync after 64 records or 10 ms, whichever comes first. Process crash loses no accepted tail; hard OS/power crash may lose the unfsynced acknowledged tail. | Current executable default |
off | Rely on OS page-cache writeback. A hard crash can lose an unbounded acknowledged tail; clean shutdown performs a final sync. | Benchmark/test only |
DDL records are fsynced regardless of the DML mode.
group is not fsync-before-ack. If the written RPO says an acknowledged
write must survive power loss, use --wal-sync=always or a table policy
that enforces synchronous WAL, and verify the storage stack honors fsync.
Built-in table postures
| Policy | Serving posture | Durable |
|---|---|---|
ephemeral_memory | Process memory | No |
memory_data | Memory-resident serving with durable WAL/parts under --data-dir | Yes |
disk_data | Local disk serving/parts with memory indexes | Yes |
sync_disk | Local disk with synchronous policy WAL | Yes |
default | Synthesized durable local-disk policy | Yes |
memory_data is not ephemeral. It needs a persistent data directory for its
durable artifacts and rebuilds its serving state during recovery.
Recovery sequence
On startup with --data-dir, the engine:
- opens and validates the local catalog state;
- loads committed catalog and parts checkpoints;
- validates immutable part metadata;
- replays the valid WAL suffix in sequence order;
- applies only complete logical commits; and
- restores maintained in-memory structures before serving queries.
CRC and commit boundaries prevent a torn final frame or incomplete multi-record batch from becoming visible. Corruption in the middle of required history is a fatal recovery error, not an invitation to skip data.
Clean versus hard failure
| Failure | Expected handling |
|---|---|
Graceful SIGTERM | Stop intake, complete shutdown lifecycle, perform configured final sync |
| Process crash | Reopen durable files and replay valid WAL; group mode's accepted process-local tail is preserved |
| OS/power crash | Recovery stops at the durable storage frontier; group may lose its unfsynced tail |
| Disk loss/corruption | Restore from an independent verified backup |
Verification procedure
For each production policy:
- insert a uniquely identified canary;
- capture whether the response was successful;
- test clean restart;
- test process termination;
- test hard failure on equivalent storage;
- compare recovered keys and expected frontier; and
- retain the results with the declared acknowledgement/RPO contract.
Do not use filesystem modification times to infer commit order. Native backup PITR uses the unified journal's inclusive sequence and logical commit boundary.