Troubleshooting

Before changing state, record:

• UTC time and incident start;
• artifact digest/build revision and startup arguments;
• recent startup/runtime logs;
• protocol-ping result;
• system.queries, query_log, events, metrics, parts, subscriptions, transforms, and memory_profile;
• disk/cgroup/process state; and
• the newest committed backup manifest.

Cannot connect

1. Confirm the process is running and recovery has completed.
2. Read the actual bind address and port from the startup log.
3. Remember native defaults are 127.0.0.1:7510, while the container profile commonly uses 0.0.0.0:7777 internally.
4. Check host-side port publication and closed-network routing.
5. Run protocol ping; a successful TCP handshake alone does not prove protocol health.
6. Check connection and worker caps plus handshake/frame/write timeouts.

Do not delete the data directory to make startup faster.

Data disappeared after restart

The most common cause is a process started without --data-dir or a table created with ephemeral_memory.

1. Compare the current and previous startup arguments.
2. Confirm the expected volume is mounted at the same path and is not empty.
3. Inspect the startup log for data_dir=<memory>, recovery errors, or a different path.
4. Check SHOW TABLES and SHOW CREATE TABLE for surviving catalog state and policies.
5. If the durable path is lost/corrupt, stop and restore a verified manifest into a new empty target.

Never point the process at a guessed directory or merge two data directories.

Server busy or query timeout

1. Inspect live elapsed_us, queue_us, memory, rows, disk, and operator timings.
2. Inspect recent ok='0' query-log errors.
3. Cancel a confirmed runaway query after preserving evidence.
4. Reduce intake or expensive concurrency.
5. Fix missing pruning, unbounded joins/aggregates/sorts, or oversized output.
6. Change admission limits only after a representative load test.

A larger wait queue converts rejection into latency and memory; it does not add execution capacity.

Writes slow or reject under memory pressure

1. Check memory and committed-tail gauges.
2. Check memory_governor_ramp_engagements and memory_governor_blocks deltas.
3. Capture system.memory_profile and matching heap file if present.
4. Check flush progress, disk latency/capacity, sealed bytes, and spill state.
5. Check high-memory queries and lagging stream subscribers.
6. Reduce ingest until the tier returns to Normal.

Do not disable the governor during an incident.

No parts appear or tail does not drain

• ephemeral_memory intentionally has no durable part contract.
• memory_data serves from memory; inspect its durable policy/WAL path rather than assuming the table is ephemeral.
• Confirm --data-dir and the target policy/resource are valid.
• Check flush thresholds and polling; 0 can disable individual triggers.
• Check disk space, ownership, I/O errors, and policy drift.
• Compare committed-tail rows/bytes with system.parts progress.
• Inspect compaction/flush logs before restarting.

Part count or disk use keeps growing

1. Confirm compaction is enabled (--compact-poll-ms is not zero).
2. Inspect active jobs, queued shards, completed merges, and I/O throttle.
3. Check free space for compaction output and movement.
4. Confirm TTL/policy definitions from the catalog, not a stale YAML file.
5. Identify active scan epochs delaying physical part retirement.
6. Tune only after foreground-latency and convergence measurements.

Stream is lagging, overflowing, or disconnected

SELECT subscription_id, query_id, client, psi_source_relation,
       psi_pending_events, psi_pending_bytes, psi_overflow_count,
       psi_delivery_lag_p95_us, psi_delivery_lag_p99_us
FROM system.subscriptions
ORDER BY psi_pending_bytes DESC;

Fix downstream consumption, transport, and query cost. A disconnect is terminal for that subscription. Reopen only under an application retry/reconciliation policy; the current engine does not promise resume-token continuity.

If the source was dropped, the terminal event is intentional. A recreated table with the same name is a new generation.

Storage topology drift

At boot, YAML objects absent from the catalog are imported. Existing catalog objects win and differences warn.

1. Compare SHOW STORAGE RESOURCES/POOLS/POLICIES with the file.
2. Decide which catalog change is intended.
3. Use storage DDL to converge the running catalog.
4. Drain before resource removal.
5. Do not repeatedly edit YAML expecting it to overwrite existing objects.

Backup creation fails

Read the exact error. Missing checkpoint parts, WAL gaps/corruption, multi-root/mirror placement, inaccessible files, and exhausted capture retries are deliberate fail-closed conditions.

Repair source/storage health before retrying. Never remove a referenced object or edit a manifest to make the command pass.

Restore will not start

• Target must be empty or a matching tool-created restore target.
• Manifest, checksum, catalog identity, objects, checkpoints, and WAL chain must validate.
• An interrupted target remains RESTORE_IN_PROGRESS and can resume only with the same manifest/sequence.
• A successful target has RESTORE_READY.

Do not start NYXDB, rename markers, or copy missing objects manually. Restore to a new empty directory if identity differs.

Upgrade startup fails

1. Stop retry loops and preserve logs plus the untouched backup repository.
2. Confirm the artifact and on-disk transition were supported by release notes.
3. Do not run an older binary against a directory the new revision may have modified unless backward-open compatibility is explicit.
4. Restore the verified pre-upgrade manifest into a new empty target.
5. Require RESTORE_READY, then start the old pinned artifact and validate.

See Upgrade compatibility.

Escalation bundle

Provide:

• exact reproduction or incident timeline;
• build/artifact identity and full startup configuration;
• minimal affected schema and query;
• relevant system-table snapshots;
• logs spanning the first symptom;
• storage/cgroup/kernel facts;
• backup/restore manifest identifiers; and
• a focused profile or thread dump when SQL telemetry cannot attribute the issue.

Remove unrelated application data while preserving identifiers, sequence frontiers, timings, and error text needed for diagnosis.