Continuous transforms

Continuous transforms are Experimental. The current implementation maintains one local target table from one local source inside EngineHost. It is not a distributed stream-processing service.

Define source and target

The target must already exist, must be keyed, and the transform body must project its primary key:

CREATE TABLE raw_prices (
  symbol String,
  price Float64
) SETTINGS storage_policy = 'default';

CREATE TABLE latest_prices (
  symbol String,
  price Float64,
  PRIMARY KEY (symbol)
) SETTINGS mode = 'keyed', storage_policy = 'default';

CREATE TRANSFORM price_current
INTO latest_prices AS
SELECT symbol, price
FROM raw_prices;

The current supported start position is latest, and it is the default:

CREATE TRANSFORM price_current
INTO latest_prices AS
SELECT symbol, price
FROM raw_prices
SETTINGS start = 'latest';

start='earliest' full backfill is rejected in the current revision.

A fresh registration first evaluates the body over the source's currently readable state, writes that derived result to the keyed target, captures the source boundary, and then follows later changes. For a keyed source, this re-derives collapsed latest state rather than replaying every historical version.

Delivery and recovery

The transform registers an engine-owned PSI subscriber for the source's exact stream:key route. Target writes use the normal table path; transforms do not bypass storage, visibility, or durability rules.

The definition and paused state survive local restart. A live transform records its reflected position and catches up locally after restart. This is not a cross-node recovery or failover guarantee.

Lifecycle

SHOW TRANSFORMS;
SHOW CREATE TRANSFORM price_current;

ALTER TRANSFORM price_current PAUSE;
ALTER TRANSFORM price_current RESUME;

DROP TRANSFORM price_current;

Pausing detaches the runtime subscriber and persists the paused definition. Writes that arrive while paused do not update the target until a supported resume catch-up occurs. Dropping a transform removes its definition and runtime but leaves the target table intact.

Observe

SELECT name, source, target, state, start,
       reflected_position, lag, rows_emitted, last_error
FROM system.transforms
ORDER BY name;

SHOW TRANSFORMS is the compact catalog view. system.transforms adds the runtime position, lag, counters, and latest error.

Composition and safety

Transforms may form local source-to-target chains. Creation is rejected when it would introduce a dependency cycle, target an incompatible table, omit a target key, or use an unsupported body. A v1 body reads exactly one source and may use stateless projection, filtering, and scalar arithmetic. Joins, derived tables, CTEs, UNION, aggregation/GROUP BY/HAVING, ORDER BY, LIMIT, OFFSET, DISTINCT, scalar subqueries, and temporal or HISTORICAL reads are rejected.

Do not treat an accepted dataflow ADR as evidence that arbitrary nested DAGs, distributed watermarks, or cross-node replay are implemented. The capability matrix classifies the present transform and materialized-view surface as Experimental.