Hassan Abedi ee52b850e4 Add a new note file for expressing CRDTs with Datalog

2026-05-18 17:29:50 +02:00

12 KiB

Raw Blame History

Expressing CRDTs with Datalog Lessons

A reading note on the prototype and benchmark lessons from Expressing CRDTs with Datalog.

Short Answer

The thesis makes the CRDT-as-query idea more concrete by building a Rust query engine that translates a Datalog dialect into a relational intermediate representation and executes it incrementally with DBSP.

The missing lesson for the existing DBSP notes is not only that Datalog can express a multi-value register and an RGA-like list. The additional lesson is where the approach spends time:

simple antijoin-shaped CRDT queries can be fast after hydration
causal-readiness queries can remain dependent on the age of the causal history
recursive graph traversal is the main performance risk
list CRDTs can be expressed, but their output shape is a linked relation, not an ordered array
query optimization is required if this is meant to compete with hand-written CRDTs

The prototype supports the architectural direction, but it also narrows the research problem: the hard part is not only expressing CRDT semantics in Datalog. The hard part is making common causal and structural queries incremental in the way applications expect.

What the Prototype Adds

The existing notes already cover the conceptual model:

operation facts -> deterministic Datalog query -> derived CRDT state

The prototype adds an execution model:

Datalog dialect
-> abstract syntax tree
-> predicate dependency graph
-> relational intermediate representation
-> interpreted query plan
-> DBSP circuit
-> output deltas

The query engine is interpreted rather than compiled. The interpreter builds the DBSP circuit from the relational intermediate representation, then DBSP maintains the relational operators over input changes. Scalar expressions, variable lookup, and tuple-field access remain in the interpreter.

That choice matters because it separates two concerns:

DBSP owns incremental relational state.
The custom engine owns Datalog parsing, rule scheduling, schema tracking, scalar expression evaluation, and frontend semantics.

For Geomerge or Geolog, this supports the idea that DBSP should be treated as an incremental relational backend, not as the whole language runtime.

Datalog Dialect Boundaries

The thesis dialect is deliberately smaller than full Datalog.

It supports:

named fields rather than positional-only predicate arguments
explicit extensional database predicates for input schemas
explicit distinct for set semantics
positive atoms
negated atoms
scalar comparisons and arithmetic
self-recursion

It excludes:

mutual recursion
aggregation
multiple main outputs
strong type checking
tuple-valued scalar fields
richer error reporting

The self-recursion restriction is important. It is enough for the examples in the thesis, including causal readiness and list traversal, but it is narrower than stratified Datalog. If a future Geolog or Geomerge integration needs several recursively defined predicates in the same strongly connected component, this prototype shape would need to grow.

The single-output limitation is also important. It reinforces the design choice from the Geomerge integration note: compile many checks into one combined output relation when possible.

Benchmark Split

The thesis separates two performance situations that should stay separate in design discussions.

Hydration is the cold-start case:

stored operation history
-> parse query
-> build circuit
-> feed all existing facts
-> rebuild maintained operator state
-> produce current view

Near-real-time processing is the warm case:

existing circuit and operator state
-> small new batch
-> DBSP step
-> output deltas

This split is useful for local-first systems because a design can be acceptable in one mode and poor in the other. A collaborative editor needs both: startup must not scale badly with document history, and live editing must stay responsive after the document is open.

Key-Value Store without Causal Broadcast

The simplest benchmark is a multi-value register key-value store without causal broadcast.

The query shape is:

overwritten(RepId, Ctr) :-
    pred(RepId, Ctr, _, _).

mvrStore(Key, Value) :-
    set(RepId, Ctr, Key, Value),
    not overwritten(RepId, Ctr).

This is mostly projection plus antijoin. In the near-real-time benchmark, the size of the existing causal history has little effect on processing a small new batch. Processing remains below a quarter millisecond in the reported setup.

This is the favorable case for DBSP-backed CRDTs. The maintained state of the antijoin lets the engine process new facts without scanning the whole history.

The caveat is semantic: this version assumes causal delivery or otherwise accepts intermediate states that can expose causally incomplete operations.

Causal Broadcast Cost

Adding causal readiness changes the problem.

The query must derive which operations are reachable from causal roots:

isCausallyReady(RepId, Ctr) :-
    isRoot(RepId, Ctr).

isCausallyReady(RepId, Ctr) :-
    isCausallyReady(FromRepId, FromCtr),
    pred(FromRepId, FromCtr, RepId, Ctr).

This is a recursive graph traversal. In the hydration benchmark, increasing the causal-history diameter by 1000 operations adds roughly 40 ms for the key-value store with causal broadcast, compared with roughly 2 ms for the version without it.

In the near-real-time benchmark, the causal-broadcast version is still affected by the existing history. Increasing the base diameter by 1000 adds roughly 20 ms even when the new delta is small.

This is the main warning in the thesis. Incremental execution does not automatically mean update cost is independent of history size. If the recursive query walks from roots through a long chain, the warm update can still depend on the age or diameter of the causal graph.

The likely optimization target is causal readiness from current leaves or recent frontier state, not repeated root-to-leaf traversal.

List CRDT Lessons

The thesis implements a list CRDT similar to causal-tree or RGA-style designs.

The input facts are:

insert(RepId, Ctr, ParentRepId, ParentCtr, Value)
remove(ElemId, ElemCtr)

Each insertion points to the element after which it was inserted. Concurrent insertions after the same parent become siblings, and siblings are ordered by operation identifier. The visible order is a depth-first preorder traversal of the insertion tree.

The query derives relations such as:

firstChild
nextSibling
nextSiblingAnc
nextElem
hasValue
nextElemSkipTombstones
nextVisible
listElem

The output is not the final array of values. It is a linked relation:

listElem(PrevRepId, PrevCtr, Value, NextRepId, NextCtr)

The application reconstructs the visible list by walking from the sentinel root. This matters for API design. A DBSP output delta can describe local pointer changes, but an application still needs an integrated view or traversal helper to expose the list in ordinary UI order.

Deletes show the delta behavior clearly. Deleting the last visible element removes one output tuple. Deleting a non-last visible element can remove the element tuple, remove the old successor link, and add a replacement successor link. The output delta is relational, not a high-level list edit.

List Performance

The list hydration benchmark uses consecutive inserts at the end of the list. Increasing the base text length by 10000 insertions adds roughly 200 ms. Loading 50000 operations takes about one second in the reported setup.

The near-real-time benchmark adds bursts of 20 to 100 consecutive insertions on top of a hydrated base text. Processing takes roughly 1.5 ms to 5.25 ms depending on base length and delta size.

The base text length still matters. Increasing the base text length by 10000 adds roughly half a millisecond for warm updates. Increasing the delta by 20 insertions adds roughly 0.4 ms.

This is promising for append-like text workloads, but the benchmark is narrow. It does not settle the cost of high concurrency at one parent, random cursor jumps, many tombstones, or move-like tree operations.

Optimization Agenda

The thesis points to three optimization problems that should be treated as first-class work.

Join Ordering: Datalog rules compile to joins, and join order determines intermediate relation size. For long-running incremental queries, the choice is harder than for a one-shot query because the operator state is tied to the plan.

Antijoin Pushdown: Negative atoms are implemented as antijoins. Scheduling antijoins as soon as their variables are available may reduce intermediate results, similar to predicate pushdown.

Datalog-Level Optimization: Some rewrites may be easier before lowering to relational algebra. The system likely needs both Datalog-level and relational-level optimization passes.

The causal-readiness result suggests one more optimization category:

Frontier-Aware Recursion: Recursive graph queries should use maintained frontier information when possible. A query that is logically expressed as reachability from roots may need a physical plan that checks readiness near current leaves.

Runtime and Storage Implications

The prototype is a computation engine, not a durable database.

A practical system still needs:

durable storage for operation facts
atomic updates across all relations touched by one operation
hydration strategy
checkpointing or persistence for DBSP operator state
rollback or preview support for failed transactions
compaction that preserves query results
partial synchronization semantics
integrated views for application reads

The atomic-update point is easy to miss. A logical CRDT operation may insert into multiple input relations, such as set and pred. The storage layer must avoid exposing a state where only part of the operation has arrived in the query engine.

For Geomerge, this reinforces the need for transaction coupling between storage state and DBSP state.

What This Adds to the DBSP Folder

The earlier notes establish:

CRDT state as a deterministic query over operation facts
DBSP as a model for incremental view maintenance
Geomerge integration through maintained violation relations

This note adds:

the prototype's actual language boundaries
the hydration versus warm-update split
the empirical cost of causal readiness
the linked-relation output shape for list CRDTs
the need for query optimization before production use

The main conclusion is that the approach is plausible, but the default physical plan is not enough. The next useful research step is to identify CRDT query patterns and give them better incremental plans.

Design Questions

Which CRDT queries are antijoin-shaped and likely to be fast under DBSP?
Which CRDT queries require recursive graph traversal?
Can causal readiness be maintained from a frontier instead of recomputed from roots?
Should hydration use DBSP, a batch engine, or persisted DBSP operator state?
What integrated views should the runtime expose for linked outputs such as lists?
Can operation facts be compacted while preserving future query semantics?
How should one logical operation atomically update several input relations?
Does the Datalog frontend need multiple output predicates for real applications?
Which optimizations belong in Datalog, and which belong in relational algebra?
What workloads should be used before comparing this approach with hand-written CRDTs?

Changelog

May 18, 2026 -- First version created from Expressing CRDTs with Datalog.

12 KiB Raw Blame History