Add two new note files (mianly technical notes on usage)

2026-05-20 15:36:14 +02:00 · 2026-05-20 15:36:14 +02:00 · 99c30190a8
commit 99c30190a8
parent 2d3c02315a
2 changed files with 768 additions and 0 deletions
--- a/flowlog/004-flowlog-technical-planning-notes.md
+++ b/flowlog/004-flowlog-technical-planning-notes.md
@ -0,0 +1,388 @@
+# FlowLog Technical Planning Notes
+
+A technical note on the FlowLog planning layer: catalogs, transformations, key-value shapes, and recursive execution.
+
+---
+
+## Short Answer
+
+FlowLog's most reusable technical idea is not the Datalog syntax or the Differential Dataflow backend. It is the planner boundary between them.
+
+The planner turns a rule into a sequence of typed relational transformations:
+
+```text
+rule atoms and variables
+-> catalog metadata
+-> collection signatures
+-> transformation flows
+-> physical operator choices
+```
+
+That layer is useful because it records how variables move through projections, filters, joins, antijoins, and recursive strata before the backend starts maintaining state.
+
+---
+
+## Catalog as Rule Metadata
+
+The catalog is the rule-level semantic summary.
+
+For each rule, it needs to know:
+
+- the head relation
+- the body atoms
+- which atoms are positive
+- which atoms are negated
+- which atoms are core join inputs
+- where each variable appears
+- which constants constrain fields
+- which comparisons constrain tuples
+- which output fields are projected into the head
+
+This is the information needed to go from syntax to a relational plan.
+
+For example:
+
+```text
+violation(x, z) :-
+    A(x, y),
+    B(y, z),
+    not C(x, z),
+    x != z.
+```
+
+The catalog should make the following facts explicit:
+
+- `A` and `B` are positive join inputs.
+- `C` is a negated input.
+- `y` joins `A` to `B`.
+- `(x, z)` is needed for the output and for the antijoin against `C`.
+- `x != z` is a filter after both variables are available.
+
+Without this catalog, the executor has to rediscover planning information from rule syntax.
+
+---
+
+## Collection Shapes
+
+FlowLog lowers logical relations into physical collection shapes.
+
+The main shapes are:
+
+```text
+row
+key
+key-value
+```
+
+A row collection is a plain tuple relation.
+
+A key-only collection is useful for semijoins and antijoins. It represents membership of keys.
+
+A key-value collection is useful for joins. The key is the join attribute set, and the value is the payload carried forward.
+
+The same logical relation may need several physical views:
+
+```text
+Arc(x, y)
+-> row view:       (x, y)
+-> key view:       key=(x)
+-> key-value view: key=(x), value=(y)
+-> key-value view: key=(y), value=(x)
+```
+
+This is a central planning choice. The key determines which arrangement or maintained index the backend can use.
+
+---
+
+## Transformation Types
+
+FlowLog's transformations separate unary reshaping from binary combination.
+
+Unary transformations include:
+
+- row to row
+- row to key
+- row to key-value
+- key-value to key-value
+- key-value to key
+
+These cover:
+
+- projection
+- filtering
+- constant checks
+- equality checks
+- comparison checks
+- arranging a relation by a join key
+- dropping fields that are no longer needed
+
+Binary transformations include:
+
+- key join key
+- key-value join key
+- key-value join key-value
+- cartesian product
+- key antijoin key
+- key-value antijoin key
+
+These cover joins and negation. The planner must choose both the inputs and the output shape.
+
+---
+
+## Transformation Flow
+
+A transformation flow records how input fields become output fields.
+
+For a unary transformation, the flow answers:
+
+```text
+Which input fields form the new key?
+Which input fields remain as value?
+Which constants and comparisons filter rows?
+```
+
+For a binary transformation, the flow answers:
+
+```text
+Which fields came from the left input?
+Which fields came from the right input?
+Which joined fields are retained?
+Which new key should the output use?
+Which payload fields must continue to the next step?
+```
+
+This matters because Datalog variables are logical names, but the backend sees tuple positions.
+
+The planner's job is to keep those two worlds aligned.
+
+---
+
+## Join Graph
+
+A rule body induces a join graph.
+
+Atoms are nodes. Shared variables are edges or hyperedges between atoms.
+
+Example:
+
+```text
+R(a, b, c) :-
+    A(a, x),
+    B(x, y),
+    C(y, c).
+```
+
+The join graph is a chain:
+
+```text
+A --x-- B --y-- C
+```
+
+A rule like this is sensitive to join order:
+
+```text
+R(a, d) :-
+    A(a, x),
+    B(x, y),
+    C(y, z),
+    D(z, d).
+```
+
+Joining `A` with `D` first is a cross product. Joining adjacent atoms first preserves bindings and reduces intermediate results.
+
+FlowLog's structural planning uses variable overlap to choose a plan tree that keeps joins connected and intermediate width smaller.
+
+---
+
+## Width-Oriented Planning
+
+FlowLog's planner is robustness-oriented rather than fully cost-based.
+
+A conventional cost model needs statistics:
+
+- relation sizes
+- distinct counts
+- skew
+- selectivity
+- correlation
+
+Recursive Datalog makes those estimates unstable because intermediate relations change across fixed-point iterations.
+
+FlowLog instead uses structural signals:
+
+- how many variables two atoms share
+- how many variables an intermediate result must carry
+- whether a candidate plan creates disconnected joins
+- how deep the plan tree becomes
+
+This is not guaranteed to be optimal. It is meant to avoid obviously bad plans.
+
+That is a good fit for DBSP-backed work too, because a bad plan becomes maintained operator state.
+
+---
+
+## Antijoin Timing
+
+Negated atoms become antijoins.
+
+An antijoin can only run after all of its variables are bound by prior positive atoms.
+
+Example:
+
+```text
+missing_src(graph, src) :-
+    edge(graph, src, dst),
+    not vertex(graph, src).
+```
+
+The antijoin against `vertex(graph, src)` can run immediately after `edge` because both `graph` and `src` are available.
+
+In a larger rule:
+
+```text
+bad(x, z) :-
+    A(x, y),
+    B(y, z),
+    C(z, w),
+    not D(x, z).
+```
+
+The antijoin against `D(x, z)` can run after `A` and `B`; it does not need to wait for `C`. Running it earlier may reduce the input to the later join with `C`.
+
+This is the same issue as antijoin pushdown in the DBSP CRDT note.
+
+---
+
+## Sideways Information Passing
+
+Sideways information passing is semijoin-style filtering across a rule.
+
+The intuition is:
+
+```text
+derive useful keys
+-> filter another relation to those keys
+-> join less data
+```
+
+Example:
+
+```text
+Reach(y) :- Reach(x), Arc(x, y).
+```
+
+If the current delta contains only a small set of `Reach(x)` values, then `Arc` only needs edges whose source is in that set. A semijoin can prefilter `Arc` before the recursive join.
+
+For CRDT causal readiness, this suggests a physical plan centered on frontier operations:
+
+```text
+new ready operations
+-> candidate outgoing pred edges
+-> newly ready operations
+```
+
+rather than a plan that repeatedly starts from roots.
+
+---
+
+## Recursive Strata
+
+Recursive rules require fixed-point execution.
+
+FlowLog groups recursive rules into recursive strata, then executes them inside an iterative dataflow scope.
+
+The important design point is that a recursive stratum can contain several rules deriving related IDBs. The planner must know:
+
+- which IDBs are loop variables
+- which relations enter the recursive scope from earlier strata
+- which outputs must be collected after convergence
+- which intermediate arrangements are useful across iterations
+
+For DBSP, this maps to recursive circuits with feedback and delay. The frontend still needs the same rule-level information before it can produce a good circuit.
+
+---
+
+## Subplan Sharing
+
+Multiple rules may derive the same relation, or several relations may reuse the same intermediate computation.
+
+Example:
+
+```text
+required_src(g, s) :- edge(g, s, d).
+required_dst(g, d) :- edge(g, s, d).
+```
+
+Both rules scan or project from `edge`.
+
+In larger Geomerge theories, many violation rules may share antecedent fragments. A planner should be able to notice common subplans:
+
+```text
+common_antecedent(x, y)
+-> violation_a(x)
+-> violation_b(y)
+```
+
+FlowLog's explicit rule plans and collection signatures are a useful place to represent this sharing.
+
+---
+
+## Physical Key Choice
+
+Backend performance depends on key choice.
+
+For a join:
+
+```text
+R(x, z) :- A(x, y), B(y, z).
+```
+
+both `A` and `B` should be arranged by `y`.
+
+For a later join:
+
+```text
+S(x, w) :- R(x, z), C(z, w).
+```
+
+the output of the first join may need to be arranged by `z`, not by `x`.
+
+That means the planner should choose output keys based on the next operation, not only the current operation.
+
+This is one reason a simple relational algebra tree is not enough. The physical plan needs key and payload annotations.
+
+---
+
+## Transfer to a DBSP Frontend
+
+A DBSP frontend inspired by FlowLog should probably have these data structures:
+
+- relation schemas
+- rule catalogs
+- variable occurrence maps
+- dependency graph
+- strata
+- join graph per rule
+- logical relational plan
+- physical key annotations
+- backend lowering rules
+
+The lowering should treat DBSP as the maintained execution backend:
+
+```text
+projection -> DBSP projection
+selection -> DBSP filter
+join -> DBSP join with maintained state
+antijoin -> DBSP antijoin or difference plan
+union -> DBSP addition or union
+distinct -> DBSP distinct
+recursion -> DBSP fixed-point circuit
+```
+
+The key point is that DBSP should receive an already planned circuit, not raw Datalog text.
+
+---
+
+## Changelog
+
+* **May 20, 2026** -- First version created from the FlowLog implementation and DBSP synergy notes.
--- a/flowlog/005-using-flowlog-ideas.md
+++ b/flowlog/005-using-flowlog-ideas.md
@ -0,0 +1,380 @@
+# Using FlowLog Ideas
+
+A practical note on how FlowLog or FlowLog-style planning could be used for the local DBSP, CRDT, and Geomerge work.
+
+---
+
+## Short Answer
+
+The most useful way to use FlowLog is as a planning reference, not as a direct dependency.
+
+There are three possible levels of use:
+
+```text
+Level 1: run FlowLog examples to learn workload behavior
+Level 2: borrow FlowLog planning ideas for a DBSP frontend
+Level 3: compare DBSP and Differential Dataflow backends on the same Datalog programs
+```
+
+The practical near-term path is Level 2. Use FlowLog's catalog, join planning, antijoin scheduling, and SIP ideas to design a better compiler layer before DBSP.
+
+---
+
+## Initial Non-Goals
+
+The first step should not be replacing the DBSP backend with FlowLog.
+
+That would conflate two separate questions:
+
+- Which incremental backend should maintain deltas?
+- Which frontend planner should produce the backend plan?
+
+The DBSP notes are already about DBSP as a formal view-maintenance backend. FlowLog is more useful as a guide for the missing frontend and optimizer.
+
+The first step should not be adopting FlowLog's syntax as the durable source language either. Geomerge and Geolog already have their own source concepts. Datalog should be an intermediate or testing language unless the user-facing language decision is explicit.
+
+---
+
+## Use Case 1: Better CRDT Query Planning
+
+The CRDT queries in the DBSP notes include:
+
+- multi-value register queries
+- causal-readiness queries
+- list traversal queries
+- tombstone-skipping queries
+
+The multi-value register is already simple enough:
+
+```text
+set + pred -> overwritten -> visible values
+```
+
+The planning value is higher for causal readiness:
+
+```text
+pred graph
+-> roots
+-> recursive reachability
+-> ready operations
+```
+
+and list traversal:
+
+```text
+insert tree
+-> first child
+-> next sibling
+-> ancestor sibling
+-> next element
+-> next visible element
+```
+
+These queries contain several recursive or join-heavy rules. FlowLog-style planning can help by choosing join keys, pushing antijoins earlier, and adding semijoin filters around the current frontier.
+
+The concrete experiment:
+
+```text
+causal-readiness Datalog rules
+rule catalog
+naive plan
+FlowLog-style planned version
+DBSP hydration and warm-update comparison
+```
+
+The success criterion is not only lower total runtime. It is lower dependence on causal-history depth for small warm updates.
+
+---
+
+## Use Case 2: Geomerge Violation Planning
+
+The Geomerge integration note proposes compiling supported laws into violation relations.
+
+For simple laws, this is direct:
+
+```text
+missing_src(g, s) :-
+    edge(g, s, d),
+    not vertex(g, s).
+```
+
+For larger laws, the compiler needs a planner:
+
+```text
+violation(vars) :-
+    antecedent_atom_1(...),
+    antecedent_atom_2(...),
+    antecedent_atom_3(...),
+    not consequent_atom(...).
+```
+
+FlowLog-style catalogs would help the compiler answer:
+
+- which variables are introduced by each atom
+- which atoms join on which variables
+- when each negated consequent can be checked
+- which projected values are needed for the violation row
+- whether two laws share a common antecedent
+
+The concrete experiment:
+
+```text
+one Geomerge fixture theory
+Datalog-like rule per supported law
+join graph per rule
+planned relational tree
+comparison with the current direct validator's binding order
+```
+
+This can be useful before any DBSP integration exists, because it tests whether the compiler can understand the law shape.
+
+---
+
+## Use Case 3: Backend Comparison
+
+FlowLog can also be used as a comparison point for DBSP.
+
+The fair comparison is not:
+
+```text
+FlowLog product vs DBSP product
+```
+
+The useful comparison is:
+
+```text
+same Datalog query
+same input facts
+same output relation
+different backend lowering
+```
+
+Candidate workloads:
+
+- transitive closure
+- causal readiness
+- list next-element traversal
+- missing foreign-key violations
+- multi-atom Geomerge antecedents
+
+The comparison should measure:
+
+- hydration time
+- warm-update time
+- memory use
+- sensitivity to join order
+- output delta size
+- ease of rollback or preview execution
+
+This helps decide whether DBSP needs FlowLog-like planning, whether Differential Dataflow is better for some recursive workloads, or whether a hybrid batch-plus-incremental strategy is needed.
+
+---
+
+## Use Case 4: Test Corpus for Datalog Lowering
+
+FlowLog's examples suggest a useful test corpus shape.
+
+A local Datalog-to-DBSP frontend should include small programs for:
+
+- reachability
+- transitive closure
+- connected components
+- antijoin checks
+- aggregation checks
+- CRDT multi-value register
+- CRDT causal readiness
+- CRDT list traversal
+- Geomerge-style violation detection
+
+Each test should define:
+
+- input schemas
+- input facts
+- expected output facts
+- expected output deltas for at least one update
+- whether recursion or negation is used
+- whether the program should be accepted or rejected
+
+This gives a better foundation than testing only one CRDT or one Geomerge law.
+
+---
+
+## First Prototype
+
+The first useful prototype should be small.
+
+A planning-only tool:
+
+```text
+Datalog-like rule text
+-> parsed rules
+-> dependency graph
+-> strata
+-> rule catalog
+-> join graph
+-> planned relational tree
+```
+
+It does not need to run DBSP at first.
+
+The output can be textual:
+
+```text
+rule: missing_src
+positive atoms: edge
+negative atoms: vertex
+join graph: none
+plan:
+  scan edge
+  project (graph, src)
+  antijoin vertex on (graph, src)
+  emit violation row
+```
+
+For recursive rules, the output can identify the loop:
+
+```text
+recursive stratum:
+  ready
+
+base:
+  roots -> ready
+
+step:
+  ready join pred on operation id
+  project successor operation id
+```
+
+This prototype would validate the compiler shape before depending on a backend API.
+
+---
+
+## Second Prototype
+
+The second prototype should lower a narrow subset to DBSP.
+
+Supported subset:
+
+- relation declarations
+- positive atoms
+- equality joins
+- constants
+- simple comparisons
+- stratified negation
+- union of repeated rule heads
+- one recursive IDB at a time
+
+Excluded subset:
+
+- aggregation
+- mutual recursion
+- disjunction
+- existential generation
+- equality saturation
+- custom scalar functions
+
+The target workloads:
+
+- `missing_src` and `missing_dst`
+- multi-value register
+- transitive closure
+- causal readiness
+
+This subset is enough to test the important bridge:
+
+```text
+planned rules -> DBSP-maintained outputs
+```
+
+---
+
+## Data Model Decisions
+
+Several decisions should be made explicitly before implementation.
+
+**Set or Multiset Semantics**: CRDT operation facts are usually set-like. DBSP uses Z-set weights internally. The frontend should define when `distinct` is applied.
+
+**Operation Identity**: CRDT examples use `(replica_id, counter)`. The planner should treat this pair either as two scalar fields or as one logical key with two physical fields.
+
+**Violation Rows**: Geomerge violations should include enough context for error messages, not just a boolean.
+
+**Output Integration**: DBSP emits deltas. Applications often need an integrated current view. The runtime boundary should say who owns that integration.
+
+**Rollback**: Geomerge validation needs preview or rollback behavior. If using weighted deltas, inverse deltas are plausible but must stay transactionally coupled to storage.
+
+---
+
+## Evaluation Plan
+
+The evaluation should separate correctness from performance.
+
+Correctness checks:
+
+```text
+planned evaluation == naive snapshot evaluation
+DBSP maintained result == snapshot result
+failed Geomerge transaction leaves no checker drift
+```
+
+Performance checks:
+
+```text
+hydration time
+warm-update time
+memory used by maintained state
+number of output delta rows
+history-depth sensitivity
+join-order sensitivity
+```
+
+The most important performance test is causal readiness:
+
+```text
+large causal history
+ small new update
+-> does update cost grow with history depth?
+```
+
+If the answer is yes, the frontend needs frontier-aware planning or a different physical representation.
+
+---
+
+## Decision Points
+
+The main decision points are:
+
+- whether to implement a Datalog frontend or compile directly from Geolog laws
+- whether the relational IR should be FlowLog-like, DBSP-like, or custom
+- whether recursive planning should support mutual recursion early
+- whether SIP should be automatic, directive-controlled, or both
+- whether hydration should use the same backend as warm updates
+- whether to persist backend operator state
+- whether to compare against Differential Dataflow for recursive workloads
+
+These decisions should stay separate. Choosing DBSP as the backend does not force a particular Datalog syntax. Choosing a FlowLog-like planner does not force Differential Dataflow as the backend.
+
+---
+
+## Practical Recommendation
+
+The first practical step is a planning-only FlowLog-inspired compiler layer.
+
+The next step is lowering a small subset to DBSP.
+
+After that, FlowLog itself can serve as a comparison backend for the same small programs.
+
+The goal should be:
+
+```text
+one rule frontend
+one relational IR
+two possible execution backends
+```
+
+That architecture would make it possible to test whether performance problems come from the query semantics, the planner, or the backend.
+
+---
+
+## Changelog
+
+* **May 20, 2026** -- First version created from FlowLog, DBSP, CRDT, and Geomerge notes.