Add the base setup and initial directory structure

AGENTS.md

@@ -4,151 +4,272 @@ This file provides guidance to coding agents collaborating on this repository.
 
 ## Mission
 
-`nix-playground` is a personal learning playground for Nix and flakes.
+`storage-engine-playground` is an experimental Rust project for testing ideas from the FlowLog, DBSP, CRDT-as-query, and Geomerge notes.
-The goal is not production software but clear, runnable, progressively more advanced examples plus prose notes that explain them.
+
+The goal is not production software. The goal is a clear, runnable playground for small prototypes that help answer concrete architecture questions:
+
+- how Datalog-like rules should be parsed, cataloged, planned, and optimized
+- how FlowLog-style planning ideas transfer to a DBSP-oriented frontend
+- how CRDT queries behave under naive plans versus planned relational execution
+- how Geomerge-style laws can compile into maintained violation relations
+- how backend behavior changes across snapshot, DBSP-like, and Differential Dataflow-like execution models
 
 Priorities, in order:
 
-1. Correctness: examples must actually evaluate and build.
+1. Correctness: prototypes must have clear expected outputs and tests.
-2. Clarity: each example teaches one concept; names, comments, and directory structure should make that concept obvious.
+2. Clarity: each module and test should answer one research or engineering question.
-3. Minimality: prefer the shortest flake or expression that demonstrates the idea.
+3. Small scope: prefer narrow experiments over broad engine rewrites.
-4. Accuracy of notes: prose under `notes/` must not describe behavior the examples do not demonstrate.
+4. Explainability: planners should emit inspectable plans, not only executable structures.
-5. Reproducibility: every flake commits its `flake.lock`; nothing depends on ambient state.
+5. Reproducibility: examples should use committed fixtures, deterministic tests, and documented commands.
 
 ## Core Rules
 
-- Use English for code, comments, and prose.
+- Use English for code, comments, tests, and prose.
-- Keep each numbered example self-contained: its own `flake.nix`, own `flake.lock`, no cross-example imports.
+- Treat ignored local reference material as source material only. Do not import copied code into durable modules without an explicit decision.
-- Prefer small, focused changes over broad rewrites across examples.
+- Prefer implementing small vertical slices: parse a subset, build a catalog, plan one rule shape, and test it.
-- Add comments only when they clarify non-obvious Nix behavior (laziness, `rec`, string vs. path, `with` scoping, etc.).
+- Do not build a full Datalog engine before the planning layer is useful and tested.
-- Do not describe Nix features in notes or comments as if they were implemented in an example unless the example actually uses them.
+- Keep source language, relational planning, and backend execution separated.
-- When an example grows beyond one concept, split it into a new numbered directory rather than expanding the existing one.
+- Prefer backend-neutral intermediate structures until a specific backend API requires specialization.
-
+- Add comments only when they explain non-obvious planning, recursion, delta, or storage behavior.
-Quick examples:
+- Treat tests and fixtures as part of the design, not as afterthoughts.
-
-- Good: add `03-multi-system/` that demonstrates `forAllSystems` in isolation.
-- Good: add a `checks` output to an existing flake with a one-line comment explaining what `nix flake check` will do with it.
-- Bad: combine overlays, NixOS modules, and home-manager into one "comprehensive" example.
-- Bad: edit `notes/` to describe an approach no example in the repo uses.
 
 ## Writing Style
 
 - Use Oxford commas in inline lists: "a, b, and c" not "a, b, c".
 - Do not use em dashes. Restructure the sentence, or use a colon or semicolon instead.
-- Avoid colorful adjectives and adverbs. Write "dev shell" not "lightweight dev shell", "overlay" not "flexible overlay".
+- Avoid colorful adjectives and adverbs. Write "storage engine" not "lightweight storage engine", "planner" not "clever planner".
-- Use noun phrases for checklist items, not imperative verbs. Write "input pinning" not "pin inputs".
+- Use noun phrases for checklist items, not imperative verbs. Write "rule catalog construction" not "construct rule catalogs".
-- Headings in Markdown files must be in title case: "Build from Source" not "Build from source". Minor words (a, an, the, and, but, or, for, in, on,
+- Headings in Markdown files must be in title case: "Query Planning" not "Query planning". Minor words (a, an, the, and, but, or, for, in, on, at, to,
-  at, to, by, of) stay lowercase unless they are the first word.
+  by, of) stay lowercase unless they are the first word.
 
 ## Repository Layout
 
-- `NN-<topic>/`: self-contained numbered examples. Each matching top-level directory is a flake root.
+The repository is new and may change. Discover the current layout from the filesystem before editing.
-- `notes/NNN-*.md`: prose companions numbered to match reading order.
-    - Lower note numbers cover shared foundations such as the glossary, the Nix language, and flakes.
-    - Later note numbers may cover specific example tracks or cross-example guides.
-- `Makefile`: discovery-based helpers that run formatting, linting, and `nix flake check` across all examples.
-- `AGENTS.md`: this file.
-- `.pre-commit-config.yaml`, `.editorconfig`, `.gitattributes`, `.gitignore`: repository hygiene.
-- `pyproject.toml`: Python environment metadata used only to install `pre-commit`.
 
-New examples follow `NN-<short-topic>/` where `NN` is a two-digit ordinal.
+Expected durable areas may include:
 
-Do not assume the directory list or note list in this file is exhaustive. The repository is expected to grow over time, and agents should discover the
+- `src/`: Rust source for parser, catalog, planner, execution experiments, and storage prototypes.
-current layout from the filesystem when needed.
+- `tests/`: integration tests for rule planning, evaluation, and storage behavior.
+- `examples/`: small runnable Datalog-like programs or storage scenarios.
+- `fixtures/`: committed input facts and expected outputs.
+- `notes/`: local design notes that belong to this project.
+- `flowlog/`: project-local notes or sketches derived from the FlowLog line of work.
 
-Example ordering should still feel like a readable path from simpler to more involved, but the repository may branch into themed subtracks, for example
+Do not assume this list is exhaustive. If the project grows a different structure, follow the actual codebase and update this file when conventions
-Nix and flake outputs, Haskell with Nix, or future ecosystem-specific tracks. Keep each example focused on one concept even when the broader sequence
+stabilize.
-branches.
 
-## Example Layout Constraints
+## Technical Direction
 
-- Each example owns exactly one `flake.nix` at its root and commits its `flake.lock`.
+The main experimental architecture is:
-- Examples do not import each other. Copy and adapt if a pattern needs to be shown twice.
-- An example may depend only on flakes it declares in its own `inputs`.
-- Prefer `nixpkgs` pinned to `nixos-unstable` for consistency across examples unless the example's point is pinning strategy.
-- Keep the `outputs` attrset flat enough that `nix flake show` reads as a single screen.
-- If an example exposes `checks.<system>.*`, those checks must pass under `nix flake check`.
 
-## Nix and Flake Conventions
+```text
+Datalog-like rules or Geolog-shaped laws
+-> dependency analysis and strata
+-> rule catalog
+-> join graph
+-> relational plan
+-> FlowLog-style optimization
+-> backend lowering
+-> maintained or snapshot outputs
+```
 
-- Target Nix with `experimental-features = nix-command flakes` enabled (already the case on this machine).
+Keep these layers explicit:
-- Prefer `pkgs.mkShell` for dev shells; reach for `mkShellNoCC` only when explaining the distinction.
-- Use `nixpkgs.lib.genAttrs` or `flake-utils.lib.eachDefaultSystem` for multi-system outputs; pick one per example and say which in a comment.
-- Use `follows` to unify transitive `nixpkgs` inputs when pulling in ecosystem flakes.
-- Prefer `inherit` over repetition in attrsets.
-- Avoid top-level `with` statements; keep `with` narrowly scoped to package lists.
-- Format every `.nix` file with `nixfmt` (RFC 166 style) before committing.
 
-## Required Validation
+- **Source Layer**: Datalog-like test programs, CRDT query definitions, and Geomerge-style laws.
+- **Catalog Layer**: rule heads, body atoms, variables, constants, comparisons, negation, and projections.
+- **Planning Layer**: join graphs, join order, antijoin placement, SIP-style filtering, subplan sharing, and physical key choice.
+- **Execution Layer**: snapshot evaluator first, then DBSP-like or Differential Dataflow-like experiments.
+- **Storage Layer**: facts, transactions, rollback, preview state, and violation output integration.
 
-Run these checks for any non-trivial change:
+## FlowLog-Inspired Planning
 
-1. `make fmt-check`
+FlowLog should be treated as a planning reference, not as an automatic dependency.
-2. `make lint`
-3. `make check`
 
-These map to `nixfmt --check`, `statix check` plus `deadnix`, and `nix flake check` across every numbered example.
+Reusable ideas:
 
-For notes-only changes, `make fmt-check` and a manual read-through suffice.
+- rule catalog construction
+- dependency graph and stratification
+- per-rule join graph extraction
+- width-oriented structural planning
+- sideways information passing
+- antijoin scheduling
+- physical key and payload selection
+- shared subplan detection
 
-## First Contribution Flow
+When adapting an idea, write the smallest test that demonstrates the behavior. For example:
 
-Use this sequence for your first change:
+```text
+rule with three positive atoms
+-> catalog variables
+-> join graph
+-> planned join tree
+-> expected textual plan
+```
 
-1. Read the relevant `notes/` file and the nearest existing example.
+## DBSP and Incremental Execution
-2. Add the smallest possible flake or expression demonstrating the new concept.
-3. Add a short header comment in the new `flake.nix` stating what the example teaches.
-4. Run `nix flake check` inside the new example directory.
-5. Run `make fmt-check` and `make lint` from the repository root.
-6. Add or update the matching entry in `notes/` if the concept is not yet covered there.
 
-Example scopes that are good first tasks:
+DBSP-related work should preserve a clean boundary:
 
-- Add `02-package/` with a trivial `stdenv.mkDerivation` and one-line install phase.
+```text
-- Add a `checks` output to `01-devshell/` that asserts a tool is on `$PATH`.
+planned relational IR
-- Add a short section to `notes/003-flakes.md` referencing a newly added example.
+-> DBSP lowering
-- Convert an existing example from a hand-rolled `forAllSystems` to `flake-utils`, or vice versa, with a comment explaining the tradeoff.
+-> maintained output deltas
+```
 
-When the repository contains multiple themed tracks, "the nearest existing example" means the nearest example in the relevant track, not necessarily the
+Do not make DBSP responsible for source-language semantics. The frontend should check supported syntax, stratification, and rule shape before backend
-numerically closest directory overall.
+lowering.
+
+For each DBSP-like experiment, also provide a snapshot oracle when feasible:
+
+```text
+snapshot result == maintained result after each update
+```
+
+Track these measurements when relevant:
+
+- hydration time
+- warm-update time
+- output delta size
+- maintained state size if available
+- sensitivity to join order
+- sensitivity to causal-history depth
+
+## CRDT Query Experiments
+
+Initial CRDT workloads should stay small and explicit:
+
+- multi-value register
+- causal readiness over `pred`
+- list next-element traversal
+- tombstone skipping
+
+Use operation facts shaped like:
+
+```text
+set(replica_id, counter, key, value)
+pred(from_replica_id, from_counter, to_replica_id, to_counter)
+insert(replica_id, counter, parent_replica_id, parent_counter, value)
+remove(replica_id, counter)
+```
+
+Important questions:
+
+- Does the query require recursion, negation, or both?
+- Can antijoins run earlier?
+- Can causal readiness be maintained from a frontier?
+- Does warm-update cost depend on history depth?
+- Does the output need integration into a current view?
+
+## Geomerge-Style Validation Experiments
+
+The first Geomerge-style target is maintained violation detection for supported relational laws.
+
+A useful lowering shape is:
+
+```text
+required_consequent(x) :- antecedent(x).
+violation(x) :- required_consequent(x), not consequent(x).
+```
+
+Start with:
+
+- foreign-key-style laws
+- totality-as-validation laws
+- equality-as-violation laws
+- multi-atom antecedents without existential witnesses
+
+Exclude at first:
+
+- existential witness generation
+- disjunctive consequents
+- equality saturation
+- model branching
+- full chase behavior
+
+Violation rows should carry enough context for diagnostics:
+
+```text
+law_id
+violation_kind
+relation_or_consequent
+bound_variable_values
+```
+
+## Rust Conventions
+
+- Prefer small modules with explicit data structures over large generic abstractions.
+- Use enums and structs to model rule syntax, catalog entries, plan nodes, and execution results.
+- Prefer typed identifiers for relation names, variable names, rule ids, and field positions when it improves clarity.
+- Keep parser errors and unsupported-feature errors explicit.
+- Avoid panics in library code except for internal invariants that tests already cover.
+- Use deterministic ordering for plans and diagnostics so tests are stable.
+- Prefer simple snapshot evaluators as correctness oracles before optimizing.
 
 ## Testing Expectations
 
-- This repository has no runtime test suite; "tests" are `nix flake check` outcomes and successful builds of each example's default output.
+Add tests for every non-trivial behavior.
-- Any example that exposes non-trivial behavior (a derivation, a module) should expose a `checks.<system>.*` attribute that `nix flake check`
+
-  exercises.
+Recommended test groups:
-- Do not merge changes that regress `make check`.
+
+- parser acceptance and rejection
+- rule catalog construction
+- dependency graph and strata
+- join graph construction
+- structural planning
+- antijoin scheduling
+- SIP-style filtering
+- snapshot evaluation
+- maintained-output equivalence
+- CRDT fixtures
+- Geomerge-style violation fixtures
+
+Tests should prefer small facts with readable expected outputs. Avoid large benchmark fixtures unless the test is explicitly performance-oriented.
+
+## Required Validation
+
+Use the repository's actual tooling. At the time this file was written, the copied `Makefile` is still Nix-playground-oriented and may not match this
+project. Do not assume `make check` is meaningful until the Makefile is updated for this repository.
+
+For Rust changes, prefer:
+
+1. `cargo fmt`
+2. `cargo clippy --all-targets --all-features`
+3. `cargo test --all-targets --all-features`
+
+If the project does not yet have a `Cargo.toml`, record that validation was not available.
+
+For Markdown-only changes, run a manual read-through and check that headings follow the writing style.
 
 ## Change Design Checklist
 
 Before coding:
 
-1. Identify which existing example or notes file the change belongs to, or whether it needs a new `NN-<topic>/`.
+1. Problem statement and target question
-2. Confirm the change teaches one concept, not several.
+2. Existing module or new module decision
-3. Confirm `nixpkgs` input choice is consistent with surrounding examples.
+3. Snapshot oracle or expected output
+4. Supported and unsupported feature boundary
+5. Small fixture or example shape
 
 Before submitting:
 
-1. Verify `make fmt-check`, `make lint`, and `make check` pass.
+1. Formatting status
-2. Verify every modified flake's `flake.lock` is committed.
+2. Test status
-3. Verify `notes/` accurately reflects what the examples now demonstrate.
+3. Unsupported cases documented
+4. No durable references to ignored local paths
+5. Notes or examples updated when behavior changes
 
-## Review Guidelines (P0/P1 Focus)
+## Review Guidelines
 
-Review output should be concise and only include critical issues.
+Review output should prioritize correctness and experiment quality.
 
-- `P0`: must-fix defects (a flake fails to evaluate, an example documents the wrong mechanism, notes contradict the code).
+- `P0`: must-fix defects, such as incorrect query results, invalid rollback behavior, unsupported syntax accepted silently, or tests that cannot run.
-- `P1`: high-priority defects (eval warnings, missing `flake.lock`, unpinned or inconsistent inputs, misleading comment).
+- `P1`: high-priority defects, such as nondeterministic plans, unclear unsupported-feature errors, missing snapshot oracle for a planner change, or
-
+  misleading notes.
-Do not include:
+- `P2`: useful follow-up, such as additional fixtures, clearer diagnostics, or broader benchmark coverage.
-
-- style-only nitpicks,
-- praise or summary of what is already good,
-- exhaustive restatement of the patch.
 
 Use this review format:
 
-1. `Severity` (`P0`/`P1`)
+1. `Severity` (`P0`/`P1`/`P2`)
 2. `File:line`
 3. `Issue`
 4. `Why it matters`
@@ -156,28 +277,29 @@ Use this review format:
 
 ## Practical Notes for Agents
 
-- Prefer targeted edits over broad mechanical rewrites across examples.
+- Read the relevant durable project notes before changing architecture.
-- If two examples disagree on a convention, prefer the newer one and update the older example in a dedicated commit.
+- Treat copied papers, cloned repositories, and generated files in ignored local paths as reference material only.
-- When uncertain whether a concept deserves its own example, start by expanding the notes; promote to an example once the idea stabilizes.
+- Prefer a planning-only prototype before backend integration.
-- Keep presentational prose in `notes/`. Keep runnable material in numbered directories. Do not cross the streams.
+- Prefer textual plan explanations in early tests. They make the planner easier to debug.
-- Prefer layout guidance based on naming patterns and discovery, not hard-coded counts of examples or notes. If you need the current tree, inspect it.
+- Keep backend comparison fair: same rule, same input facts, same expected output.
-- Keep user-facing naming consistent with the repository name: `nix-playground`. The directory spelling `nix-playgraound` is intentional and should
+- Keep transaction and rollback behavior explicit for validation experiments.
-  not be "fixed".
+- When the Makefile becomes project-specific, update this file's validation section.
 
 ## Commit and PR Hygiene
 
-- Keep commits scoped to one logical change: one example, one notes update, one convention shift.
+- Keep commits scoped to one logical change: parser, catalog, planner, evaluator, fixture, note, or tooling.
-- Commit `flake.lock` in the same commit that introduces or updates the `flake.nix` it belongs to.
+- Do not mix broad formatting churn with semantic changes.
 - PR descriptions should include:
-    1. what concept the change teaches or clarifies,
+    1. the experiment or feature being tested,
-    2. which example directories or notes files are affected,
+    2. the source rules or fixtures affected,
-    3. any new `inputs` added and why,
+    3. the expected behavior,
-    4. output of `make check` (pass/fail).
+    4. validation commands and results,
+    5. known unsupported cases.
 
 Suggested PR checklist:
 
-- [ ] `make fmt-check` passes
+- [ ] `cargo fmt` passes, if applicable
-- [ ] `make lint` passes
+- [ ] `cargo clippy --all-targets --all-features` passes, if applicable
-- [ ] `make check` passes
+- [ ] `cargo test --all-targets --all-features` passes, if applicable
-- [ ] `flake.lock` committed for every new or updated `flake.nix`
+- [ ] Snapshot oracle or expected output included for planner behavior
-- [ ] Notes updated where the change introduces or changes a concept
+- [ ] Unsupported cases documented

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md