From 0e5572d1c8a2bda3e91b4ba62e2e6affa10e92da Mon Sep 17 00:00:00 2001 From: Hassan Abedi Date: Thu, 21 May 2026 15:56:29 +0200 Subject: [PATCH] Add the base setup and initial directory structure --- AGENTS.md | 350 ++++++++++++++++++++++++++++++++++++------------------ CLAUDE.md | 1 + 2 files changed, 237 insertions(+), 114 deletions(-) create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md index 205e1e6..25d9256 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,151 +4,272 @@ This file provides guidance to coding agents collaborating on this repository. ## Mission -`nix-playground` is a personal learning playground for Nix and flakes. -The goal is not production software but clear, runnable, progressively more advanced examples plus prose notes that explain them. +`storage-engine-playground` is an experimental Rust project for testing ideas from the FlowLog, DBSP, CRDT-as-query, and Geomerge notes. + +The goal is not production software. The goal is a clear, runnable playground for small prototypes that help answer concrete architecture questions: + +- how Datalog-like rules should be parsed, cataloged, planned, and optimized +- how FlowLog-style planning ideas transfer to a DBSP-oriented frontend +- how CRDT queries behave under naive plans versus planned relational execution +- how Geomerge-style laws can compile into maintained violation relations +- how backend behavior changes across snapshot, DBSP-like, and Differential Dataflow-like execution models Priorities, in order: -1. Correctness: examples must actually evaluate and build. -2. Clarity: each example teaches one concept; names, comments, and directory structure should make that concept obvious. -3. Minimality: prefer the shortest flake or expression that demonstrates the idea. -4. Accuracy of notes: prose under `notes/` must not describe behavior the examples do not demonstrate. -5. Reproducibility: every flake commits its `flake.lock`; nothing depends on ambient state. +1. Correctness: prototypes must have clear expected outputs and tests. +2. Clarity: each module and test should answer one research or engineering question. +3. Small scope: prefer narrow experiments over broad engine rewrites. +4. Explainability: planners should emit inspectable plans, not only executable structures. +5. Reproducibility: examples should use committed fixtures, deterministic tests, and documented commands. ## Core Rules -- Use English for code, comments, and prose. -- Keep each numbered example self-contained: its own `flake.nix`, own `flake.lock`, no cross-example imports. -- Prefer small, focused changes over broad rewrites across examples. -- Add comments only when they clarify non-obvious Nix behavior (laziness, `rec`, string vs. path, `with` scoping, etc.). -- Do not describe Nix features in notes or comments as if they were implemented in an example unless the example actually uses them. -- When an example grows beyond one concept, split it into a new numbered directory rather than expanding the existing one. - -Quick examples: - -- Good: add `03-multi-system/` that demonstrates `forAllSystems` in isolation. -- Good: add a `checks` output to an existing flake with a one-line comment explaining what `nix flake check` will do with it. -- Bad: combine overlays, NixOS modules, and home-manager into one "comprehensive" example. -- Bad: edit `notes/` to describe an approach no example in the repo uses. +- Use English for code, comments, tests, and prose. +- Treat ignored local reference material as source material only. Do not import copied code into durable modules without an explicit decision. +- Prefer implementing small vertical slices: parse a subset, build a catalog, plan one rule shape, and test it. +- Do not build a full Datalog engine before the planning layer is useful and tested. +- Keep source language, relational planning, and backend execution separated. +- Prefer backend-neutral intermediate structures until a specific backend API requires specialization. +- Add comments only when they explain non-obvious planning, recursion, delta, or storage behavior. +- Treat tests and fixtures as part of the design, not as afterthoughts. ## Writing Style - Use Oxford commas in inline lists: "a, b, and c" not "a, b, c". - Do not use em dashes. Restructure the sentence, or use a colon or semicolon instead. -- Avoid colorful adjectives and adverbs. Write "dev shell" not "lightweight dev shell", "overlay" not "flexible overlay". -- Use noun phrases for checklist items, not imperative verbs. Write "input pinning" not "pin inputs". -- Headings in Markdown files must be in title case: "Build from Source" not "Build from source". Minor words (a, an, the, and, but, or, for, in, on, - at, to, by, of) stay lowercase unless they are the first word. +- Avoid colorful adjectives and adverbs. Write "storage engine" not "lightweight storage engine", "planner" not "clever planner". +- Use noun phrases for checklist items, not imperative verbs. Write "rule catalog construction" not "construct rule catalogs". +- Headings in Markdown files must be in title case: "Query Planning" not "Query planning". Minor words (a, an, the, and, but, or, for, in, on, at, to, + by, of) stay lowercase unless they are the first word. ## Repository Layout -- `NN-/`: self-contained numbered examples. Each matching top-level directory is a flake root. -- `notes/NNN-*.md`: prose companions numbered to match reading order. - - Lower note numbers cover shared foundations such as the glossary, the Nix language, and flakes. - - Later note numbers may cover specific example tracks or cross-example guides. -- `Makefile`: discovery-based helpers that run formatting, linting, and `nix flake check` across all examples. -- `AGENTS.md`: this file. -- `.pre-commit-config.yaml`, `.editorconfig`, `.gitattributes`, `.gitignore`: repository hygiene. -- `pyproject.toml`: Python environment metadata used only to install `pre-commit`. +The repository is new and may change. Discover the current layout from the filesystem before editing. -New examples follow `NN-/` where `NN` is a two-digit ordinal. +Expected durable areas may include: -Do not assume the directory list or note list in this file is exhaustive. The repository is expected to grow over time, and agents should discover the -current layout from the filesystem when needed. +- `src/`: Rust source for parser, catalog, planner, execution experiments, and storage prototypes. +- `tests/`: integration tests for rule planning, evaluation, and storage behavior. +- `examples/`: small runnable Datalog-like programs or storage scenarios. +- `fixtures/`: committed input facts and expected outputs. +- `notes/`: local design notes that belong to this project. +- `flowlog/`: project-local notes or sketches derived from the FlowLog line of work. -Example ordering should still feel like a readable path from simpler to more involved, but the repository may branch into themed subtracks, for example -Nix and flake outputs, Haskell with Nix, or future ecosystem-specific tracks. Keep each example focused on one concept even when the broader sequence -branches. +Do not assume this list is exhaustive. If the project grows a different structure, follow the actual codebase and update this file when conventions +stabilize. -## Example Layout Constraints +## Technical Direction -- Each example owns exactly one `flake.nix` at its root and commits its `flake.lock`. -- Examples do not import each other. Copy and adapt if a pattern needs to be shown twice. -- An example may depend only on flakes it declares in its own `inputs`. -- Prefer `nixpkgs` pinned to `nixos-unstable` for consistency across examples unless the example's point is pinning strategy. -- Keep the `outputs` attrset flat enough that `nix flake show` reads as a single screen. -- If an example exposes `checks..*`, those checks must pass under `nix flake check`. +The main experimental architecture is: -## Nix and Flake Conventions +```text +Datalog-like rules or Geolog-shaped laws +-> dependency analysis and strata +-> rule catalog +-> join graph +-> relational plan +-> FlowLog-style optimization +-> backend lowering +-> maintained or snapshot outputs +``` -- Target Nix with `experimental-features = nix-command flakes` enabled (already the case on this machine). -- Prefer `pkgs.mkShell` for dev shells; reach for `mkShellNoCC` only when explaining the distinction. -- Use `nixpkgs.lib.genAttrs` or `flake-utils.lib.eachDefaultSystem` for multi-system outputs; pick one per example and say which in a comment. -- Use `follows` to unify transitive `nixpkgs` inputs when pulling in ecosystem flakes. -- Prefer `inherit` over repetition in attrsets. -- Avoid top-level `with` statements; keep `with` narrowly scoped to package lists. -- Format every `.nix` file with `nixfmt` (RFC 166 style) before committing. +Keep these layers explicit: -## Required Validation +- **Source Layer**: Datalog-like test programs, CRDT query definitions, and Geomerge-style laws. +- **Catalog Layer**: rule heads, body atoms, variables, constants, comparisons, negation, and projections. +- **Planning Layer**: join graphs, join order, antijoin placement, SIP-style filtering, subplan sharing, and physical key choice. +- **Execution Layer**: snapshot evaluator first, then DBSP-like or Differential Dataflow-like experiments. +- **Storage Layer**: facts, transactions, rollback, preview state, and violation output integration. -Run these checks for any non-trivial change: +## FlowLog-Inspired Planning -1. `make fmt-check` -2. `make lint` -3. `make check` +FlowLog should be treated as a planning reference, not as an automatic dependency. -These map to `nixfmt --check`, `statix check` plus `deadnix`, and `nix flake check` across every numbered example. +Reusable ideas: -For notes-only changes, `make fmt-check` and a manual read-through suffice. +- rule catalog construction +- dependency graph and stratification +- per-rule join graph extraction +- width-oriented structural planning +- sideways information passing +- antijoin scheduling +- physical key and payload selection +- shared subplan detection -## First Contribution Flow +When adapting an idea, write the smallest test that demonstrates the behavior. For example: -Use this sequence for your first change: +```text +rule with three positive atoms +-> catalog variables +-> join graph +-> planned join tree +-> expected textual plan +``` -1. Read the relevant `notes/` file and the nearest existing example. -2. Add the smallest possible flake or expression demonstrating the new concept. -3. Add a short header comment in the new `flake.nix` stating what the example teaches. -4. Run `nix flake check` inside the new example directory. -5. Run `make fmt-check` and `make lint` from the repository root. -6. Add or update the matching entry in `notes/` if the concept is not yet covered there. +## DBSP and Incremental Execution -Example scopes that are good first tasks: +DBSP-related work should preserve a clean boundary: -- Add `02-package/` with a trivial `stdenv.mkDerivation` and one-line install phase. -- Add a `checks` output to `01-devshell/` that asserts a tool is on `$PATH`. -- Add a short section to `notes/003-flakes.md` referencing a newly added example. -- Convert an existing example from a hand-rolled `forAllSystems` to `flake-utils`, or vice versa, with a comment explaining the tradeoff. +```text +planned relational IR +-> DBSP lowering +-> maintained output deltas +``` -When the repository contains multiple themed tracks, "the nearest existing example" means the nearest example in the relevant track, not necessarily the -numerically closest directory overall. +Do not make DBSP responsible for source-language semantics. The frontend should check supported syntax, stratification, and rule shape before backend +lowering. + +For each DBSP-like experiment, also provide a snapshot oracle when feasible: + +```text +snapshot result == maintained result after each update +``` + +Track these measurements when relevant: + +- hydration time +- warm-update time +- output delta size +- maintained state size if available +- sensitivity to join order +- sensitivity to causal-history depth + +## CRDT Query Experiments + +Initial CRDT workloads should stay small and explicit: + +- multi-value register +- causal readiness over `pred` +- list next-element traversal +- tombstone skipping + +Use operation facts shaped like: + +```text +set(replica_id, counter, key, value) +pred(from_replica_id, from_counter, to_replica_id, to_counter) +insert(replica_id, counter, parent_replica_id, parent_counter, value) +remove(replica_id, counter) +``` + +Important questions: + +- Does the query require recursion, negation, or both? +- Can antijoins run earlier? +- Can causal readiness be maintained from a frontier? +- Does warm-update cost depend on history depth? +- Does the output need integration into a current view? + +## Geomerge-Style Validation Experiments + +The first Geomerge-style target is maintained violation detection for supported relational laws. + +A useful lowering shape is: + +```text +required_consequent(x) :- antecedent(x). +violation(x) :- required_consequent(x), not consequent(x). +``` + +Start with: + +- foreign-key-style laws +- totality-as-validation laws +- equality-as-violation laws +- multi-atom antecedents without existential witnesses + +Exclude at first: + +- existential witness generation +- disjunctive consequents +- equality saturation +- model branching +- full chase behavior + +Violation rows should carry enough context for diagnostics: + +```text +law_id +violation_kind +relation_or_consequent +bound_variable_values +``` + +## Rust Conventions + +- Prefer small modules with explicit data structures over large generic abstractions. +- Use enums and structs to model rule syntax, catalog entries, plan nodes, and execution results. +- Prefer typed identifiers for relation names, variable names, rule ids, and field positions when it improves clarity. +- Keep parser errors and unsupported-feature errors explicit. +- Avoid panics in library code except for internal invariants that tests already cover. +- Use deterministic ordering for plans and diagnostics so tests are stable. +- Prefer simple snapshot evaluators as correctness oracles before optimizing. ## Testing Expectations -- This repository has no runtime test suite; "tests" are `nix flake check` outcomes and successful builds of each example's default output. -- Any example that exposes non-trivial behavior (a derivation, a module) should expose a `checks..*` attribute that `nix flake check` - exercises. -- Do not merge changes that regress `make check`. +Add tests for every non-trivial behavior. + +Recommended test groups: + +- parser acceptance and rejection +- rule catalog construction +- dependency graph and strata +- join graph construction +- structural planning +- antijoin scheduling +- SIP-style filtering +- snapshot evaluation +- maintained-output equivalence +- CRDT fixtures +- Geomerge-style violation fixtures + +Tests should prefer small facts with readable expected outputs. Avoid large benchmark fixtures unless the test is explicitly performance-oriented. + +## Required Validation + +Use the repository's actual tooling. At the time this file was written, the copied `Makefile` is still Nix-playground-oriented and may not match this +project. Do not assume `make check` is meaningful until the Makefile is updated for this repository. + +For Rust changes, prefer: + +1. `cargo fmt` +2. `cargo clippy --all-targets --all-features` +3. `cargo test --all-targets --all-features` + +If the project does not yet have a `Cargo.toml`, record that validation was not available. + +For Markdown-only changes, run a manual read-through and check that headings follow the writing style. ## Change Design Checklist Before coding: -1. Identify which existing example or notes file the change belongs to, or whether it needs a new `NN-/`. -2. Confirm the change teaches one concept, not several. -3. Confirm `nixpkgs` input choice is consistent with surrounding examples. +1. Problem statement and target question +2. Existing module or new module decision +3. Snapshot oracle or expected output +4. Supported and unsupported feature boundary +5. Small fixture or example shape Before submitting: -1. Verify `make fmt-check`, `make lint`, and `make check` pass. -2. Verify every modified flake's `flake.lock` is committed. -3. Verify `notes/` accurately reflects what the examples now demonstrate. +1. Formatting status +2. Test status +3. Unsupported cases documented +4. No durable references to ignored local paths +5. Notes or examples updated when behavior changes -## Review Guidelines (P0/P1 Focus) +## Review Guidelines -Review output should be concise and only include critical issues. +Review output should prioritize correctness and experiment quality. -- `P0`: must-fix defects (a flake fails to evaluate, an example documents the wrong mechanism, notes contradict the code). -- `P1`: high-priority defects (eval warnings, missing `flake.lock`, unpinned or inconsistent inputs, misleading comment). - -Do not include: - -- style-only nitpicks, -- praise or summary of what is already good, -- exhaustive restatement of the patch. +- `P0`: must-fix defects, such as incorrect query results, invalid rollback behavior, unsupported syntax accepted silently, or tests that cannot run. +- `P1`: high-priority defects, such as nondeterministic plans, unclear unsupported-feature errors, missing snapshot oracle for a planner change, or + misleading notes. +- `P2`: useful follow-up, such as additional fixtures, clearer diagnostics, or broader benchmark coverage. Use this review format: -1. `Severity` (`P0`/`P1`) +1. `Severity` (`P0`/`P1`/`P2`) 2. `File:line` 3. `Issue` 4. `Why it matters` @@ -156,28 +277,29 @@ Use this review format: ## Practical Notes for Agents -- Prefer targeted edits over broad mechanical rewrites across examples. -- If two examples disagree on a convention, prefer the newer one and update the older example in a dedicated commit. -- When uncertain whether a concept deserves its own example, start by expanding the notes; promote to an example once the idea stabilizes. -- Keep presentational prose in `notes/`. Keep runnable material in numbered directories. Do not cross the streams. -- Prefer layout guidance based on naming patterns and discovery, not hard-coded counts of examples or notes. If you need the current tree, inspect it. -- Keep user-facing naming consistent with the repository name: `nix-playground`. The directory spelling `nix-playgraound` is intentional and should - not be "fixed". +- Read the relevant durable project notes before changing architecture. +- Treat copied papers, cloned repositories, and generated files in ignored local paths as reference material only. +- Prefer a planning-only prototype before backend integration. +- Prefer textual plan explanations in early tests. They make the planner easier to debug. +- Keep backend comparison fair: same rule, same input facts, same expected output. +- Keep transaction and rollback behavior explicit for validation experiments. +- When the Makefile becomes project-specific, update this file's validation section. ## Commit and PR Hygiene -- Keep commits scoped to one logical change: one example, one notes update, one convention shift. -- Commit `flake.lock` in the same commit that introduces or updates the `flake.nix` it belongs to. +- Keep commits scoped to one logical change: parser, catalog, planner, evaluator, fixture, note, or tooling. +- Do not mix broad formatting churn with semantic changes. - PR descriptions should include: - 1. what concept the change teaches or clarifies, - 2. which example directories or notes files are affected, - 3. any new `inputs` added and why, - 4. output of `make check` (pass/fail). + 1. the experiment or feature being tested, + 2. the source rules or fixtures affected, + 3. the expected behavior, + 4. validation commands and results, + 5. known unsupported cases. Suggested PR checklist: -- [ ] `make fmt-check` passes -- [ ] `make lint` passes -- [ ] `make check` passes -- [ ] `flake.lock` committed for every new or updated `flake.nix` -- [ ] Notes updated where the change introduces or changes a concept +- [ ] `cargo fmt` passes, if applicable +- [ ] `cargo clippy --all-targets --all-features` passes, if applicable +- [ ] `cargo test --all-targets --all-features` passes, if applicable +- [ ] Snapshot oracle or expected output included for planner behavior +- [ ] Unsupported cases documented diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md