habedi-work/storage-engine-playground
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the base setup and initial directory structure

Browse Source
This commit is contained in:
Hassan Abedi 2026-05-21 15:56:29 +02:00
parent 447f298ac5
commit 0e5572d1c8
2 changed files with 237 additions and 114 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

350
AGENTS.md
View File

@ -4,151 +4,272 @@ This file provides guidance to coding agents collaborating on this repository.
## Mission
`nix-playground` is a personal learning playground for Nix and flakes.
The goal is not production software but clear, runnable, progressively more advanced examples plus prose notes that explain them.
`storage-engine-playground` is an experimental Rust project for testing ideas from the FlowLog, DBSP, CRDT-as-query, and Geomerge notes.
The goal is not production software. The goal is a clear, runnable playground for small prototypes that help answer concrete architecture questions:
- how Datalog-like rules should be parsed, cataloged, planned, and optimized
- how FlowLog-style planning ideas transfer to a DBSP-oriented frontend
- how CRDT queries behave under naive plans versus planned relational execution
- how Geomerge-style laws can compile into maintained violation relations
- how backend behavior changes across snapshot, DBSP-like, and Differential Dataflow-like execution models
Priorities, in order:
1. Correctness: examples must actually evaluate and build.
2. Clarity: each example teaches one concept; names, comments, and directory structure should make that concept obvious.
3. Minimality: prefer the shortest flake or expression that demonstrates the idea.
4. Accuracy of notes: prose under `notes/` must not describe behavior the examples do not demonstrate.
5. Reproducibility: every flake commits its `flake.lock`; nothing depends on ambient state.
1. Correctness: prototypes must have clear expected outputs and tests.
2. Clarity: each module and test should answer one research or engineering question.
3. Small scope: prefer narrow experiments over broad engine rewrites.
4. Explainability: planners should emit inspectable plans, not only executable structures.
5. Reproducibility: examples should use committed fixtures, deterministic tests, and documented commands.
## Core Rules
- Use English for code, comments, and prose.
- Keep each numbered example self-contained: its own `flake.nix`, own `flake.lock`, no cross-example imports.
- Prefer small, focused changes over broad rewrites across examples.
- Add comments only when they clarify non-obvious Nix behavior (laziness, `rec`, string vs. path, `with` scoping, etc.).
- Do not describe Nix features in notes or comments as if they were implemented in an example unless the example actually uses them.
- When an example grows beyond one concept, split it into a new numbered directory rather than expanding the existing one.
Quick examples:
- Good: add `03-multi-system/` that demonstrates `forAllSystems` in isolation.
- Good: add a `checks` output to an existing flake with a one-line comment explaining what `nix flake check` will do with it.
- Bad: combine overlays, NixOS modules, and home-manager into one "comprehensive" example.
- Bad: edit `notes/` to describe an approach no example in the repo uses.
- Use English for code, comments, tests, and prose.
- Treat ignored local reference material as source material only. Do not import copied code into durable modules without an explicit decision.
- Prefer implementing small vertical slices: parse a subset, build a catalog, plan one rule shape, and test it.
- Do not build a full Datalog engine before the planning layer is useful and tested.
- Keep source language, relational planning, and backend execution separated.
- Prefer backend-neutral intermediate structures until a specific backend API requires specialization.
- Add comments only when they explain non-obvious planning, recursion, delta, or storage behavior.
- Treat tests and fixtures as part of the design, not as afterthoughts.
## Writing Style
- Use Oxford commas in inline lists: "a, b, and c" not "a, b, c".
- Do not use em dashes. Restructure the sentence, or use a colon or semicolon instead.
- Avoid colorful adjectives and adverbs. Write "dev shell" not "lightweight dev shell", "overlay" not "flexible overlay".
- Use noun phrases for checklist items, not imperative verbs. Write "input pinning" not "pin inputs".
- Headings in Markdown files must be in title case: "Build from Source" not "Build from source". Minor words (a, an, the, and, but, or, for, in, on,
at, to, by, of) stay lowercase unless they are the first word.
- Avoid colorful adjectives and adverbs. Write "storage engine" not "lightweight storage engine", "planner" not "clever planner".
- Use noun phrases for checklist items, not imperative verbs. Write "rule catalog construction" not "construct rule catalogs".
- Headings in Markdown files must be in title case: "Query Planning" not "Query planning". Minor words (a, an, the, and, but, or, for, in, on, at, to,
by, of) stay lowercase unless they are the first word.
## Repository Layout
- `NN-<topic>/`: self-contained numbered examples. Each matching top-level directory is a flake root.
- `notes/NNN-*.md`: prose companions numbered to match reading order.
- Lower note numbers cover shared foundations such as the glossary, the Nix language, and flakes.
- Later note numbers may cover specific example tracks or cross-example guides.
- `Makefile`: discovery-based helpers that run formatting, linting, and `nix flake check` across all examples.
- `AGENTS.md`: this file.
- `.pre-commit-config.yaml`, `.editorconfig`, `.gitattributes`, `.gitignore`: repository hygiene.
- `pyproject.toml`: Python environment metadata used only to install `pre-commit`.
The repository is new and may change. Discover the current layout from the filesystem before editing.
New examples follow `NN-<short-topic>/` where `NN` is a two-digit ordinal.
Expected durable areas may include:
Do not assume the directory list or note list in this file is exhaustive. The repository is expected to grow over time, and agents should discover the
current layout from the filesystem when needed.
- `src/`: Rust source for parser, catalog, planner, execution experiments, and storage prototypes.
- `tests/`: integration tests for rule planning, evaluation, and storage behavior.
- `examples/`: small runnable Datalog-like programs or storage scenarios.
- `fixtures/`: committed input facts and expected outputs.
- `notes/`: local design notes that belong to this project.
- `flowlog/`: project-local notes or sketches derived from the FlowLog line of work.
Example ordering should still feel like a readable path from simpler to more involved, but the repository may branch into themed subtracks, for example
Nix and flake outputs, Haskell with Nix, or future ecosystem-specific tracks. Keep each example focused on one concept even when the broader sequence
branches.
Do not assume this list is exhaustive. If the project grows a different structure, follow the actual codebase and update this file when conventions
stabilize.
## Example Layout Constraints
## Technical Direction
- Each example owns exactly one `flake.nix` at its root and commits its `flake.lock`.
- Examples do not import each other. Copy and adapt if a pattern needs to be shown twice.
- An example may depend only on flakes it declares in its own `inputs`.
- Prefer `nixpkgs` pinned to `nixos-unstable` for consistency across examples unless the example's point is pinning strategy.
- Keep the `outputs` attrset flat enough that `nix flake show` reads as a single screen.
- If an example exposes `checks.<system>.*`, those checks must pass under `nix flake check`.
The main experimental architecture is:
## Nix and Flake Conventions
```text
Datalog-like rules or Geolog-shaped laws
-> dependency analysis and strata
-> rule catalog
-> join graph
-> relational plan
-> FlowLog-style optimization
-> backend lowering
-> maintained or snapshot outputs
```
- Target Nix with `experimental-features = nix-command flakes` enabled (already the case on this machine).
- Prefer `pkgs.mkShell` for dev shells; reach for `mkShellNoCC` only when explaining the distinction.
- Use `nixpkgs.lib.genAttrs` or `flake-utils.lib.eachDefaultSystem` for multi-system outputs; pick one per example and say which in a comment.
- Use `follows` to unify transitive `nixpkgs` inputs when pulling in ecosystem flakes.
- Prefer `inherit` over repetition in attrsets.
- Avoid top-level `with` statements; keep `with` narrowly scoped to package lists.
- Format every `.nix` file with `nixfmt` (RFC 166 style) before committing.
Keep these layers explicit:
## Required Validation
- **Source Layer**: Datalog-like test programs, CRDT query definitions, and Geomerge-style laws.
- **Catalog Layer**: rule heads, body atoms, variables, constants, comparisons, negation, and projections.
- **Planning Layer**: join graphs, join order, antijoin placement, SIP-style filtering, subplan sharing, and physical key choice.
- **Execution Layer**: snapshot evaluator first, then DBSP-like or Differential Dataflow-like experiments.
- **Storage Layer**: facts, transactions, rollback, preview state, and violation output integration.
Run these checks for any non-trivial change:
## FlowLog-Inspired Planning
1. `make fmt-check`
2. `make lint`
3. `make check`
FlowLog should be treated as a planning reference, not as an automatic dependency.
These map to `nixfmt --check`, `statix check` plus `deadnix`, and `nix flake check` across every numbered example.
Reusable ideas:
For notes-only changes, `make fmt-check` and a manual read-through suffice.
- rule catalog construction
- dependency graph and stratification
- per-rule join graph extraction
- width-oriented structural planning
- sideways information passing
- antijoin scheduling
- physical key and payload selection
- shared subplan detection
## First Contribution Flow
When adapting an idea, write the smallest test that demonstrates the behavior. For example:
Use this sequence for your first change:
```text
rule with three positive atoms
-> catalog variables
-> join graph
-> planned join tree
-> expected textual plan
```
1. Read the relevant `notes/` file and the nearest existing example.
2. Add the smallest possible flake or expression demonstrating the new concept.
3. Add a short header comment in the new `flake.nix` stating what the example teaches.
4. Run `nix flake check` inside the new example directory.
5. Run `make fmt-check` and `make lint` from the repository root.
6. Add or update the matching entry in `notes/` if the concept is not yet covered there.
## DBSP and Incremental Execution
Example scopes that are good first tasks:
DBSP-related work should preserve a clean boundary:
- Add `02-package/` with a trivial `stdenv.mkDerivation` and one-line install phase.
- Add a `checks` output to `01-devshell/` that asserts a tool is on `$PATH`.
- Add a short section to `notes/003-flakes.md` referencing a newly added example.
- Convert an existing example from a hand-rolled `forAllSystems` to `flake-utils`, or vice versa, with a comment explaining the tradeoff.
```text
planned relational IR
-> DBSP lowering
-> maintained output deltas
```
When the repository contains multiple themed tracks, "the nearest existing example" means the nearest example in the relevant track, not necessarily the
numerically closest directory overall.
Do not make DBSP responsible for source-language semantics. The frontend should check supported syntax, stratification, and rule shape before backend
lowering.
For each DBSP-like experiment, also provide a snapshot oracle when feasible:
```text
snapshot result == maintained result after each update
```
Track these measurements when relevant:
- hydration time
- warm-update time
- output delta size
- maintained state size if available
- sensitivity to join order
- sensitivity to causal-history depth
## CRDT Query Experiments
Initial CRDT workloads should stay small and explicit:
- multi-value register
- causal readiness over `pred`
- list next-element traversal
- tombstone skipping
Use operation facts shaped like:
```text
set(replica_id, counter, key, value)
pred(from_replica_id, from_counter, to_replica_id, to_counter)
insert(replica_id, counter, parent_replica_id, parent_counter, value)
remove(replica_id, counter)
```
Important questions:
- Does the query require recursion, negation, or both?
- Can antijoins run earlier?
- Can causal readiness be maintained from a frontier?
- Does warm-update cost depend on history depth?
- Does the output need integration into a current view?
## Geomerge-Style Validation Experiments
The first Geomerge-style target is maintained violation detection for supported relational laws.
A useful lowering shape is:
```text
required_consequent(x) :- antecedent(x).
violation(x) :- required_consequent(x), not consequent(x).
```
Start with:
- foreign-key-style laws
- totality-as-validation laws
- equality-as-violation laws
- multi-atom antecedents without existential witnesses
Exclude at first:
- existential witness generation
- disjunctive consequents
- equality saturation
- model branching
- full chase behavior
Violation rows should carry enough context for diagnostics:
```text
law_id
violation_kind
relation_or_consequent
bound_variable_values
```
## Rust Conventions
- Prefer small modules with explicit data structures over large generic abstractions.
- Use enums and structs to model rule syntax, catalog entries, plan nodes, and execution results.
- Prefer typed identifiers for relation names, variable names, rule ids, and field positions when it improves clarity.
- Keep parser errors and unsupported-feature errors explicit.
- Avoid panics in library code except for internal invariants that tests already cover.
- Use deterministic ordering for plans and diagnostics so tests are stable.
- Prefer simple snapshot evaluators as correctness oracles before optimizing.
## Testing Expectations
- This repository has no runtime test suite; "tests" are `nix flake check` outcomes and successful builds of each example's default output.
- Any example that exposes non-trivial behavior (a derivation, a module) should expose a `checks.<system>.*` attribute that `nix flake check`
exercises.
- Do not merge changes that regress `make check`.
Add tests for every non-trivial behavior.
Recommended test groups:
- parser acceptance and rejection
- rule catalog construction
- dependency graph and strata
- join graph construction
- structural planning
- antijoin scheduling
- SIP-style filtering
- snapshot evaluation
- maintained-output equivalence
- CRDT fixtures
- Geomerge-style violation fixtures
Tests should prefer small facts with readable expected outputs. Avoid large benchmark fixtures unless the test is explicitly performance-oriented.
## Required Validation
Use the repository's actual tooling. At the time this file was written, the copied `Makefile` is still Nix-playground-oriented and may not match this
project. Do not assume `make check` is meaningful until the Makefile is updated for this repository.
For Rust changes, prefer:
1. `cargo fmt`
2. `cargo clippy --all-targets --all-features`
3. `cargo test --all-targets --all-features`
If the project does not yet have a `Cargo.toml`, record that validation was not available.
For Markdown-only changes, run a manual read-through and check that headings follow the writing style.
## Change Design Checklist
Before coding:
1. Identify which existing example or notes file the change belongs to, or whether it needs a new `NN-<topic>/`.
2. Confirm the change teaches one concept, not several.
3. Confirm `nixpkgs` input choice is consistent with surrounding examples.
1. Problem statement and target question
2. Existing module or new module decision
3. Snapshot oracle or expected output
4. Supported and unsupported feature boundary
5. Small fixture or example shape
Before submitting:
1. Verify `make fmt-check`, `make lint`, and `make check` pass.
2. Verify every modified flake's `flake.lock` is committed.
3. Verify `notes/` accurately reflects what the examples now demonstrate.
1. Formatting status
2. Test status
3. Unsupported cases documented
4. No durable references to ignored local paths
5. Notes or examples updated when behavior changes
## Review Guidelines (P0/P1 Focus)
## Review Guidelines
Review output should be concise and only include critical issues.
Review output should prioritize correctness and experiment quality.
- `P0`: must-fix defects (a flake fails to evaluate, an example documents the wrong mechanism, notes contradict the code).
- `P1`: high-priority defects (eval warnings, missing `flake.lock`, unpinned or inconsistent inputs, misleading comment).
Do not include:
- style-only nitpicks,
- praise or summary of what is already good,
- exhaustive restatement of the patch.
- `P0`: must-fix defects, such as incorrect query results, invalid rollback behavior, unsupported syntax accepted silently, or tests that cannot run.
- `P1`: high-priority defects, such as nondeterministic plans, unclear unsupported-feature errors, missing snapshot oracle for a planner change, or
misleading notes.
- `P2`: useful follow-up, such as additional fixtures, clearer diagnostics, or broader benchmark coverage.
Use this review format:
1. `Severity` (`P0`/`P1`)
1. `Severity` (`P0`/`P1`/`P2`)
2. `File:line`
3. `Issue`
4. `Why it matters`
@ -156,28 +277,29 @@ Use this review format:
## Practical Notes for Agents
- Prefer targeted edits over broad mechanical rewrites across examples.
- If two examples disagree on a convention, prefer the newer one and update the older example in a dedicated commit.
- When uncertain whether a concept deserves its own example, start by expanding the notes; promote to an example once the idea stabilizes.
- Keep presentational prose in `notes/`. Keep runnable material in numbered directories. Do not cross the streams.
- Prefer layout guidance based on naming patterns and discovery, not hard-coded counts of examples or notes. If you need the current tree, inspect it.
- Keep user-facing naming consistent with the repository name: `nix-playground`. The directory spelling `nix-playgraound` is intentional and should
not be "fixed".
- Read the relevant durable project notes before changing architecture.
- Treat copied papers, cloned repositories, and generated files in ignored local paths as reference material only.
- Prefer a planning-only prototype before backend integration.
- Prefer textual plan explanations in early tests. They make the planner easier to debug.
- Keep backend comparison fair: same rule, same input facts, same expected output.
- Keep transaction and rollback behavior explicit for validation experiments.
- When the Makefile becomes project-specific, update this file's validation section.
## Commit and PR Hygiene
- Keep commits scoped to one logical change: one example, one notes update, one convention shift.
- Commit `flake.lock` in the same commit that introduces or updates the `flake.nix` it belongs to.
- Keep commits scoped to one logical change: parser, catalog, planner, evaluator, fixture, note, or tooling.
- Do not mix broad formatting churn with semantic changes.
- PR descriptions should include:
1. what concept the change teaches or clarifies,
2. which example directories or notes files are affected,
3. any new `inputs` added and why,
4. output of `make check` (pass/fail).
1. the experiment or feature being tested,
2. the source rules or fixtures affected,
3. the expected behavior,
4. validation commands and results,
5. known unsupported cases.
Suggested PR checklist:
- [ ] `make fmt-check` passes
- [ ] `make lint` passes
- [ ] `make check` passes
- [ ] `flake.lock` committed for every new or updated `flake.nix`
- [ ] Notes updated where the change introduces or changes a concept
- [ ] `cargo fmt` passes, if applicable
- [ ] `cargo clippy --all-targets --all-features` passes, if applicable
- [ ] `cargo test --all-targets --all-features` passes, if applicable
- [ ] Snapshot oracle or expected output included for planner behavior
- [ ] Unsupported cases documented

1
CLAUDE.md Normal file
View File

@ -0,0 +1 @@
@AGENTS.md