Add the base setup and initial directory structure
This commit is contained in:
Hassan Abedi
parent
447f298ac5
commit
0e5572d1c8
350
AGENTS.md
350
AGENTS.md
@ -4,151 +4,272 @@ This file provides guidance to coding agents collaborating on this repository.
|
||||
|
||||
## Mission
|
||||
|
||||
`nix-playground` is a personal learning playground for Nix and flakes.
|
||||
The goal is not production software but clear, runnable, progressively more advanced examples plus prose notes that explain them.
|
||||
`storage-engine-playground` is an experimental Rust project for testing ideas from the FlowLog, DBSP, CRDT-as-query, and Geomerge notes.
|
||||
|
||||
The goal is not production software. The goal is a clear, runnable playground for small prototypes that help answer concrete architecture questions:
|
||||
|
||||
- how Datalog-like rules should be parsed, cataloged, planned, and optimized
|
||||
- how FlowLog-style planning ideas transfer to a DBSP-oriented frontend
|
||||
- how CRDT queries behave under naive plans versus planned relational execution
|
||||
- how Geomerge-style laws can compile into maintained violation relations
|
||||
- how backend behavior changes across snapshot, DBSP-like, and Differential Dataflow-like execution models
|
||||
|
||||
Priorities, in order:
|
||||
|
||||
1. Correctness: examples must actually evaluate and build.
|
||||
2. Clarity: each example teaches one concept; names, comments, and directory structure should make that concept obvious.
|
||||
3. Minimality: prefer the shortest flake or expression that demonstrates the idea.
|
||||
4. Accuracy of notes: prose under `notes/` must not describe behavior the examples do not demonstrate.
|
||||
5. Reproducibility: every flake commits its `flake.lock`; nothing depends on ambient state.
|
||||
1. Correctness: prototypes must have clear expected outputs and tests.
|
||||
2. Clarity: each module and test should answer one research or engineering question.
|
||||
3. Small scope: prefer narrow experiments over broad engine rewrites.
|
||||
4. Explainability: planners should emit inspectable plans, not only executable structures.
|
||||
5. Reproducibility: examples should use committed fixtures, deterministic tests, and documented commands.
|
||||
|
||||
## Core Rules
|
||||
|
||||
- Use English for code, comments, and prose.
|
||||
- Keep each numbered example self-contained: its own `flake.nix`, own `flake.lock`, no cross-example imports.
|
||||
- Prefer small, focused changes over broad rewrites across examples.
|
||||
- Add comments only when they clarify non-obvious Nix behavior (laziness, `rec`, string vs. path, `with` scoping, etc.).
|
||||
- Do not describe Nix features in notes or comments as if they were implemented in an example unless the example actually uses them.
|
||||
- When an example grows beyond one concept, split it into a new numbered directory rather than expanding the existing one.
|
||||
|
||||
Quick examples:
|
||||
|
||||
- Good: add `03-multi-system/` that demonstrates `forAllSystems` in isolation.
|
||||
- Good: add a `checks` output to an existing flake with a one-line comment explaining what `nix flake check` will do with it.
|
||||
- Bad: combine overlays, NixOS modules, and home-manager into one "comprehensive" example.
|
||||
- Bad: edit `notes/` to describe an approach no example in the repo uses.
|
||||
- Use English for code, comments, tests, and prose.
|
||||
- Treat ignored local reference material as source material only. Do not import copied code into durable modules without an explicit decision.
|
||||
- Prefer implementing small vertical slices: parse a subset, build a catalog, plan one rule shape, and test it.
|
||||
- Do not build a full Datalog engine before the planning layer is useful and tested.
|
||||
- Keep source language, relational planning, and backend execution separated.
|
||||
- Prefer backend-neutral intermediate structures until a specific backend API requires specialization.
|
||||
- Add comments only when they explain non-obvious planning, recursion, delta, or storage behavior.
|
||||
- Treat tests and fixtures as part of the design, not as afterthoughts.
|
||||
|
||||
## Writing Style
|
||||
|
||||
- Use Oxford commas in inline lists: "a, b, and c" not "a, b, c".
|
||||
- Do not use em dashes. Restructure the sentence, or use a colon or semicolon instead.
|
||||
- Avoid colorful adjectives and adverbs. Write "dev shell" not "lightweight dev shell", "overlay" not "flexible overlay".
|
||||
- Use noun phrases for checklist items, not imperative verbs. Write "input pinning" not "pin inputs".
|
||||
- Headings in Markdown files must be in title case: "Build from Source" not "Build from source". Minor words (a, an, the, and, but, or, for, in, on,
|
||||
at, to, by, of) stay lowercase unless they are the first word.
|
||||
- Avoid colorful adjectives and adverbs. Write "storage engine" not "lightweight storage engine", "planner" not "clever planner".
|
||||
- Use noun phrases for checklist items, not imperative verbs. Write "rule catalog construction" not "construct rule catalogs".
|
||||
- Headings in Markdown files must be in title case: "Query Planning" not "Query planning". Minor words (a, an, the, and, but, or, for, in, on, at, to,
|
||||
by, of) stay lowercase unless they are the first word.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `NN-<topic>/`: self-contained numbered examples. Each matching top-level directory is a flake root.
|
||||
- `notes/NNN-*.md`: prose companions numbered to match reading order.
|
||||
- Lower note numbers cover shared foundations such as the glossary, the Nix language, and flakes.
|
||||
- Later note numbers may cover specific example tracks or cross-example guides.
|
||||
- `Makefile`: discovery-based helpers that run formatting, linting, and `nix flake check` across all examples.
|
||||
- `AGENTS.md`: this file.
|
||||
- `.pre-commit-config.yaml`, `.editorconfig`, `.gitattributes`, `.gitignore`: repository hygiene.
|
||||
- `pyproject.toml`: Python environment metadata used only to install `pre-commit`.
|
||||
The repository is new and may change. Discover the current layout from the filesystem before editing.
|
||||
|
||||
New examples follow `NN-<short-topic>/` where `NN` is a two-digit ordinal.
|
||||
Expected durable areas may include:
|
||||
|
||||
Do not assume the directory list or note list in this file is exhaustive. The repository is expected to grow over time, and agents should discover the
|
||||
current layout from the filesystem when needed.
|
||||
- `src/`: Rust source for parser, catalog, planner, execution experiments, and storage prototypes.
|
||||
- `tests/`: integration tests for rule planning, evaluation, and storage behavior.
|
||||
- `examples/`: small runnable Datalog-like programs or storage scenarios.
|
||||
- `fixtures/`: committed input facts and expected outputs.
|
||||
- `notes/`: local design notes that belong to this project.
|
||||
- `flowlog/`: project-local notes or sketches derived from the FlowLog line of work.
|
||||
|
||||
Example ordering should still feel like a readable path from simpler to more involved, but the repository may branch into themed subtracks, for example
|
||||
Nix and flake outputs, Haskell with Nix, or future ecosystem-specific tracks. Keep each example focused on one concept even when the broader sequence
|
||||
branches.
|
||||
Do not assume this list is exhaustive. If the project grows a different structure, follow the actual codebase and update this file when conventions
|
||||
stabilize.
|
||||
|
||||
## Example Layout Constraints
|
||||
## Technical Direction
|
||||
|
||||
- Each example owns exactly one `flake.nix` at its root and commits its `flake.lock`.
|
||||
- Examples do not import each other. Copy and adapt if a pattern needs to be shown twice.
|
||||
- An example may depend only on flakes it declares in its own `inputs`.
|
||||
- Prefer `nixpkgs` pinned to `nixos-unstable` for consistency across examples unless the example's point is pinning strategy.
|
||||
- Keep the `outputs` attrset flat enough that `nix flake show` reads as a single screen.
|
||||
- If an example exposes `checks.<system>.*`, those checks must pass under `nix flake check`.
|
||||
The main experimental architecture is:
|
||||
|
||||
## Nix and Flake Conventions
|
||||
```text
|
||||
Datalog-like rules or Geolog-shaped laws
|
||||
-> dependency analysis and strata
|
||||
-> rule catalog
|
||||
-> join graph
|
||||
-> relational plan
|
||||
-> FlowLog-style optimization
|
||||
-> backend lowering
|
||||
-> maintained or snapshot outputs
|
||||
```
|
||||
|
||||
- Target Nix with `experimental-features = nix-command flakes` enabled (already the case on this machine).
|
||||
- Prefer `pkgs.mkShell` for dev shells; reach for `mkShellNoCC` only when explaining the distinction.
|
||||
- Use `nixpkgs.lib.genAttrs` or `flake-utils.lib.eachDefaultSystem` for multi-system outputs; pick one per example and say which in a comment.
|
||||
- Use `follows` to unify transitive `nixpkgs` inputs when pulling in ecosystem flakes.
|
||||
- Prefer `inherit` over repetition in attrsets.
|
||||
- Avoid top-level `with` statements; keep `with` narrowly scoped to package lists.
|
||||
- Format every `.nix` file with `nixfmt` (RFC 166 style) before committing.
|
||||
Keep these layers explicit:
|
||||
|
||||
## Required Validation
|
||||
- **Source Layer**: Datalog-like test programs, CRDT query definitions, and Geomerge-style laws.
|
||||
- **Catalog Layer**: rule heads, body atoms, variables, constants, comparisons, negation, and projections.
|
||||
- **Planning Layer**: join graphs, join order, antijoin placement, SIP-style filtering, subplan sharing, and physical key choice.
|
||||
- **Execution Layer**: snapshot evaluator first, then DBSP-like or Differential Dataflow-like experiments.
|
||||
- **Storage Layer**: facts, transactions, rollback, preview state, and violation output integration.
|
||||
|
||||
Run these checks for any non-trivial change:
|
||||
## FlowLog-Inspired Planning
|
||||
|
||||
1. `make fmt-check`
|
||||
2. `make lint`
|
||||
3. `make check`
|
||||
FlowLog should be treated as a planning reference, not as an automatic dependency.
|
||||
|
||||
These map to `nixfmt --check`, `statix check` plus `deadnix`, and `nix flake check` across every numbered example.
|
||||
Reusable ideas:
|
||||
|
||||
For notes-only changes, `make fmt-check` and a manual read-through suffice.
|
||||
- rule catalog construction
|
||||
- dependency graph and stratification
|
||||
- per-rule join graph extraction
|
||||
- width-oriented structural planning
|
||||
- sideways information passing
|
||||
- antijoin scheduling
|
||||
- physical key and payload selection
|
||||
- shared subplan detection
|
||||
|
||||
## First Contribution Flow
|
||||
When adapting an idea, write the smallest test that demonstrates the behavior. For example:
|
||||
|
||||
Use this sequence for your first change:
|
||||
```text
|
||||
rule with three positive atoms
|
||||
-> catalog variables
|
||||
-> join graph
|
||||
-> planned join tree
|
||||
-> expected textual plan
|
||||
```
|
||||
|
||||
1. Read the relevant `notes/` file and the nearest existing example.
|
||||
2. Add the smallest possible flake or expression demonstrating the new concept.
|
||||
3. Add a short header comment in the new `flake.nix` stating what the example teaches.
|
||||
4. Run `nix flake check` inside the new example directory.
|
||||
5. Run `make fmt-check` and `make lint` from the repository root.
|
||||
6. Add or update the matching entry in `notes/` if the concept is not yet covered there.
|
||||
## DBSP and Incremental Execution
|
||||
|
||||
Example scopes that are good first tasks:
|
||||
DBSP-related work should preserve a clean boundary:
|
||||
|
||||
- Add `02-package/` with a trivial `stdenv.mkDerivation` and one-line install phase.
|
||||
- Add a `checks` output to `01-devshell/` that asserts a tool is on `$PATH`.
|
||||
- Add a short section to `notes/003-flakes.md` referencing a newly added example.
|
||||
- Convert an existing example from a hand-rolled `forAllSystems` to `flake-utils`, or vice versa, with a comment explaining the tradeoff.
|
||||
```text
|
||||
planned relational IR
|
||||
-> DBSP lowering
|
||||
-> maintained output deltas
|
||||
```
|
||||
|
||||
When the repository contains multiple themed tracks, "the nearest existing example" means the nearest example in the relevant track, not necessarily the
|
||||
numerically closest directory overall.
|
||||
Do not make DBSP responsible for source-language semantics. The frontend should check supported syntax, stratification, and rule shape before backend
|
||||
lowering.
|
||||
|
||||
For each DBSP-like experiment, also provide a snapshot oracle when feasible:
|
||||
|
||||
```text
|
||||
snapshot result == maintained result after each update
|
||||
```
|
||||
|
||||
Track these measurements when relevant:
|
||||
|
||||
- hydration time
|
||||
- warm-update time
|
||||
- output delta size
|
||||
- maintained state size if available
|
||||
- sensitivity to join order
|
||||
- sensitivity to causal-history depth
|
||||
|
||||
## CRDT Query Experiments
|
||||
|
||||
Initial CRDT workloads should stay small and explicit:
|
||||
|
||||
- multi-value register
|
||||
- causal readiness over `pred`
|
||||
- list next-element traversal
|
||||
- tombstone skipping
|
||||
|
||||
Use operation facts shaped like:
|
||||
|
||||
```text
|
||||
set(replica_id, counter, key, value)
|
||||
pred(from_replica_id, from_counter, to_replica_id, to_counter)
|
||||
insert(replica_id, counter, parent_replica_id, parent_counter, value)
|
||||
remove(replica_id, counter)
|
||||
```
|
||||
|
||||
Important questions:
|
||||
|
||||
- Does the query require recursion, negation, or both?
|
||||
- Can antijoins run earlier?
|
||||
- Can causal readiness be maintained from a frontier?
|
||||
- Does warm-update cost depend on history depth?
|
||||
- Does the output need integration into a current view?
|
||||
|
||||
## Geomerge-Style Validation Experiments
|
||||
|
||||
The first Geomerge-style target is maintained violation detection for supported relational laws.
|
||||
|
||||
A useful lowering shape is:
|
||||
|
||||
```text
|
||||
required_consequent(x) :- antecedent(x).
|
||||
violation(x) :- required_consequent(x), not consequent(x).
|
||||
```
|
||||
|
||||
Start with:
|
||||
|
||||
- foreign-key-style laws
|
||||
- totality-as-validation laws
|
||||
- equality-as-violation laws
|
||||
- multi-atom antecedents without existential witnesses
|
||||
|
||||
Exclude at first:
|
||||
|
||||
- existential witness generation
|
||||
- disjunctive consequents
|
||||
- equality saturation
|
||||
- model branching
|
||||
- full chase behavior
|
||||
|
||||
Violation rows should carry enough context for diagnostics:
|
||||
|
||||
```text
|
||||
law_id
|
||||
violation_kind
|
||||
relation_or_consequent
|
||||
bound_variable_values
|
||||
```
|
||||
|
||||
## Rust Conventions
|
||||
|
||||
- Prefer small modules with explicit data structures over large generic abstractions.
|
||||
- Use enums and structs to model rule syntax, catalog entries, plan nodes, and execution results.
|
||||
- Prefer typed identifiers for relation names, variable names, rule ids, and field positions when it improves clarity.
|
||||
- Keep parser errors and unsupported-feature errors explicit.
|
||||
- Avoid panics in library code except for internal invariants that tests already cover.
|
||||
- Use deterministic ordering for plans and diagnostics so tests are stable.
|
||||
- Prefer simple snapshot evaluators as correctness oracles before optimizing.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
- This repository has no runtime test suite; "tests" are `nix flake check` outcomes and successful builds of each example's default output.
|
||||
- Any example that exposes non-trivial behavior (a derivation, a module) should expose a `checks.<system>.*` attribute that `nix flake check`
|
||||
exercises.
|
||||
- Do not merge changes that regress `make check`.
|
||||
Add tests for every non-trivial behavior.
|
||||
|
||||
Recommended test groups:
|
||||
|
||||
- parser acceptance and rejection
|
||||
- rule catalog construction
|
||||
- dependency graph and strata
|
||||
- join graph construction
|
||||
- structural planning
|
||||
- antijoin scheduling
|
||||
- SIP-style filtering
|
||||
- snapshot evaluation
|
||||
- maintained-output equivalence
|
||||
- CRDT fixtures
|
||||
- Geomerge-style violation fixtures
|
||||
|
||||
Tests should prefer small facts with readable expected outputs. Avoid large benchmark fixtures unless the test is explicitly performance-oriented.
|
||||
|
||||
## Required Validation
|
||||
|
||||
Use the repository's actual tooling. At the time this file was written, the copied `Makefile` is still Nix-playground-oriented and may not match this
|
||||
project. Do not assume `make check` is meaningful until the Makefile is updated for this repository.
|
||||
|
||||
For Rust changes, prefer:
|
||||
|
||||
1. `cargo fmt`
|
||||
2. `cargo clippy --all-targets --all-features`
|
||||
3. `cargo test --all-targets --all-features`
|
||||
|
||||
If the project does not yet have a `Cargo.toml`, record that validation was not available.
|
||||
|
||||
For Markdown-only changes, run a manual read-through and check that headings follow the writing style.
|
||||
|
||||
## Change Design Checklist
|
||||
|
||||
Before coding:
|
||||
|
||||
1. Identify which existing example or notes file the change belongs to, or whether it needs a new `NN-<topic>/`.
|
||||
2. Confirm the change teaches one concept, not several.
|
||||
3. Confirm `nixpkgs` input choice is consistent with surrounding examples.
|
||||
1. Problem statement and target question
|
||||
2. Existing module or new module decision
|
||||
3. Snapshot oracle or expected output
|
||||
4. Supported and unsupported feature boundary
|
||||
5. Small fixture or example shape
|
||||
|
||||
Before submitting:
|
||||
|
||||
1. Verify `make fmt-check`, `make lint`, and `make check` pass.
|
||||
2. Verify every modified flake's `flake.lock` is committed.
|
||||
3. Verify `notes/` accurately reflects what the examples now demonstrate.
|
||||
1. Formatting status
|
||||
2. Test status
|
||||
3. Unsupported cases documented
|
||||
4. No durable references to ignored local paths
|
||||
5. Notes or examples updated when behavior changes
|
||||
|
||||
## Review Guidelines (P0/P1 Focus)
|
||||
## Review Guidelines
|
||||
|
||||
Review output should be concise and only include critical issues.
|
||||
Review output should prioritize correctness and experiment quality.
|
||||
|
||||
- `P0`: must-fix defects (a flake fails to evaluate, an example documents the wrong mechanism, notes contradict the code).
|
||||
- `P1`: high-priority defects (eval warnings, missing `flake.lock`, unpinned or inconsistent inputs, misleading comment).
|
||||
|
||||
Do not include:
|
||||
|
||||
- style-only nitpicks,
|
||||
- praise or summary of what is already good,
|
||||
- exhaustive restatement of the patch.
|
||||
- `P0`: must-fix defects, such as incorrect query results, invalid rollback behavior, unsupported syntax accepted silently, or tests that cannot run.
|
||||
- `P1`: high-priority defects, such as nondeterministic plans, unclear unsupported-feature errors, missing snapshot oracle for a planner change, or
|
||||
misleading notes.
|
||||
- `P2`: useful follow-up, such as additional fixtures, clearer diagnostics, or broader benchmark coverage.
|
||||
|
||||
Use this review format:
|
||||
|
||||
1. `Severity` (`P0`/`P1`)
|
||||
1. `Severity` (`P0`/`P1`/`P2`)
|
||||
2. `File:line`
|
||||
3. `Issue`
|
||||
4. `Why it matters`
|
||||
@ -156,28 +277,29 @@ Use this review format:
|
||||
|
||||
## Practical Notes for Agents
|
||||
|
||||
- Prefer targeted edits over broad mechanical rewrites across examples.
|
||||
- If two examples disagree on a convention, prefer the newer one and update the older example in a dedicated commit.
|
||||
- When uncertain whether a concept deserves its own example, start by expanding the notes; promote to an example once the idea stabilizes.
|
||||
- Keep presentational prose in `notes/`. Keep runnable material in numbered directories. Do not cross the streams.
|
||||
- Prefer layout guidance based on naming patterns and discovery, not hard-coded counts of examples or notes. If you need the current tree, inspect it.
|
||||
- Keep user-facing naming consistent with the repository name: `nix-playground`. The directory spelling `nix-playgraound` is intentional and should
|
||||
not be "fixed".
|
||||
- Read the relevant durable project notes before changing architecture.
|
||||
- Treat copied papers, cloned repositories, and generated files in ignored local paths as reference material only.
|
||||
- Prefer a planning-only prototype before backend integration.
|
||||
- Prefer textual plan explanations in early tests. They make the planner easier to debug.
|
||||
- Keep backend comparison fair: same rule, same input facts, same expected output.
|
||||
- Keep transaction and rollback behavior explicit for validation experiments.
|
||||
- When the Makefile becomes project-specific, update this file's validation section.
|
||||
|
||||
## Commit and PR Hygiene
|
||||
|
||||
- Keep commits scoped to one logical change: one example, one notes update, one convention shift.
|
||||
- Commit `flake.lock` in the same commit that introduces or updates the `flake.nix` it belongs to.
|
||||
- Keep commits scoped to one logical change: parser, catalog, planner, evaluator, fixture, note, or tooling.
|
||||
- Do not mix broad formatting churn with semantic changes.
|
||||
- PR descriptions should include:
|
||||
1. what concept the change teaches or clarifies,
|
||||
2. which example directories or notes files are affected,
|
||||
3. any new `inputs` added and why,
|
||||
4. output of `make check` (pass/fail).
|
||||
1. the experiment or feature being tested,
|
||||
2. the source rules or fixtures affected,
|
||||
3. the expected behavior,
|
||||
4. validation commands and results,
|
||||
5. known unsupported cases.
|
||||
|
||||
Suggested PR checklist:
|
||||
|
||||
- [ ] `make fmt-check` passes
|
||||
- [ ] `make lint` passes
|
||||
- [ ] `make check` passes
|
||||
- [ ] `flake.lock` committed for every new or updated `flake.nix`
|
||||
- [ ] Notes updated where the change introduces or changes a concept
|
||||
- [ ] `cargo fmt` passes, if applicable
|
||||
- [ ] `cargo clippy --all-targets --all-features` passes, if applicable
|
||||
- [ ] `cargo test --all-targets --all-features` passes, if applicable
|
||||
- [ ] Snapshot oracle or expected output included for planner behavior
|
||||
- [ ] Unsupported cases documented
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user