diff --git a/scratches/query-engine-big-picture.md b/scratches/query-engine-big-picture.md index 1970706..abbd7c1 100644 --- a/scratches/query-engine-big-picture.md +++ b/scratches/query-engine-big-picture.md @@ -35,9 +35,9 @@ So a useful architecture shape would be: 2. `Intermediate Representation` 3. `IR Execution / Query Engine` 4. backend implementations: - - in-memory execution - - Postgres execution - - Rust storage/query execution + - in-memory execution + - Postgres execution + - Rust storage/query execution ## Two ways to think about the query engine @@ -88,7 +88,9 @@ So the IR should be treated as the interface, and the query engine should be tre 4. Do we expect the first real executor to be in-memory, Postgres-backed, or Rust-native? 5. Is Postgres only a storage/query backend, or is it intended to become a serious execution target for law checking and chase steps too? 6. What parts of execution must be backend-independent: query planning, law translation, chase scheduling, provenance, branch management? -7. Do existential witness generation, disjunctive branching, and equality merging belong inside the query engine, or in a separate chase/runtime layer above it? -8. Should the query engine operate directly on relational tables, or should it also understand higher-level Geolog structure like paths, theories, and dependencies? +7. Do existential witness generation, disjunctive branching, and equality merging belong inside the query engine, or in a separate chase/runtime layer + above it? +8. Should the query engine operate directly on relational tables, or should it also understand higher-level Geolog structure like paths, theories, and + dependencies? 9. What is the expected contract between Haskell lowering and Rust execution: raw IR data only, or also precompiled queries/plans? 10. What is the earliest milestone for the engine: run queries, check laws, perform simple fixpoint evaluation, or support full chase behavior? diff --git a/scratches/query-engine-broad.svg b/scratches/query-engine-broad.svg new file mode 100644 index 0000000..67a2c41 --- /dev/null +++ b/scratches/query-engine-broad.svg @@ -0,0 +1,397 @@ + + + + + + + GeologQueryEngineBroad + + + cluster_frontend + + Geolog Front End + + + + cluster_language + + Language Pipeline + + + + cluster_tools + + Developer Tooling + + + + cluster_runtime + + Shared Runtime / Execution Layer + + + + cluster_engine + + IR Execution Engine + + + + cluster_adapters + + Backend Adapters + + + + cluster_backends + + Execution Targets + + + + + concrete + + + Concrete Syntax + + + + + parser + + + Parser + + + + + concrete->parser + + + + + + abstract + + + Abstract Syntax + + + + + parser->abstract + + + + + + lsp + + + LSP + + + (parsing + elaboration diagnostics) + + + + + parser->lsp + + + syntax info + + + + + elaborator + + + Elaborator + + + + + abstract->elaborator + + + + + + typed + + + Elaborated Syntax + + + (with types) + + + + + elaborator->typed + + + + + + elaborator->lsp + + + type info + + + + + lowering + + Lowering + + + + + typed->lowering + + + + + + ir + + Intermediate Representation + + (shared execution contract) + + + + + lowering->ir + + + + + + engine + + Shared Query Engine + + + • user queries + + + • law checking + + • + chase / fixpoint steps + + • + planning / rewriting + + + + + ir->engine + + + execute IR + + + + + runtime_ir + + Runtime State + + (branches, witnesses, + + + equality, provenance) + + + + + engine->runtime_ir + + + maintains + + + + + inmem_adapter + + In-Memory Adapter + + + + + engine->inmem_adapter + + + + + + pg_adapter + + Postgres Adapter + + + + + engine->pg_adapter + + + + + + ffi + + FFI Haskell to Rust + + (IR marshalling) + + + + + engine->ffi + + + + + + inmem_db + + In-Memory DB + + + + + inmem_adapter->inmem_db + + + + + + pg_db + + Postgres DB + + + + + pg_adapter->pg_db + + + + + + rust_adapter + + Rust Adapter + + + + + rust_store + + Rust Storage / Query Layer + + + + + rust_adapter->rust_store + + + + + + ffi->rust_adapter + + + + + diff --git a/scratches/query-engine-glossary.md b/scratches/query-engine-glossary.md new file mode 100644 index 0000000..01402b9 --- /dev/null +++ b/scratches/query-engine-glossary.md @@ -0,0 +1,36 @@ +# Query Engine Glossary + +This is a short glossary for the query-engine and IR discussion. + +## Terms + +- `IR`: Intermediate representation. A simpler form that sits between elaboration and execution. +- `lowering`: The step that converts elaborated Geolog code into the IR. +- `table`: A relational object in the IR. It describes stored data with columns and sometimes a primary key. +- `law`: A logical rule or constraint in the IR. It says what relationships must hold about the data. +- `law checking`: Testing whether the current data satisfies a law. +- `query engine`: The execution component that consumes the IR and answers queries. In a broader design it may also do law checking, chase steps, and + fixpoint work. +- `backend`: A concrete execution target, such as an in-memory database, Postgres, or a Rust-native storage/query layer. +- `backend-neutral`: Not tied to one storage or execution technology. The IR should ideally be backend-neutral. +- `RelTable`: A table representing a relation or predicate. +- `FunTable`: A table representing something function-like, where input columns determine an output. +- `foreignKeys`: Generated laws saying that if one row exists, dependent rows must also exist. +- `total`: Generated laws saying that if the inputs to a function exist, an output row must exist too. +- `runtime IR`: A second execution-oriented representation above the lowered theory IR. It would hold changing state needed during execution. +- `runtime state`: Mutable or evolving execution data, such as current facts, branch identity, witness allocation, equality classes, or provenance. +- `fixpoint`: The point where running the rules produces no new facts or changes. +- `chase`: The general reasoning process that repeatedly checks laws and adds whatever is needed to satisfy them. +- `witness`: A fresh element introduced to satisfy an existential conclusion. +- `branch`: One possible world or execution path created by a disjunctive rule. +- `equality merging`: Collapsing two terms or elements when the theory says they must be equal. +- `provenance`: Information about where a fact came from or which rules produced it. +- `planning`: Choosing how to execute a query or check a law efficiently. +- `adapter`: A layer that maps shared IR execution concepts onto one specific backend. + +## Short mental model + +- the front end parses, elaborates, and lowers +- the IR is the contract +- the query engine consumes that contract +- the backend is where execution actually lands diff --git a/scratches/query-engine-narrow.dot b/scratches/query-engine-narrow.dot index f769b63..75be7fe 100644 --- a/scratches/query-engine-narrow.dot +++ b/scratches/query-engine-narrow.dot @@ -1,147 +1,147 @@ digraph GeologQueryEngineNarrow { -fontname = "Helvetica,Arial,sans-serif" -layout = dot -rankdir = LR -ranksep = 0.9; -nodesep = 0.7; -splines = true; -compound = true; -bgcolor = "white" + fontname = "Helvetica,Arial,sans-serif" + layout = dot + rankdir = LR + ranksep = 0.9; + nodesep = 0.7; + splines = true; + compound = true; + bgcolor = "white" -node [ -fontname = "Helvetica,Arial,sans-serif", -shape = box, -style = "filled,rounded", -color = "#555555", -fillcolor = "white", -penwidth = 1.5 -] -edge [ -fontname = "Helvetica,Arial,sans-serif", -color = "#333333", -fontsize = 9, -fontcolor = "#555555", -labeldistance = 2.0, -penwidth = 1.2 -] + node [ + fontname = "Helvetica,Arial,sans-serif", + shape = box, + style = "filled,rounded", + color = "#555555", + fillcolor = "white", + penwidth = 1.5 + ] + edge [ + fontname = "Helvetica,Arial,sans-serif", + color = "#333333", + fontsize = 9, + fontcolor = "#555555", + labeldistance = 2.0, + penwidth = 1.2 + ] -subgraph cluster_frontend { -label = "Geolog Front End" -style = "rounded" -color = "#666666" -fontcolor = "#333333" -bgcolor = "#FAFAFA" -margin = 20 + subgraph cluster_frontend { + label = "Geolog Front End" + style = "rounded" + color = "#666666" + fontcolor = "#333333" + bgcolor = "#FAFAFA" + margin = 20 -subgraph cluster_language { -label = "Language Pipeline" -style = "dashed" -color = "#9C27B0" -fontcolor = "#7B1FA2" -margin = 14 -concrete [label = "Concrete Syntax", fillcolor = "#F3E5F5", color = "#9C27B0"] -parser [label = "Parser", fillcolor = "#F3E5F5", color = "#9C27B0"] -abstract [label = "Abstract Syntax", fillcolor = "#F3E5F5", color = "#9C27B0"] -elaborator [label = "Elaborator", fillcolor = "#F3E5F5", color = "#9C27B0"] -typed [label = "Elaborated Syntax\n(with types)", fillcolor = "#F3E5F5", color = "#9C27B0"] -lowering [label = "Lowering", fillcolor = "#F3E5F5", color = "#9C27B0"] -ir [label = "Intermediate Representation\n(backend-neutral contract)", fillcolor = "#F3E5F5", color = "#9C27B0"] -} - -subgraph cluster_tools { -label = "Developer Tooling" -style = "dashed" -color = "#4CAF50" -fontcolor = "#388E3C" -margin = 14 -lsp [label = "LSP\n(parsing + elaboration diagnostics)", fillcolor = "#E8F5E9", color = "#4CAF50"] -} -} - -subgraph cluster_execution { -label = "Execution Layer" -style = "dashed" -color = "#4CAF50" -fontcolor = "#388E3C" -margin = 18 -query_translation [label = "IR to Backend Translation", fillcolor = "#E8F5E9", color = "#4CAF50"] -} - -subgraph cluster_backends { -label = "Backend Implementations" -style = "rounded" -color = "#666666" -fontcolor = "#333333" -bgcolor = "#FAFAFA" -margin = 20 - -subgraph cluster_inmemory { -label = "In-Memory Backend" -style = "dashed" -color = "#FF9800" -fontcolor = "#F57C00" -margin = 14 -inmem_rel [label = "Relation Translation", fillcolor = "#FFF3E0", color = "#FF9800"] -inmem_laws [label = "Law / Chase Translation", fillcolor = "#FFF3E0", color = "#FF9800"] -inmem_qe [label = "In-Memory Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] -inmem_db [label = "In-Memory DB", fillcolor = "#ECEFF1", color = "#607D8B"] -} - -subgraph cluster_postgres { -label = "Postgres Backend" -style = "dashed" -color = "#FF9800" -fontcolor = "#F57C00" -margin = 14 -pg_schema [label = "IR to Schema", fillcolor = "#FFF3E0", color = "#FF9800"] -pg_queries [label = "IR to SQL Queries", fillcolor = "#FFF3E0", color = "#FF9800"] -pg_qe [label = "Postgres Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] -pg_db [label = "Postgres DB", fillcolor = "#ECEFF1", color = "#607D8B"] -} - -subgraph cluster_rust { -label = "Rust Backend" -style = "dashed" -color = "#FF9800" -fontcolor = "#F57C00" -margin = 14 -ffi [label = "FFI Haskell to Rust\n(IR marshalling)", fillcolor = "#FFF3E0", color = "#FF9800"] -rust_rel [label = "Relation Translation", fillcolor = "#FFF3E0", color = "#FF9800"] -rust_laws [label = "Law / Chase Translation", fillcolor = "#FFF3E0", color = "#FF9800"] -rust_qe [label = "Rust Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] -rust_store [label = "Rust Storage / Query Layer", fillcolor = "#ECEFF1", color = "#607D8B"] -} -} - -concrete -> parser [color = "#9C27B0"] -parser -> abstract [color = "#9C27B0"] -abstract -> elaborator [color = "#9C27B0"] -elaborator -> typed [color = "#9C27B0"] -typed -> lowering [color = "#9C27B0"] -lowering -> ir [color = "#9C27B0"] - -parser -> lsp [style = "dashed", color = "#4CAF50", label = "syntax info"] -elaborator -> lsp [style = "dashed", color = "#4CAF50", label = "type info"] - -ir -> query_translation [color = "#4CAF50", label = "compile IR"] - -query_translation -> inmem_rel [color = "#FF9800"] -query_translation -> inmem_laws [color = "#FF9800"] -query_translation -> pg_schema [color = "#FF9800"] -query_translation -> pg_queries [color = "#FF9800"] -query_translation -> ffi [color = "#FF9800"] - -inmem_rel -> inmem_qe [color = "#FF9800"] -inmem_laws -> inmem_qe [color = "#FF9800"] -inmem_qe -> inmem_db [color = "#607D8B"] - -pg_schema -> pg_db [color = "#607D8B"] -pg_queries -> pg_qe [color = "#FF9800"] -pg_qe -> pg_db [color = "#607D8B"] - -ffi -> rust_rel [color = "#FF9800"] -ffi -> rust_laws [color = "#FF9800"] -rust_rel -> rust_qe [color = "#FF9800"] -rust_laws -> rust_qe [color = "#FF9800"] -rust_qe -> rust_store [color = "#607D8B"] + subgraph cluster_language { + label = "Language Pipeline" + style = "dashed" + color = "#9C27B0" + fontcolor = "#7B1FA2" + margin = 14 + concrete [label = "Concrete Syntax", fillcolor = "#F3E5F5", color = "#9C27B0"] + parser [label = "Parser", fillcolor = "#F3E5F5", color = "#9C27B0"] + abstract [label = "Abstract Syntax", fillcolor = "#F3E5F5", color = "#9C27B0"] + elaborator [label = "Elaborator", fillcolor = "#F3E5F5", color = "#9C27B0"] + typed [label = "Elaborated Syntax\n(with types)", fillcolor = "#F3E5F5", color = "#9C27B0"] + lowering [label = "Lowering", fillcolor = "#F3E5F5", color = "#9C27B0"] + ir [label = "Intermediate Representation\n(backend-neutral contract)", fillcolor = "#F3E5F5", color = "#9C27B0"] + } + + subgraph cluster_tools { + label = "Developer Tooling" + style = "dashed" + color = "#4CAF50" + fontcolor = "#388E3C" + margin = 14 + lsp [label = "LSP\n(parsing + elaboration diagnostics)", fillcolor = "#E8F5E9", color = "#4CAF50"] + } + } + + subgraph cluster_execution { + label = "Execution Layer" + style = "dashed" + color = "#4CAF50" + fontcolor = "#388E3C" + margin = 18 + query_translation [label = "IR to Backend Translation", fillcolor = "#E8F5E9", color = "#4CAF50"] + } + + subgraph cluster_backends { + label = "Backend Implementations" + style = "rounded" + color = "#666666" + fontcolor = "#333333" + bgcolor = "#FAFAFA" + margin = 20 + + subgraph cluster_inmemory { + label = "In-Memory Backend" + style = "dashed" + color = "#FF9800" + fontcolor = "#F57C00" + margin = 14 + inmem_rel [label = "Relation Translation", fillcolor = "#FFF3E0", color = "#FF9800"] + inmem_laws [label = "Law / Chase Translation", fillcolor = "#FFF3E0", color = "#FF9800"] + inmem_qe [label = "In-Memory Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] + inmem_db [label = "In-Memory DB", fillcolor = "#ECEFF1", color = "#607D8B"] + } + + subgraph cluster_postgres { + label = "Postgres Backend" + style = "dashed" + color = "#FF9800" + fontcolor = "#F57C00" + margin = 14 + pg_schema [label = "IR to Schema", fillcolor = "#FFF3E0", color = "#FF9800"] + pg_queries [label = "IR to SQL Queries", fillcolor = "#FFF3E0", color = "#FF9800"] + pg_qe [label = "Postgres Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] + pg_db [label = "Postgres DB", fillcolor = "#ECEFF1", color = "#607D8B"] + } + + subgraph cluster_rust { + label = "Rust Backend" + style = "dashed" + color = "#FF9800" + fontcolor = "#F57C00" + margin = 14 + ffi [label = "FFI Haskell to Rust\n(IR marshalling)", fillcolor = "#FFF3E0", color = "#FF9800"] + rust_rel [label = "Relation Translation", fillcolor = "#FFF3E0", color = "#FF9800"] + rust_laws [label = "Law / Chase Translation", fillcolor = "#FFF3E0", color = "#FF9800"] + rust_qe [label = "Rust Query Engine", fillcolor = "#FFF3E0", color = "#FF9800"] + rust_store [label = "Rust Storage / Query Layer", fillcolor = "#ECEFF1", color = "#607D8B"] + } + } + + concrete -> parser [color = "#9C27B0"] + parser -> abstract [color = "#9C27B0"] + abstract -> elaborator [color = "#9C27B0"] + elaborator -> typed [color = "#9C27B0"] + typed -> lowering [color = "#9C27B0"] + lowering -> ir [color = "#9C27B0"] + + parser -> lsp [style = "dashed", color = "#4CAF50", label = "syntax info"] + elaborator -> lsp [style = "dashed", color = "#4CAF50", label = "type info"] + + ir -> query_translation [color = "#4CAF50", label = "compile IR"] + + query_translation -> inmem_rel [color = "#FF9800"] + query_translation -> inmem_laws [color = "#FF9800"] + query_translation -> pg_schema [color = "#FF9800"] + query_translation -> pg_queries [color = "#FF9800"] + query_translation -> ffi [color = "#FF9800"] + + inmem_rel -> inmem_qe [color = "#FF9800"] + inmem_laws -> inmem_qe [color = "#FF9800"] + inmem_qe -> inmem_db [color = "#607D8B"] + + pg_schema -> pg_db [color = "#607D8B"] + pg_queries -> pg_qe [color = "#FF9800"] + pg_qe -> pg_db [color = "#607D8B"] + + ffi -> rust_rel [color = "#FF9800"] + ffi -> rust_laws [color = "#FF9800"] + rust_rel -> rust_qe [color = "#FF9800"] + rust_laws -> rust_qe [color = "#FF9800"] + rust_qe -> rust_store [color = "#607D8B"] } diff --git a/scratches/query-engine-narrow.svg b/scratches/query-engine-narrow.svg new file mode 100644 index 0000000..05610ae --- /dev/null +++ b/scratches/query-engine-narrow.svg @@ -0,0 +1,493 @@ + + + + + + + GeologQueryEngineNarrow + + + cluster_frontend + + Geolog Front End + + + + cluster_language + + Language Pipeline + + + + cluster_tools + + Developer Tooling + + + + cluster_execution + + Execution Layer + + + + cluster_backends + + Backend Implementations + + + + cluster_inmemory + + In-Memory Backend + + + + cluster_postgres + + Postgres Backend + + + + cluster_rust + + Rust Backend + + + + + concrete + + + Concrete Syntax + + + + + parser + + + Parser + + + + + concrete->parser + + + + + + abstract + + + Abstract Syntax + + + + + parser->abstract + + + + + + lsp + + + LSP + + + (parsing + elaboration diagnostics) + + + + + parser->lsp + + + syntax info + + + + + elaborator + + + Elaborator + + + + + abstract->elaborator + + + + + + typed + + + Elaborated Syntax + + + (with types) + + + + + elaborator->typed + + + + + + elaborator->lsp + + + type info + + + + + lowering + + Lowering + + + + + typed->lowering + + + + + + ir + + Intermediate Representation + + (backend-neutral contract) + + + + + lowering->ir + + + + + + query_translation + + + IR to Backend Translation + + + + + ir->query_translation + + + compile IR + + + + + inmem_rel + + Relation Translation + + + + + query_translation->inmem_rel + + + + + + inmem_laws + + Law / Chase Translation + + + + + query_translation->inmem_laws + + + + + + pg_schema + + + IR to Schema + + + + + query_translation->pg_schema + + + + + + pg_queries + + IR to SQL Queries + + + + + query_translation->pg_queries + + + + + + ffi + + FFI Haskell to Rust + + (IR marshalling) + + + + + query_translation->ffi + + + + + + inmem_qe + + In-Memory Query Engine + + + + + inmem_rel->inmem_qe + + + + + + inmem_laws->inmem_qe + + + + + + inmem_db + + In-Memory DB + + + + + inmem_qe->inmem_db + + + + + + pg_db + + Postgres DB + + + + + pg_schema->pg_db + + + + + + pg_qe + + Postgres Query Engine + + + + + pg_queries->pg_qe + + + + + + pg_qe->pg_db + + + + + + rust_rel + + Relation Translation + + + + + ffi->rust_rel + + + + + + rust_laws + + Law / Chase Translation + + + + + ffi->rust_laws + + + + + + rust_qe + + Rust Query Engine + + + + + rust_rel->rust_qe + + + + + + rust_laws->rust_qe + + + + + + rust_store + + Rust Storage / Query Layer + + + + + rust_qe->rust_store + + + + +