Add note files about fast search and rule engines
This commit is contained in:
parent
9c6f689447
commit
40ccf7ae69
104
hqew/010-search-and-vector-query-engines.md
Normal file
104
hqew/010-search-and-vector-query-engines.md
Normal file
@ -0,0 +1,104 @@
|
||||
# Search and Vector Query Engines
|
||||
|
||||
A reference for how search-oriented engines differ from classical relational ones.
|
||||
|
||||
---
|
||||
|
||||
## Short answer
|
||||
|
||||
Search and vector engines are still query engines, but their core operators, indexes, and ranking models differ from standard relational SQL systems.
|
||||
|
||||
They are often built around retrieval rather than exact relational transformation.
|
||||
|
||||
---
|
||||
|
||||
## Search engines
|
||||
|
||||
Traditional search engines center on:
|
||||
|
||||
- inverted indexes
|
||||
- document retrieval
|
||||
- ranking and scoring
|
||||
- boolean and text queries
|
||||
|
||||
The core question is often not just "which rows satisfy a predicate?" but "which documents are most relevant?"
|
||||
|
||||
That makes ranking a first-class part of execution.
|
||||
|
||||
---
|
||||
|
||||
## Vector engines
|
||||
|
||||
Vector engines center on:
|
||||
|
||||
- embeddings
|
||||
- nearest-neighbor search
|
||||
- approximate similarity retrieval
|
||||
- metadata filtering
|
||||
|
||||
The core operation is often "find the closest vectors to this query vector," which is different from equality joins or exact predicate evaluation.
|
||||
|
||||
Approximation is often acceptable because speed matters and exact nearest-neighbor search can be too expensive at scale.
|
||||
|
||||
---
|
||||
|
||||
## Hybrid systems
|
||||
|
||||
Many modern systems combine:
|
||||
|
||||
- lexical search
|
||||
- vector search
|
||||
- metadata filters
|
||||
- reranking
|
||||
|
||||
This means the engine may need to merge several candidate-generation and scoring paths into one final result.
|
||||
|
||||
---
|
||||
|
||||
## How these engines differ from relational ones
|
||||
|
||||
Search and vector engines often emphasize:
|
||||
|
||||
- ranking
|
||||
- retrieval quality
|
||||
- approximate indexing
|
||||
- candidate generation
|
||||
- top-k execution
|
||||
|
||||
Relational engines more often emphasize:
|
||||
|
||||
- exact semantics
|
||||
- joins
|
||||
- aggregations
|
||||
- transactional or analytical correctness
|
||||
|
||||
The difference is not that one has a query engine and the other does not. The difference is what the engine is optimizing for.
|
||||
|
||||
---
|
||||
|
||||
## Example systems
|
||||
|
||||
Examples of search and vector query engines include:
|
||||
|
||||
- Lucene
|
||||
- Vespa
|
||||
- Weaviate
|
||||
- Qdrant
|
||||
|
||||
They differ in packaging and scope, but all have real planning and execution concerns around retrieval.
|
||||
|
||||
---
|
||||
|
||||
## Practical mental model
|
||||
|
||||
Relational engines are often about exact transformation of structured data.
|
||||
|
||||
Search and vector engines are often about efficient retrieval and ranking over high-dimensional or text-heavy data.
|
||||
|
||||
That is the cleanest conceptual distinction.
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
* **April 1, 2026** -- First version created.
|
||||
98
hqew/011-rule-engines-and-fixpoint-evaluation.md
Normal file
98
hqew/011-rule-engines-and-fixpoint-evaluation.md
Normal file
@ -0,0 +1,98 @@
|
||||
# Rule Engines and Fixpoint Evaluation
|
||||
|
||||
A reference for query engines whose core work is inference, recursion, or repeated rule application.
|
||||
|
||||
---
|
||||
|
||||
## Short answer
|
||||
|
||||
Not all query engines are centered on SQL-style scans, joins, and aggregates.
|
||||
|
||||
Some are organized around:
|
||||
|
||||
- facts
|
||||
- rules
|
||||
- recursion
|
||||
- derivation until no new information appears
|
||||
|
||||
That is the world of rule engines, Datalog engines, and chase-style systems.
|
||||
|
||||
---
|
||||
|
||||
## The core idea
|
||||
|
||||
Instead of asking only for direct data retrieval, a rule engine often asks:
|
||||
|
||||
- what facts follow from these rules?
|
||||
- what closure or least fixpoint do these inputs imply?
|
||||
- what new values must exist to satisfy constraints?
|
||||
|
||||
This makes the engine iterative in a deeper way than many classical relational plans.
|
||||
|
||||
---
|
||||
|
||||
## Fixpoint evaluation
|
||||
|
||||
Fixpoint evaluation means:
|
||||
|
||||
1. start with known facts
|
||||
2. apply rules
|
||||
3. add newly derived facts
|
||||
4. repeat until nothing new is produced
|
||||
|
||||
That final stable state is the fixpoint.
|
||||
|
||||
This is central to:
|
||||
|
||||
- recursive Datalog
|
||||
- transitive closure
|
||||
- many inference systems
|
||||
|
||||
---
|
||||
|
||||
## How this differs from standard relational execution
|
||||
|
||||
Relational engines often execute one finite plan over a fixed input.
|
||||
|
||||
Rule engines often need:
|
||||
|
||||
- iterative scheduling
|
||||
- duplicate elimination
|
||||
- dependency tracking
|
||||
- recursion support
|
||||
- possibly witness generation or equality merging
|
||||
|
||||
So the runtime model is often different even when some individual operators overlap.
|
||||
|
||||
---
|
||||
|
||||
## Why this matters
|
||||
|
||||
Rule/fixpoint execution is especially relevant when the system needs:
|
||||
|
||||
- recursive reasoning
|
||||
- closure computation
|
||||
- derivation of implicit facts
|
||||
- chase-like enforcement of dependencies
|
||||
|
||||
This makes it a natural topic whenever query engines overlap with logic or theorem-like data processing.
|
||||
|
||||
---
|
||||
|
||||
## Practical mental model
|
||||
|
||||
Standard query engines answer:
|
||||
|
||||
- what result follows from this query over this data?
|
||||
|
||||
Rule engines also ask:
|
||||
|
||||
- what additional facts become true once the rules have run to completion?
|
||||
|
||||
That is the essential distinction.
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
* **April 1, 2026** -- First version created.
|
||||
Loading…
x
Reference in New Issue
Block a user