storage-engine-playground/notes/backend/01-cozo-and-lmdb-findings.md

Cozo and LMDB Findings

Sources inspected: the Cozo source tree at github.com/cozodb/cozo, the LMDB source tree at github.com/LMDB/lmdb, and the heed Rust binding at github.com/meilisearch/heed. File paths in this note are relative to the root of the named project's source tree. The aim was to understand how a working Datalog engine (Cozo) implements joins and what a low-level key-value substrate (LMDB) provides that makes those joins cheap. This note summarizes the design lessons and the practical implications for the query-ops crate in this playground.

Summary

Cozo is an embedded Datalog database written in Rust. It does not have a separate semijoin operator. Instead, it has one inner-join operator that picks between two strategies based on how each relation is stored: an index-nested-loop strategy that uses ordered range scans over the substrate, and a fallback that materializes one side into a sorted vector and probes it. Semijoin behavior, when needed, emerges from a separate rewrite step called the magic-sets transformation, which converts semijoin-shaped pruning into regular inner joins against derived relations.

LMDB is a memory-mapped, ordered key-value store with a B+ tree on disk. It exposes a small set of cursor primitives that support prefix iteration, range iteration, and exact-key lookup. These primitives are exactly what an index-nested-loop join needs: seek to a key prefix, then iterate forward while the prefix matches.

The combined lesson is that a good join does not require a clever operator. It requires the relation to be stored with the join columns at the front of the key, so that the substrate's ordered iteration can do the join itself.

Cozo

What It Is

Cozo is a Datalog database with multiple swappable storage backends, including an in-memory store, SQLite, RocksDB, sled, and TiKV. The execution engine speaks a single narrow storage trait whose surface is essentially get, put, range_iter, and prefix_iter over byte keys. Each backend implements that trait. The trait definition lives at cozo-core/src/storage/mod.rs in the Cozo source tree.

Join Behavior

The relational algebra at cozo-core/src/query/ra.rs in the Cozo source tree defines a single join operator named InnerJoin. At execution time it chooses between two strategies based on a check called join_is_prefix:

• prefix join: for each tuple from the left side, the engine builds a byte prefix from the join columns and calls prefix_iter on the right relation. The substrate yields all matching tuples in key order. No hash table is built. This path is taken whenever the right side's join columns are stored as the prefix of its key.
• materialized join: used when the join columns are not a key prefix. The right side is read fully into a sorted, deduplicated vector, reordered so the join columns come first, then walked with a starts_with(prefix) check. This is the build-and-probe family, but with a sorted vector instead of a hash map.

The choice is made entirely on whether the join columns sit at the front of the stored key.

No Semijoin Operator

A search of the Cozo source for semijoin or semi_join returns nothing. Semijoin behavior comes from the magic-sets transformation at cozo-core/src/query/magic.rs in the Cozo source tree. This pass rewrites each rule so that body atoms get joined against an auxiliary "magic" relation whose contents encode the binding patterns supplied by the rule's callers. The net effect is the same as semijoining body atoms against caller-supplied filters, but the implementation is a logical rewrite, not a runtime operator.

No Auto-Maintained Secondary Indexes

Cozo does not maintain secondary indexes automatically. If you want to query a relation by a column order different from how it was declared, you declare a second relation with the columns reordered and keep its contents synchronized at insert time. A covering index is just another stored relation. The decision of which column order to store comes from how you expect to query the data, not from the engine.

LMDB

What It Is

LMDB is a single-file, memory-mapped, ordered key-value store. It uses a B+ tree on disk and exposes reads as zero-copy byte slices that point directly into the mmap. It supports a single writer at a time and many concurrent readers, and it uses shadow paging for MVCC, which means commits are atomic without a write-ahead log.

Cursor Primitives

A cursor in LMDB is a position inside the B+ tree. The full set of cursor operations is defined by the MDB_cursor_op enum in libraries/liblmdb/lmdb.h in the LMDB source tree. The operations relevant to join work are:

• MDB_SET_RANGE: position at the first key greater than or equal to a given key. This is the seek primitive that makes prefix scans possible.
• MDB_NEXT: advance one step forward in key order. Combined with MDB_SET_RANGE and a per-step prefix check, this gives you ordered range iteration.
• MDB_SET and MDB_SET_KEY: exact-key positioning, used for point lookups.
• MDB_FIRST and MDB_LAST: positional endpoints.

For databases opened with the MDB_DUPSORT flag, one key can carry multiple sorted values, and additional operations apply: MDB_GET_BOTH, MDB_NEXT_DUP, MDB_FIRST_DUP. This is useful when a relation is encoded as "key = join columns, duplicate values = remaining columns": the set of duplicates is itself a secondary index over the join key.

Rust Binding

heed is the idiomatic Rust binding for LMDB. It wraps the cursor operations as RoCursor and RwCursor and returns key and value byte slices tied to the transaction lifetime, so reads remain zero-copy. Meilisearch uses heed in production, so the binding is well exercised.

LMDB Versus RocksDB

Both LMDB and RocksDB are ordered key-value stores with prefix and range scans, but their internal designs lead to different operational profiles.

LMDB highlights:

• B+ tree on disk, memory mapped
• Single writer at a time, many concurrent readers
• Zero-copy reads from the mmap
• Append-only on-disk format; deletes leave reclaimable free pages
• File size grows up to a configured mapsize
• No background compaction
• Manual reclaim with mdb_copy --compact

RocksDB highlights:

• Log-structured merge tree
• Multiple concurrent writers
• Background compaction
• Higher write throughput at the cost of write amplification
• Reads may traverse multiple levels with bloom-filter checks
• Engine manages its own disk layout

For a read-heavy prototype with batch inserts, LMDB is the closer fit: predictable read costs, cheap range scans, and zero-copy probes. RocksDB earns its overhead when sustained write throughput is the bottleneck.

Practical Implications

The current query-ops crate works on in-memory Vec<Row> values and will implement semijoin and natural join with a transient hash on one side. The Cozo design suggests a clear upgrade path once a real substrate is added.

Short term: keep the in-memory operator and build a transient hash on the smaller side. This is correct, easy to test, and easy to reason about.

Medium term: when relations move into a substrate like LMDB, encode each relation so that the join columns sit at the prefix of the key, or use a DUPSORT database where the duplicate values carry the remaining columns. At that point the join operator becomes a cursor pattern (MDB_SET_RANGE followed by MDB_NEXT while the prefix matches), and the separate hash-building step disappears.

Index discipline: if a relation needs to be joined two different ways, store it twice with different prefix orders. There is no clever-indexing shortcut in either Cozo or LMDB, and trying to invent one is unlikely to be worth the cost.

The takeaway is that the operator surface in query-ops is fine for an in-memory prototype, but the substrate decision is the load-bearing one for performance. We do not need to design around it now, but the natural successor to the current operators is a key-encoding discipline rather than a more elaborate operator implementation.

Changelog

• June 2, 2026 -- The first version of this document was made.