Crafter.DB/AGENTS.consumer.md

## Crafter.DB

The Catcrafts embedded table store: a C++26 module, `import Crafter.DB;`, in which a table is a plain struct with a `version`, a file is a memory-mapped array of fixed-width rows with an out-of-line heap for strings and blobs, reads are zero-copy views, queries are lambdas or column expressions that compose with `std::ranges`, and schema changes are C++ migrations run on open. Everything is in `namespace Crafter::DB` (the examples below assume `using namespace Crafter;`). Linux only: the engine uses `flock`, `fallocate` and fixed-address `mmap`. No dependencies, no link flags. Source and issues: https://forgejo.catcrafts.net/Catcrafts/Crafter.DB (LGPL-3.0-only). The README there is the manual, with the whole API and the file format; these notes are its `AGENTS.consumer.md`: a fix to them goes there, not into the copy crafter-build writes.

### Building against it

- `GitProject({ .source = { .url = "https://forgejo.catcrafts.net/Catcrafts/Crafter.DB.git" } })` (add `.commit` to pin), or `LocalProject({ .projectFile = "../Crafter.DB/project.cpp" })` for a sibling checkout, into `cfg.dependencies`. It is a static library whose partitions are nearly all templates: everything a `Table<T>` does is instantiated in your translation units with your target and `--march`, and the only out-of-line code is the OS layer and the type-erased engine. It reads no project args and has no dependencies of its own, so a flag of yours that reaches it only changes its variant hash (a separate `bin/` directory per flag combination); strip your own flags from the `.args` you pass down if you would rather not, and pass the same list on every path that reaches it, or the build stops with two loads that disagree.
- Only Linux targets build it (`x86_64-unknown-linux-gnu` and `aarch64-alpine-linux-musl` have been built; the tests run on `*-linux-gnu`). A project that also targets `wasm32-*` or `x86_64-w64-mingw32` adds it to `cfg.dependencies` only when `cfg.target` contains `-linux-`, and keeps the code that uses it behind the same condition. The compiler is crafter-build's clang 23 with `import std;`; the library leans on C++26 features (structured-binding packs, pack indexing, generated `static_assert` messages) that no other toolchain in use here has.
- Nothing is defined for you to `#ifdef` on. Compile-time facts about a table come from `DB::Layout<T>`: `rowSize`, `rowAlign`, `columnCount`, `fingerprint`, and `columns[i].kind`, `.offset` and `.name.View()` (the name is a fixed-capacity text, not a `std::string_view`).

### Defining a table

A table struct is an aggregate (public members, no base classes with members, no user-declared constructors), default constructible, with `static constexpr std::uint32_t version = 1;`. Optional statics: `static constexpr auto orderedBy = &T::member;` (an integer, enum, or integral-rep `duration` or `time_point` member; enforced on every write, enables binary search), `using Previous = OlderT;` and `static T Migrate(OlderT&&)` (see "Schema changes and migrations"). Every rule about the struct and its members is a `static_assert` whose message starts with `Crafter.DB:` and names them, so read the first error, not the instantiation stack.

- Storable members: arithmetic types except `long double`, `bool`, `std::byte`, enums (stored as their underlying integer), `std::chrono::duration` and `time_point` (rep plus period, with the clock in the schema), `std::string`, `std::vector<E>` for trivially copyable non-pointer `E`, `std::optional<M>` of a storable `M` (not of another optional), `std::array<E, N>` of scalars (one column per element) or of bytes (one fixed column), `DB::FixedString<N>` (N bytes inline, NUL padded), nested aggregates of storable members (flattened to `outer.inner` columns), and anything with a `Crafter::DB::Formatter<T>` specialisation. Pointers, references and member pointers are rejected.
- A struct with constructors is not an aggregate and is refused with a message that says to replace them with designated initialisers; `Table::Emplace` builds rows from fields, so the constructors are not missed.
- Columns are not laid out in declaration order: the flags byte sits at offset 0 and the leaves follow in ascending alignment, 2-byte columns before 8-byte ones. The README's seven-member `Transaction` is 88 bytes. A `std::string` or `std::vector` column is a 16-byte handle holding values of up to 12 bytes inline, and the first 4 bytes of longer ones (a vector of 8-byte-aligned elements such as `std::uint64_t` or `double` always goes to the heap), so equality against such a column is one 8-byte compare per row unless the lengths and prefixes match. Keys you filter on often are best kept at 12 bytes or under where the data allows (a BIC is, an IBAN is not); a code with a fixed maximum length is a `FixedString<N>`.
- Your own types: specialise `Formatter<T>` the way you would `std::formatter`. A fixed-width one declares `using Stored = <storable type>;` with `static Stored Encode(const T&)` and `static T Decode(const Stored&)`; a variable-length one inherits `Formatter<std::string>` (or `Formatter<std::vector<E>>`) and overrides `static Var Encode(const T&, HeapWriter&)` and `static T Materialize(StringRef)`. A row still hands out the raw `StringRef` or `BlobRef` for a variable-length member; `Materialize()` is where your type comes back.
- Once the struct is settled, pin `DB::Layout<T>::rowSize` and `DB::Layout<T>::fingerprint` with a `static_assert` in a test: a failing assert is the compile-time form of the `SchemaError` your users would get at open, and tells you a `version` bump is due.

### Opening and lifetime

`DB::Table<T> table(path);` creates the file when missing (directories `0700`, file `0600`), migrates it when older, and throws when it cannot. `DB::Table<T>(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = true })` opens a frozen reader. `OpenOptions` takes designated initialisers in this order: `mode` (`OpenOrCreate`, `OpenExisting`, `CreateNew`), `readOnly`, `createDirectories`, `filePermissions`, `directoryPermissions`, `durability` (`OnFlush`, `EveryAppend`), `migrate`, `backupBeforeMigrate`, `reserveBytes` (32 GiB), `excludeFromCoreDumps`, `prefetch`.

- **Open once per process and keep the table open.** An open maps every extent of the file (one `mmap` each: 11 for the Benchmark's 100 000 rows appended in one batch, about 20 for the same rows appended one at a time) and a read-only open costs about 30 µs, which dominates any code that opens a table per query. A `Table` is move-only: hold it in a member or a `static`, not in each function that reads.
- **One writer per file at a time, across processes and within one.** A writer holds an exclusive `flock`, a read-only table a shared one: a second writable `Table` on the file, in another process or in this one, throws `LockedError`, a read-only table anywhere makes a writer's open fail, and a writer makes every reader's. Read-only tables coexist. Two processes that create the same table at once end up sharing one file. The lock is advisory: `cp` and other tools ignore it, and a copy taken while a writer is open reads back as of the writer's last commit record, which is its last `Flush`.
- **Durability is at `Flush()`, `Close()` and destruction, not at `Append`.** Under `Durability::OnFlush` (the default) an appended row is visible to this process at once and reaches the disk at the next `Flush`; a crash loses what was not flushed and never a committed row. An `Update` or `Set` that writes heap bytes (a string longer than 12 bytes, a blob) flushes on its own, an `fdatasync` each, and so commits every row appended before it. `Durability::EveryAppend` flushes after every write. The destructor flushes but cannot report; call `Close()` where a failed final flush must throw. After `fork()` the child may read the inherited table; every write and every explicit `Flush()` in it throws `Error`, while `Close()` and the destructor skip the flush silently, so the child's stale counters never reach the file.
- **Files beside the table**: `<file>.tmp-<12 hex>` while the table is being created and while `Compact`, `Clear` or a migration is writing (stale ones go on the next writable open), and `<file>.v<old>.bak` after a migration (a hard link of the pre-migration file; delete it when it has served). The directory must stay writable for those three operations. Use a directory per table or distinct basenames; `.cdb` is the convention.
- **Address space.** Each open table reserves `reserveBytes` twice (rows and heap), 64 GiB of virtual space by default, none of it committed. Under a `ulimit -v`, or with hundreds of tables open, lower `reserveBytes` to a few times the table's size; a reservation that fills during an append throws `IOError` naming the option.
- A table created on a 4 KiB-page kernel cannot be opened on a 16 KiB-page one (`IOError`, `not_supported`); the extent granule is recorded in the header.

### Writing

`Append(row)` and `Append(range)` return the `RowId` of the first new row (ids are slot indices from 0, stable until `Compact` or `Clear`); a range is one reservation and one publish, all or nothing. `Emplace(args...)` takes exactly one argument per member in declaration order, converted implicitly (a `std::string_view` is enough for a string member; an `int` for an enum member does not compile). `Update(id, row)` replaces a live row; `Set<&T::member>(id, value)` one member; `Erase(id)` tombstones (false when already erased); `EraseIf(pred)` returns the count; `Flush()`; `Compact()` rewrites without tombstones and heap garbage; `Clear()` empties.

- On an `orderedBy` table a write whose key would sort before the previous row (or, for `Update` and `Set`, after the next) throws `OrderError` before anything is written; equal keys are allowed; an unsorted batch is refused whole. Rows must arrive in key order: sort before `Append`, or leave `orderedBy` out and sort when reading.
- `Update`, `Set` and `Erase` leave heap garbage (the old string bytes); `Stats().heapGarbage` says how much, and only `Compact()` reclaims it. `Compact` and `Clear` invalidate every `RowId`, `Row` and `StringRef` taken before them; `Update` and `Set` invalidate the views of that row; `Append` and `Erase` invalidate nothing.
- `Update` and `Set` never leave a row pointing at heap bytes that are not on disk (those are committed first), but the row's fixed-width columns are not written atomically: a crash or power loss mid-row leaves the old row, the new one, or a mix of the two. A log of appends is the shape the library is built for.
- Limits: a value below 4 GiB (`Error`), a row of at most 65535 bytes and 65535 columns (compile time), a region of at most 32 extents, roughly 17 GiB (`IOError` saying to `Compact()`), and the disk (`fallocate` fails with `IOError` before any page is touched, never a `SIGBUS`).

### Reading and querying

`table.Size()` is the live rows, `SlotCount()` the id space including tombstones, `table[id]` an unchecked `Row<T>` view (16 bytes, no allocation), `At(id)` a checked one, `Contains(id)`, `Front()`, `Back()` (skips trailing tombstones). `row[&T::member]` and `row.Get<&T::member>()` read one column from the mapping: the value for scalars, chrono and `FixedString`; a `StringRef` for a string (converts to `std::string_view`, compares with `==` to literals, `std::string_view`, `std::string`, `Key` and other refs without building a view, orders with `<=>`); a `BlobRef<E>` for a vector (converts to `std::span<const E>`); an `std::optional` of the view for an optional. `row.Materialize()` (also `explicit operator T`) copies the row out; `table.Materialize()` every live row. `std::format("{}", row)` prints `Type{member=value, …}`.

- `for (DB::Row<T> row : table)` is the live rows in id order and composes with `std::views`; `table.Slots()` is random access over every slot with `row.Erased()`; `table.Rows(first, last)` a slot range; `table.Select<&T::member>()` a view of one column.
- `table.Where(pred)` is lazy and returns a chainable `Query`: `Materialize()`, `Count()`, `First()` (an `std::optional<Row<T>>`), `Any()`, `Where()`, `Select<&T::m>()`. `pred` is a lambda over `Row<T>` or a column expression built from tags, `inline constexpr DB::Column<&T::iban> iban{};` and then `iban == x && amount > 100'00`, with `== != < <= > >= && || !`. Operands are copied into the expression, so a temporary `std::string` is fine.
- **Encode the key once.** In a lambda, compare against a `const DB::Key key(text);` built outside the loop rather than a `std::string_view` (correct, but the encoding is redone per row); a column expression does it for you. The predicate is a template parameter and inlines into the scan: no `std::function`, no per-row call. Measured on 100 000 rows (Xeon w7-3465X, warm cache): `iban == x` at 2 to 5 ns per row, a time range covering 10 % of the rows about 1 ms materialised, every row materialised 10 to 13 ms.
- **The ordered column.** `Range<&T::key>(from, to)` (inclusive), `From`, `After`, `Until`, `Before`, `LowerBound` and `UpperBound` are binary searches and compile only for the member `orderedBy` names (a `static_assert` says so for any other). `table.Range<…>(a, b).Where(iban == x)` filters inside the slice, and `table.Where(timePoint >= a && timePoint <= b && iban == x)` does the same: bounds on the ordered column in a top-level `&&` chain become the search. Under `||` or `!` nothing is extracted and the expression runs as a full scan, still correct. Tombstoned rows keep their key, so searches stay valid until `Compact()`. `std::ranges::lower_bound(table.Slots(), key, {}, [](DB::Row<T> r) { return r[&T::m]; })` is the escape hatch for any other column.
- A `Query` over a plain range (`Rows`, `Range`, `From` and the like) can be iterated as a `const` object; one built with `Where` cannot (`std::ranges::filter_view`), so hold it in a non-const local. `Select()` on a `Query` is rvalue-only: call it on the temporary, or `std::move` the query.
- Threads: writes come from one thread at a time. While it runs `Append`, `Emplace` or `Erase`, other threads may read (a scan that started before an `Erase` may still yield the row); `Update`, `Set`, `Compact` and `Clear` need exclusive access from you. Views stay valid across `Append` and `Erase`.

### Schema changes and migrations

The file carries the struct's `version` and a fingerprint of the layout: column names, kinds, offsets, sizes, flags, chrono period and clock, blob element type and the ordered column. Any change to those without a `version` bump is refused at open with a `SchemaError` listing both column sets. That includes renaming a member, changing a type or an enum's underlying type, reordering members of the same alignment class, adding or changing `orderedBy`, and changing a `Formatter`'s `Stored`.

- Bump `version`, keep the old struct under another name (`TransactionV1`), and point `using Previous = TransactionV1;` at it. Added members need nothing more: old rows are copied by member name and new members value-initialised. A removed, renamed or retyped member needs `static T Migrate(Previous&&)`; without it the build fails naming the member. Chains are walked (`V1 → V2 → V3`) and the versions along `Previous` must strictly decrease. Keep every old struct for as long as files of that version may exist.
- The migration runs in the constructor on the first open with the newer code: every live row is streamed through the chain into a new file that replaces the old one atomically, and the old one is kept as `<file>.v<old>.bak` (`backupBeforeMigrate`). Rows that do not satisfy a new `orderedBy` are stable-sorted by it. A large table makes that first open take a while, once.
- Refusals: a file newer than the code, or one the chain does not reach, is a `SchemaError`; an older file opened `readOnly` or with `migrate = false` is a `MigrationError`. `DB::PeekVersion(path)` reads the version from the header alone, without the lock; `DB::Migrate<T>(path)` migrates without keeping the table open; `DB::TableFile::Inspect(path)` returns header, commit and schema (an `Inspection`) without the struct and without the lock; and `DB::DynamicRow` reads columns by name from any table, for dumps and tools.
- The file format itself has a `formatVersion` (1) that must match exactly; a bump there is a library release note, not something a consumer handles.

### Errors

Everything the library throws derives from `Crafter::DB::Error`, a `std::runtime_error`, except where noted. Most messages start with the file path; the 4 GiB `Error`, the heap-handle `CorruptError` and the address-space and `msync` `IOError`s carry none.

| Call | On failure |
|---|---|
| `Table` constructor, `Migrate<T>` | `IOError` (with `code` and `path`: a missing file under `OpenExisting`, an existing one under `CreateNew`, permissions, page-size mismatch, address space), `CorruptError` (not a table, unsupported format, both commit records damaged, extents or schema out of bounds), `SchemaError`, `MigrationError`, `LockedError` |
| `PeekVersion`, `TableFile::Inspect` | `IOError` (missing file, permissions), `CorruptError`; neither takes the lock |
| `Append`, `Emplace`, `Update`, `Set`, `Erase`, `EraseIf`, `Compact`, `Clear` | `ReadOnlyError`; `Error` on a closed table or one inherited through `fork`; `OrderError` before anything is written; `IOError` when the disk, the extent table or the reservation is full; `Error` for a value of 4 GiB or more; `std::length_error` for a `FixedString` that does not fit; `CorruptError` from `Compact` on a damaged heap handle |
| `Flush`, `Close` | `IOError` from `fdatasync` or `msync`; `Flush` throws `Error` in a forked child, where `Close` skips the flush; `Flush` is a no-op on a read-only table, `Close` still closes it and releases the lock |
| `Update`, `Set`, `Erase`, `At` with an id outside the table, or (`Update`, `Set`, `At`) an erased one; `Front`, `Back` on an empty table | `std::out_of_range` (`operator[]` does not check) |
| `StringRef::View`, `BlobRef::View`, `Materialize`, any read of a long value | `CorruptError` when the handle points outside the committed heap, which is a damaged file |
| `Verify()` | never throws for content; returns a `VerifyResult` listing the problems, at most a thousand |
| API misuse: a non-aggregate or `version`-less struct, an unstorable member, `orderedBy` on a float or string, `Range` on another member, a wrong `Emplace` argument, a `Previous` member with no counterpart | a compile error whose `static_assert` message starts with `Crafter.DB:` |

### Testing a consumer

- Work under `fs::temp_directory_path() / "<project>-<test>"`, `remove_all` first and last. crafter-build runs tests in parallel, so each test gets a directory of its own.
- Destroy or `Close()` a writable `Table` before opening the same file again, and a read-only one before opening it writable: the lock is held until then, and the conflicting open throws `LockedError` in the same process.
- A crash test is a `fork()`ed child that writes and ends with `_exit(0)` while its table is still in scope, so no destructor flushes; the parent then opens the file and sees exactly the flushed rows. `tests/ShouldRecoverCommits/main.cpp` in the Crafter.DB repository has the helper.
- End a test that exercised writes with `table.Verify().Ok()`, and pin `DB::Layout<T>::rowSize` and `fingerprint` with `static_assert`s so an accidental schema change fails the build rather than a user's open.
- `std::format("{}", row)` and `DB::TableFile::Inspect(path)` are the debugging tools; `table.File().Dynamic(id).Describe()` prints any row of any table by column name.

### Names to get right

`Materialize` (with a z), `Emplace`, `Erase` and `EraseIf` (not Delete or Remove), `Compact`, `Verify`, `Stats`, `Flush`, `Close`; `Formatter` is the customisation point even though it encodes bytes, not text; the statics are `version` and `orderedBy` (camelCase) and `Previous` and `Migrate` (PascalCase); `RowId` is a `std::uint64_t`; `Key` is the pre-encoded search key, `StringRef` and `BlobRef` the column views, `Column<&T::m>` the tag type.