Crafter.DB/AGENTS.consumer.md

Crafter.DB

The Catcrafts embedded table store: a C++26 module, import Crafter.DB;, in which a table is a plain struct with a version, a file is a memory-mapped array of fixed-width rows with an out-of-line heap for strings and blobs, reads are zero-copy views, queries are lambdas or column expressions that compose with std::ranges, and schema changes are C++ migrations run on open. Everything is in namespace Crafter::DB (the examples below assume using namespace Crafter;). Linux only: the engine uses flock, fallocate and fixed-address mmap. No dependencies, no link flags. Source and issues: https://forgejo.catcrafts.net/Catcrafts/Crafter.DB (LGPL-3.0-only). The README there is the manual, with the whole API and the file format; these notes are its AGENTS.consumer.md: a fix to them goes there, not into the copy crafter-build writes.

Building against it

• GitProject({ .source = { .url = "https://forgejo.catcrafts.net/Catcrafts/Crafter.DB.git" } }) (add .commit to pin), or LocalProject({ .projectFile = "../Crafter.DB/project.cpp" }) for a sibling checkout, into cfg.dependencies. It is a static library whose partitions are nearly all templates: everything a Table<T> does is instantiated in your translation units with your target and --march, and the only out-of-line code is the OS layer and the type-erased engine. It reads no project args and has no dependencies of its own, so a flag of yours that reaches it only changes its variant hash (a separate bin/ directory per flag combination); strip your own flags from the .args you pass down if you would rather not, and pass the same list on every path that reaches it, or the build stops with two loads that disagree.
• Only Linux targets build it (x86_64-unknown-linux-gnu and aarch64-alpine-linux-musl have been built; the tests run on *-linux-gnu). A project that also targets wasm32-* or x86_64-w64-mingw32 adds it to cfg.dependencies only when cfg.target contains -linux-, and keeps the code that uses it behind the same condition. The compiler is crafter-build's clang 23 with import std;; the library leans on C++26 features (structured-binding packs, pack indexing, generated static_assert messages) that no other toolchain in use here has.
• Nothing is defined for you to #ifdef on. Compile-time facts about a table come from DB::Layout<T>: rowSize, rowAlign, columnCount, fingerprint, and columns[i].kind, .offset and .name.View() (the name is a fixed-capacity text, not a std::string_view).

Defining a table

A table struct is an aggregate (public members, no base classes with members, no user-declared constructors), default constructible, with static constexpr std::uint32_t version = 1;. Optional statics: static constexpr auto orderedBy = &T::member; (an integer, enum, or integral-rep duration or time_point member; enforced on every write, enables binary search), using Previous = OlderT; and static T Migrate(OlderT&&) (see "Schema changes and migrations"). Every rule about the struct and its members is a static_assert whose message starts with Crafter.DB: and names them, so read the first error, not the instantiation stack.

• Storable members: arithmetic types except long double, bool, std::byte, enums (stored as their underlying integer), std::chrono::duration and time_point (rep plus period, with the clock in the schema), std::string, std::vector<E> for trivially copyable non-pointer E, std::optional<M> of a storable M (not of another optional), std::array<E, N> of scalars (one column per element) or of bytes (one fixed column), DB::FixedString<N> (N bytes inline, NUL padded), nested aggregates of storable members (flattened to outer.inner columns), and anything with a Crafter::DB::Formatter<T> specialisation. Pointers, references and member pointers are rejected.
• A struct with constructors is not an aggregate and is refused with a message that says to replace them with designated initialisers; Table::Emplace builds rows from fields, so the constructors are not missed.
• Columns are not laid out in declaration order: the flags byte sits at offset 0 and the leaves follow in ascending alignment, 2-byte columns before 8-byte ones. The README's seven-member Transaction is 88 bytes. A std::string or std::vector column is a 16-byte handle holding values of up to 12 bytes inline, and the first 4 bytes of longer ones (a vector of 8-byte-aligned elements such as std::uint64_t or double always goes to the heap), so equality against such a column is one 8-byte compare per row unless the lengths and prefixes match. Keys you filter on often are best kept at 12 bytes or under where the data allows (a BIC is, an IBAN is not); a code with a fixed maximum length is a FixedString<N>.
• Your own types: specialise Formatter<T> the way you would std::formatter. A fixed-width one declares using Stored = <storable type>; with static Stored Encode(const T&) and static T Decode(const Stored&); a variable-length one inherits Formatter<std::string> (or Formatter<std::vector<E>>) and overrides static Var Encode(const T&, HeapWriter&) and static T Materialize(StringRef). A row still hands out the raw StringRef or BlobRef for a variable-length member; Materialize() is where your type comes back.
• Once the struct is settled, pin DB::Layout<T>::rowSize and DB::Layout<T>::fingerprint with a static_assert in a test: a failing assert is the compile-time form of the SchemaError your users would get at open, and tells you a version bump is due.

Opening and lifetime

DB::Table<T> table(path); creates the file when missing (directories 0700, file 0600), migrates it when older, and throws when it cannot. DB::Table<T>(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = true }) opens a frozen reader. OpenOptions takes designated initialisers in this order: mode (OpenOrCreate, OpenExisting, CreateNew), readOnly, createDirectories, filePermissions, directoryPermissions, durability (OnFlush, EveryAppend), migrate, backupBeforeMigrate, reserveBytes (32 GiB), excludeFromCoreDumps, prefetch.

• Open once per process and keep the table open. An open maps every extent of the file (one mmap each: 11 for the Benchmark's 100 000 rows appended in one batch, about 20 for the same rows appended one at a time) and a read-only open costs about 30 µs, which dominates any code that opens a table per query. A Table is move-only: hold it in a member or a static, not in each function that reads.
• One writer per file at a time, across processes and within one. A writer holds an exclusive flock, a read-only table a shared one: a second writable Table on the file, in another process or in this one, throws LockedError, a read-only table anywhere makes a writer's open fail, and a writer makes every reader's. Read-only tables coexist. Two processes that create the same table at once end up sharing one file. The lock is advisory: cp and other tools ignore it, and a copy taken while a writer is open reads back as of the writer's last commit record, which is its last Flush.
• Durability is at Flush(), Close() and destruction, not at Append. Under Durability::OnFlush (the default) an appended row is visible to this process at once and reaches the disk at the next Flush; a crash loses what was not flushed and never a committed row. An Update or Set that writes heap bytes (a string longer than 12 bytes, a blob) flushes on its own, an fdatasync each, and so commits every row appended before it. Durability::EveryAppend flushes after every write. The destructor flushes but cannot report; call Close() where a failed final flush must throw. After fork() the child may read the inherited table; every write and every explicit Flush() in it throws Error, while Close() and the destructor skip the flush silently, so the child's stale counters never reach the file.
• Files beside the table: <file>.tmp-<12 hex> while the table is being created and while Compact, Clear or a migration is writing (stale ones go on the next writable open), and <file>.v<old>.bak after a migration (a hard link of the pre-migration file; delete it when it has served). The directory must stay writable for those three operations. Use a directory per table or distinct basenames; .cdb is the convention.
• Address space. Each open table reserves reserveBytes twice (rows and heap), 64 GiB of virtual space by default, none of it committed. Under a ulimit -v, or with hundreds of tables open, lower reserveBytes to a few times the table's size; a reservation that fills during an append throws IOError naming the option.
• A table created on a 4 KiB-page kernel cannot be opened on a 16 KiB-page one (IOError, not_supported); the extent granule is recorded in the header.

Writing

Append(row) and Append(range) return the RowId of the first new row (ids are slot indices from 0, stable until Compact or Clear); a range is one reservation and one publish, all or nothing. Emplace(args...) takes exactly one argument per member in declaration order, converted implicitly (a std::string_view is enough for a string member; an int for an enum member does not compile). Update(id, row) replaces a live row; Set<&T::member>(id, value) one member; Erase(id) tombstones (false when already erased); EraseIf(pred) returns the count; Flush(); Compact() rewrites without tombstones and heap garbage; Clear() empties.

• On an orderedBy table a write whose key would sort before the previous row (or, for Update and Set, after the next) throws OrderError before anything is written; equal keys are allowed; an unsorted batch is refused whole. Rows must arrive in key order: sort before Append, or leave orderedBy out and sort when reading.
• Update, Set and Erase leave heap garbage (the old string bytes); Stats().heapGarbage says how much, and only Compact() reclaims it. Compact and Clear invalidate every RowId, Row and StringRef taken before them; Update and Set invalidate the views of that row; Append and Erase invalidate nothing.
• Update and Set never leave a row pointing at heap bytes that are not on disk (those are committed first), but the row's fixed-width columns are not written atomically: a crash or power loss mid-row leaves the old row, the new one, or a mix of the two. A log of appends is the shape the library is built for.
• Limits: a value below 4 GiB (Error), a row of at most 65535 bytes and 65535 columns (compile time), a region of at most 32 extents, roughly 17 GiB (IOError saying to Compact()), and the disk (fallocate fails with IOError before any page is touched, never a SIGBUS).

Reading and querying

table.Size() is the live rows, SlotCount() the id space including tombstones, table[id] an unchecked Row<T> view (16 bytes, no allocation), At(id) a checked one, Contains(id), Front(), Back() (skips trailing tombstones). row[&T::member] and row.Get<&T::member>() read one column from the mapping: the value for scalars, chrono and FixedString; a StringRef for a string (converts to std::string_view, compares with == to literals, std::string_view, std::string, Key and other refs without building a view, orders with <=>); a BlobRef<E> for a vector (converts to std::span<const E>); an std::optional of the view for an optional. row.Materialize() (also explicit operator T) copies the row out; table.Materialize() every live row. std::format("{}", row) prints Type{member=value, …}.

• for (DB::Row<T> row : table) is the live rows in id order and composes with std::views; table.Slots() is random access over every slot with row.Erased(); table.Rows(first, last) a slot range; table.Select<&T::member>() a view of one column.
• table.Where(pred) is lazy and returns a chainable Query: Materialize(), Count(), First() (an std::optional<Row<T>>), Any(), Where(), Select<&T::m>(). pred is a lambda over Row<T> or a column expression built from tags, inline constexpr DB::Column<&T::iban> iban{}; and then iban == x && amount > 100'00, with == != < <= > >= && || !. Operands are copied into the expression, so a temporary std::string is fine.
• Encode the key once. In a lambda, compare against a const DB::Key key(text); built outside the loop rather than a std::string_view (correct, but the encoding is redone per row); a column expression does it for you. The predicate is a template parameter and inlines into the scan: no std::function, no per-row call. Measured on 100 000 rows (Xeon w7-3465X, warm cache): iban == x at 2 to 5 ns per row, a time range covering 10 % of the rows about 1 ms materialised, every row materialised 10 to 13 ms.
• The ordered column. Range<&T::key>(from, to) (inclusive), From, After, Until, Before, LowerBound and UpperBound are binary searches and compile only for the member orderedBy names (a static_assert says so for any other). table.Range<…>(a, b).Where(iban == x) filters inside the slice, and table.Where(timePoint >= a && timePoint <= b && iban == x) does the same: bounds on the ordered column in a top-level && chain become the search. Under || or ! nothing is extracted and the expression runs as a full scan, still correct. Tombstoned rows keep their key, so searches stay valid until Compact(). std::ranges::lower_bound(table.Slots(), key, {}, [](DB::Row<T> r) { return r[&T::m]; }) is the escape hatch for any other column.
• A Query over a plain range (Rows, Range, From and the like) can be iterated as a const object; one built with Where cannot (std::ranges::filter_view), so hold it in a non-const local. Select() on a Query is rvalue-only: call it on the temporary, or std::move the query.
• Threads: writes come from one thread at a time. While it runs Append, Emplace or Erase, other threads may read (a scan that started before an Erase may still yield the row); Update, Set, Compact and Clear need exclusive access from you. Views stay valid across Append and Erase.

Schema changes and migrations

The file carries the struct's version and a fingerprint of the layout: column names, kinds, offsets, sizes, flags, chrono period and clock, blob element type and the ordered column. Any change to those without a version bump is refused at open with a SchemaError listing both column sets. That includes renaming a member, changing a type or an enum's underlying type, reordering members of the same alignment class, adding or changing orderedBy, and changing a Formatter's Stored.

• Bump version, keep the old struct under another name (TransactionV1), and point using Previous = TransactionV1; at it. Added members need nothing more: old rows are copied by member name and new members value-initialised. A removed, renamed or retyped member needs static T Migrate(Previous&&); without it the build fails naming the member. Chains are walked (V1 → V2 → V3) and the versions along Previous must strictly decrease. Keep every old struct for as long as files of that version may exist.
• The migration runs in the constructor on the first open with the newer code: every live row is streamed through the chain into a new file that replaces the old one atomically, and the old one is kept as <file>.v<old>.bak (backupBeforeMigrate). Rows that do not satisfy a new orderedBy are stable-sorted by it. A large table makes that first open take a while, once.
• Refusals: a file newer than the code, or one the chain does not reach, is a SchemaError; an older file opened readOnly or with migrate = false is a MigrationError. DB::PeekVersion(path) reads the version from the header alone, without the lock; DB::Migrate<T>(path) migrates without keeping the table open; DB::TableFile::Inspect(path) returns header, commit and schema (an Inspection) without the struct and without the lock; and DB::DynamicRow reads columns by name from any table, for dumps and tools.
• The file format itself has a formatVersion (1) that must match exactly; a bump there is a library release note, not something a consumer handles.

Errors

Everything the library throws derives from Crafter::DB::Error, a std::runtime_error, except where noted. Most messages start with the file path; the 4 GiB Error, the heap-handle CorruptError and the address-space and msync IOErrors carry none.

Call	On failure
Table constructor, Migrate<T>	IOError (with code and path: a missing file under OpenExisting, an existing one under CreateNew, permissions, page-size mismatch, address space), CorruptError (not a table, unsupported format, both commit records damaged, extents or schema out of bounds), SchemaError, MigrationError, LockedError
PeekVersion, TableFile::Inspect	IOError (missing file, permissions), CorruptError; neither takes the lock
Append, Emplace, Update, Set, Erase, EraseIf, Compact, Clear	ReadOnlyError; Error on a closed table or one inherited through fork; OrderError before anything is written; IOError when the disk, the extent table or the reservation is full; Error for a value of 4 GiB or more; std::length_error for a FixedString that does not fit; CorruptError from Compact on a damaged heap handle
Flush, Close	IOError from fdatasync or msync; Flush throws Error in a forked child, where Close skips the flush; Flush is a no-op on a read-only table, Close still closes it and releases the lock
Update, Set, Erase, At with an id outside the table, or (Update, Set, At) an erased one; Front, Back on an empty table	std::out_of_range (operator[] does not check)
StringRef::View, BlobRef::View, Materialize, any read of a long value	CorruptError when the handle points outside the committed heap, which is a damaged file
Verify()	never throws for content; returns a VerifyResult listing the problems, at most a thousand
API misuse: a non-aggregate or version-less struct, an unstorable member, orderedBy on a float or string, Range on another member, a wrong Emplace argument, a Previous member with no counterpart	a compile error whose static_assert message starts with Crafter.DB:

Testing a consumer

• Work under fs::temp_directory_path() / "<project>-<test>", remove_all first and last. crafter-build runs tests in parallel, so each test gets a directory of its own.
• Destroy or Close() a writable Table before opening the same file again, and a read-only one before opening it writable: the lock is held until then, and the conflicting open throws LockedError in the same process.
• A crash test is a fork()ed child that writes and ends with _exit(0) while its table is still in scope, so no destructor flushes; the parent then opens the file and sees exactly the flushed rows. tests/ShouldRecoverCommits/main.cpp in the Crafter.DB repository has the helper.
• End a test that exercised writes with table.Verify().Ok(), and pin DB::Layout<T>::rowSize and fingerprint with static_asserts so an accidental schema change fails the build rather than a user's open.
• std::format("{}", row) and DB::TableFile::Inspect(path) are the debugging tools; table.File().Dynamic(id).Describe() prints any row of any table by column name.

Names to get right

Materialize (with a z), Emplace, Erase and EraseIf (not Delete or Remove), Compact, Verify, Stats, Flush, Close; Formatter is the customisation point even though it encodes bytes, not text; the statics are version and orderedBy (camelCase) and Previous and Migrate (PascalCase); RowId is a std::uint64_t; Key is the pre-encoded search key, StringRef and BlobRef the column views, Column<&T::m> the tag type.