Crafter.DB

An embedded, typed table store for C++26: a table is a plain struct, a file is a memory-mapped array of fixed-width rows, queries are C++. Built for the Catcrafts projects (it replaces Nostro's hand-rolled transaction log) and distributed as one C++ module, Crafter.DB.

• Structs are the schema. Any aggregate with a version is a table. No macros, no registration, no code generation: member names, types and offsets are derived at compile time.
• Fixed-width rows, out-of-line strings. Every row is rowSize bytes. Strings and blobs live in a heap region; the row holds a 16-byte handle that keeps values of up to 12 bytes, and the first 4 bytes of longer ones, inline, so iban == x is one 8-byte compare per row.
• Zero-copy reads. Row<T> is a 16-byte view; row[&T::iban] reads one column straight from the mapping. Materialize() copies only when you ask.
• Queries are C++. Lambdas over rows, or column expressions (iban == x && amount > 100), composed with std::ranges. A declared orderedBy column gives binary search (Range, From, After), also from a combined expression.
• Versioned, with C++ migrations. The file carries the struct's version and a layout fingerprint. A struct that changed without a version bump is refused; an older file is migrated through the Previous chain on open.
• Crash safe appends. Two CRC-protected commit records; a crash loses at most what was not yet flushed, never committed rows.
• Private by default. Files are created 0600 in 0700 directories and excluded from core dumps.

Modules

import Crafter.DB; re-exports these partitions, all in namespace Crafter::DB:

Partition	Contents
:Table	Table<T>: open/create/migrate, Append, Emplace, Update, Set, Erase, EraseIf, Compact, Clear, Flush, iteration, Where, Select, Range/From/After/Until/Before, LowerBound/UpperBound, Verify.
:Row	Row<T> (column views, Materialize), RowId, Column<&T::m> tags, LiveRange, SlotView, std::formatter<Row<T>>.
:Query	Column expressions (==, !=, <, <=, >, >=, &&, ||, !), bound extraction, the chainable Query result (Materialize, Count, First, Any, Where, Select).
:Formatter	Formatter<T>, the customisation point for member types; StringRef, BlobRef<E>, Key, FixedString<N>; built-in formatters for std::string, std::vector<Trivial>, std::chrono.
:Schema	Layout<T>: the compile-time row layout, fingerprint and diagnostics; TableType, HasOrderedBy, HasPrevious, HasMigrate.
:Migration	Migrate<T>(path), PeekVersion(path), the chain walk and the by-name copy.
:Storage	The type-erased engine: TableFile, Inspect(path), DynamicRow, VerifyResult, TableStats.
:File	OpenOptions, OpenMode, Durability; file descriptors, locks, reserved mappings.
:Format	The on-disk structures (FileHeader, CommitRecord, ColumnDescriptor, Var), Kind, FNV-1a, CRC-32C, SchemaInfo.
:Codec	Typed encode/decode over a Layout<T>.
:Reflect	Compile-time member count, names, types and pointer-to-member lookup for aggregates.
:Error	Error and its subclasses.

Defining a table

import Crafter.DB;
using namespace Crafter;

enum class Currency : std::uint16_t { Euro };

struct Transaction {
    std::string name;
    std::string bic;
    std::string iban;
    std::string description;
    Currency currency;                                   // enums, integers, floats: stored as is
    std::int64_t amount;
    std::chrono::system_clock::time_point timePoint;     // built-in formatter: rep + period

    static constexpr std::uint32_t version = 1;                  // required
    static constexpr auto orderedBy = &Transaction::timePoint;   // optional: enforced on writes, enables Range()
};

A table struct is an aggregate (public members, no base classes with members, no user-declared constructors) that is default constructible and declares version. Every member must be storable: arithmetic types, bool, std::byte, enums, std::chrono::duration and time_point, std::string, std::vector<E> for trivially copyable E, std::optional<M>, std::array<E, N>, FixedString<N>, nested aggregates of storable members (flattened into outer.inner columns), and anything with a Formatter. Pointers and references are rejected at compile time; each static_assert names the table and the member.

The row above is 88 bytes: a flags byte, currency, four 16-byte string handles, amount, timePoint.

Storing your own types

Specialise Crafter::DB::Formatter<T> the way you would std::formatter. A fixed-width formatter maps to a storable Stored type:

template<>
struct Crafter::DB::Formatter<Percent> {
    using Stored = std::int32_t;                                   // basis points
    static Stored Encode(const Percent& value) noexcept { return value.BasisPoints(); }
    static Percent Decode(const Stored& stored) noexcept { return Percent(stored / 100.0); }
};

A variable-length formatter writes into the heap. Inheriting the std::string one and overriding the ends is the common shape:

template<>
struct Crafter::DB::Formatter<Tags> : Crafter::DB::Formatter<std::string> {
    static Var Encode(const Tags& value, HeapWriter& heap) { return Formatter<std::string>::Encode(value.Joined(), heap); }
    static Tags Materialize(StringRef view) { return Tags::Split(view.View()); }   // row[&T::tags] still hands out the raw StringRef
};

Opening a table

DB::Table<Transaction> table(cachePath / "transactions" / "bunq" / "main.cdb");          // created if missing, migrated if older
DB::Table<Transaction> frozen(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = true });

OpenOptions (designated initialisers): mode (OpenOrCreate, OpenExisting, CreateNew), readOnly, createDirectories, filePermissions (0600), directoryPermissions (0700), durability (OnFlush: durable after Flush() and on destruction; EveryAppend), migrate, backupBeforeMigrate (keeps <file>.v<old>.bak), reserveBytes (virtual address space per region, 32 GiB), excludeFromCoreDumps, prefetch.

One Table per file at a time: a writer takes an exclusive flock, read-only tables take shared locks. A conflicting open throws LockedError, in the same process as well as from another. Two processes creating the same table at once end up sharing one file (the create never replaces an existing file).

The destructor flushes but cannot report a failure; call Close() when a failed final flush must surface as an exception. After fork(), the child may read an inherited table (the mapping is shared) but every write, Flush and the destructor's flush are refused in the child, so its stale counters never reach the file.

Writing

DB::RowId id = table.Append(tr);                      // ids are slot indices
table.Append(std::span<const Transaction>(rows));     // one reservation, one publish
table.Emplace("Albert Heijn", "ABNANL2A", iban, "groceries", Currency::Euro, -2345, now);   // one argument per member, strings need no std::string
table.Update(id, tr);
table.Set<&Transaction::description>(id, "corrected");
table.Erase(id);                                       // tombstone; ids stay valid
table.EraseIf([](DB::Row<Transaction> r) { return r[&Transaction::amount] == 0; });
table.Flush();                                         // fdatasync + commit record
table.Compact();                                       // rewrite without tombstones and heap garbage; invalidates ids and views

On an orderedBy table, a write whose key would break the order throws OrderError before anything is written; a batch is all-or-nothing. Update and Set commit any new heap bytes to disk before the row starts pointing at them, so a crash leaves the old row or the new one, never a dangling reference; the row bytes themselves are not atomic against a power loss mid-write.

Reading and querying

table.Size();                                     // live rows
DB::Row<Transaction> last = table.Back();         // view, no allocation
auto when = last[&Transaction::timePoint];        // value
DB::StringRef iban = last[&Transaction::iban];    // view into the file; converts to std::string_view
Transaction copy = last.Materialize();

for (DB::Row<Transaction> row : table) ...        // live rows in id order
std::vector<Transaction> all = table.Materialize();

// predicates: a lambda, or column tags
const DB::Key key(ibanText);                      // encode the search key once, compare with one 8-byte load per row
auto rows = table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == key; }).Materialize();

inline constexpr DB::Column<&Transaction::iban> iban{};
inline constexpr DB::Column<&Transaction::amount> amount{};
inline constexpr DB::Column<&Transaction::timePoint> timePoint{};
auto big = table.Where(iban == ibanText && amount > 100'00).Count();

// composes with std::ranges
for (auto row : table | std::views::filter(...) | std::views::take(10)) ...
std::int64_t total = std::ranges::fold_left(table.Select<&Transaction::amount>(), 0LL, std::plus{});

StringRef compares against std::string_view, std::string, literals, Key and other refs without building a std::string_view first: the handle's length-and-prefix word rejects almost every non-match, the heap is read only for prefix-equal values longer than 12 bytes.

Predicates are template parameters: the lambda body inlines into the scan loop. There is no std::function and no per-row call.

Ordered columns

auto inRange = table.Range<&Transaction::timePoint>(from, to);     // from <= key <= to, binary search
auto since   = table.From<&Transaction::timePoint>(from);          // key >= from
auto newer   = table.After<&Transaction::timePoint>(last);         // key >  last
DB::RowId lb = table.LowerBound<&Transaction::timePoint>(from);

// range + property: Range narrows by binary search, Where filters inside it
auto rent = table.Range<&Transaction::timePoint>(from, to).Where(iban == x).Materialize();
// the same from one expression: bounds on the ordered column in a top-level && become the binary search
auto same = table.Where(timePoint >= from && timePoint <= to && iban == x).Materialize();

Keys may repeat; Range is inclusive. orderedBy must be an integer, enum or integral chrono member: floats have NaN, and strings have no binary search here. Tombstoned slots keep their key, so the search stays valid until Compact(). std::ranges::lower_bound(table.Slots(), …) is the escape hatch for anything else.

Migrations

struct TransactionV1 { /* as above */ static constexpr std::uint32_t version = 1; };

struct TransactionV2 {              // added `reference`: additive, no Migrate needed
    /* V1 members + */ std::string reference;
    static constexpr std::uint32_t version = 2;
    using Previous = TransactionV1;
};

struct Transaction {                // renamed description -> memo, merged amount + currency: needs Migrate
    std::string name, bic, iban, memo, reference;
    Money money;
    std::chrono::system_clock::time_point timePoint;

    static constexpr std::uint32_t version = 3;
    using Previous = TransactionV2;
    static Transaction Migrate(TransactionV2&& old) {
        return { .name = std::move(old.name), .bic = std::move(old.bic), .iban = std::move(old.iban), .memo = std::move(old.description),
                 .reference = std::move(old.reference), .money = { old.amount, old.currency }, .timePoint = old.timePoint };
    }
};

Opening a version-1 file with Table<Transaction> streams every live row through V1 → V2 (copied by member name, new members value-initialised) → Migrate → V3 into a new file that replaces the old one atomically; the original stays as main.cdb.v1.bak. When the new struct declares an orderedBy the old rows do not satisfy, the migration sorts them (stably) by the new key. The fingerprint covers column names, kinds, offsets, chrono periods and clocks, blob element types and the ordered column, so any of those changing without a version bump is refused. Without Migrate, every member of Previous must exist in the new struct with the same name and type (a removal or rename fails to compile with a message saying so). A file whose version equals the struct's but whose fingerprint differs throws SchemaError listing both column sets. migrate = false or readOnly on an older file throws MigrationError. DB::Migrate<T>(path) runs a migration without keeping the table open; DB::PeekVersion(path) reads the version alone.

Thread safety and lifetimes

Like a standard container: any number of concurrent readers, writes need exclusive access from the caller. Exception: Append/Emplace/Erase are safe while other threads read (rows are published with a release store, tombstones with an atomic byte and a monotonic flag that switches scans to the checking path). A scan started before an Erase may still yield that row. Row views and StringRefs stay valid across Append and Erase and are invalidated by Update/Set of their row, Compact, Clear, migration and destruction. Extents are mapped into a reserved virtual range, so the mapping never moves while the table is open. A Query over a plain range (Rows, Range, From, …) can be iterated as a const object; one built with Where cannot, because std::ranges::filter_view has no const iteration.

Errors

All exceptions derive from Crafter::DB::Error (a std::runtime_error): IOError (with code and path), CorruptError, SchemaError, MigrationError, LockedError, OrderError, ReadOnlyError. Out-of-range or erased ids throw std::out_of_range from At(); operator[] does not check.

File format

One file per table: a 4 KiB header (magic, format and schema version, fingerprint, extent table, two CRC-32C commit records), schema pages (column descriptors and names, so Inspect() and DynamicRow can read any table without its struct), then extents. Rows and heap are two regions, each a chain of doubling extents appended at the end of the file and mapped back to back. Extents are aligned to the page size of the system that created the file (4 KiB on x86-64); a file created with smaller pages than the running kernel's cannot be mapped and is refused with a clear error. Filesystems without fallocate fall back to posix_fallocate. Appends publish rows to in-process readers immediately and become durable at Flush(): fdatasync, then the inactive commit record, then msync. On open the newest valid commit wins; rows and heap bytes beyond it, and uncommitted extents, are discarded. Deletes set a flag byte; Compact() rewrites. Update is safe against a process crash but not atomic against a power loss mid-row (documented limitation). Little-endian only. Verify() checks every handle, the ordered column and the live count.

Performance

tests/Benchmark compares Crafter.DB with a cleaned-up copy of Nostro's format (iostream, 2-byte length-prefixed strings) on 100 000 transactions, warm cache, one run on a Xeon w7-3465X:

Operation	Nostro	Crafter.DB	Note
append 100 000 rows	16.5 ms	11.1 ms	Crafter.DB includes fdatasync; Nostro syncs nothing
last timestamp, 1000 × open + read	1.9 ms	30.0 ms	opening a mapped table costs one mmap per extent (about 20 for this size); open once per process
load all, materialised	36.0 ms	10.4 ms	zero-copy scan of two columns: 0.8 ms
time range, 10 % of rows	10.0 ms	0.9 ms	binary search + contiguous slice
iban == x	16.2 ms	0.23–0.30 ms	2–3 ns per row
time range + iban == x	11.2 ms	0.03 ms

File size for that table: 17.2 MB against Nostro's 9.3 MB (fixed-width handles plus the slack of the last extents; Compact() right-sizes).

Building

This library uses Crafter.Build, which ships clang 23 and libc++ 23:

crafter-build            # static library
crafter-build test       # build and run the tests

Linux only: the engine uses flock, fallocate and fixed-address mmap.

Testing

Each test is tests/<Name>/main.cpp and is declared in project.cpp:

• ShouldRoundTrip, ShouldKeepLayoutGolden, ShouldKeepViewsStableAcrossGrowth, ShouldCompareStringRefs
• ShouldQuery, ShouldEraseAndCompact, ShouldFormatCustomTypes
• ShouldMigrate, ShouldRejectSchemaChange, ShouldRecoverCommits, ShouldLockAcrossProcesses, ShouldVerify, ShouldReadWhileAppending
• Benchmark (always passes, prints the table above)

License

This library is licensed under the GNU Lesser General Public License, version 3 only (LGPL-3.0-only). See LICENSE for the license text. The LGPL supplements the GNU General Public License version 3, a copy of which is included as LICENSE.GPL.

Copyright

Copyright (C) 2026 Catcrafts®

catcrafts.net