Crafter.DB/README.md

# Crafter.DB

An embedded, typed table store for C++26: a table is a plain struct, a file is a memory-mapped array of fixed-width rows, queries are C++. Built for the Catcrafts projects (it replaces Nostro's hand-rolled transaction log) and distributed as one C++ module, `Crafter.DB`.

- **Structs are the schema.** Any aggregate with a `version` is a table. No macros, no registration, no code generation: member names, types and offsets are derived at compile time.
- **Fixed-width rows, out-of-line strings.** Every row is `rowSize` bytes. Strings and blobs live in a heap region; the row holds a 16-byte handle that keeps values of up to 12 bytes, and the first 4 bytes of longer ones, inline, so `iban == x` is one 8-byte compare per row.
- **Zero-copy reads.** `Row<T>` is a 16-byte view; `row[&T::iban]` reads one column straight from the mapping. `Materialize()` copies only when you ask.
- **Queries are C++.** Lambdas over rows, or column expressions (`iban == x && amount > 100`), composed with `std::ranges`. A declared `orderedBy` column gives binary search (`Range`, `From`, `After`), also from a combined expression.
- **Versioned, with C++ migrations.** The file carries the struct's `version` and a layout fingerprint. A struct that changed without a version bump is refused; an older file is migrated through the `Previous` chain on open.
- **Crash safe appends.** Two CRC-protected commit records; a crash loses at most what was not yet flushed, never committed rows.
- **Private by default.** Files are created `0600` in `0700` directories and excluded from core dumps.

## Modules

`import Crafter.DB;` re-exports these partitions, all in `namespace Crafter::DB`:

| Partition | Contents |
| --- | --- |
| `:Table` | `Table<T>`: open/create/migrate, `Append`, `Emplace`, `Update`, `Set`, `Erase`, `EraseIf`, `Compact`, `Clear`, `Flush`, iteration, `Where`, `Select`, `Range`/`From`/`After`/`Until`/`Before`, `LowerBound`/`UpperBound`, `Verify`. |
| `:Row` | `Row<T>` (column views, `Materialize`), `RowId`, `Column<&T::m>` tags, `LiveRange`, `SlotView`, `std::formatter<Row<T>>`. |
| `:Query` | Column expressions (`==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `\|\|`, `!`), bound extraction, the chainable `Query` result (`Materialize`, `Count`, `First`, `Any`, `Where`, `Select`). |
| `:Formatter` | `Formatter<T>`, the customisation point for member types; `StringRef`, `BlobRef<E>`, `Key`, `FixedString<N>`; built-in formatters for `std::string`, `std::vector<Trivial>`, `std::chrono`. |
| `:Schema` | `Layout<T>`: the compile-time row layout, fingerprint and diagnostics; `TableType`, `HasOrderedBy`, `HasPrevious`, `HasMigrate`. |
| `:Migration` | `Migrate<T>(path)`, `PeekVersion(path)`, the chain walk and the by-name copy. |
| `:Storage` | The type-erased engine: `TableFile`, `Inspect(path)`, `DynamicRow`, `VerifyResult`, `TableStats`. |
| `:File` | `OpenOptions`, `OpenMode`, `Durability`; file descriptors, locks, reserved mappings. |
| `:Format` | The on-disk structures (`FileHeader`, `CommitRecord`, `ColumnDescriptor`, `Var`), `Kind`, FNV-1a, CRC-32C, `SchemaInfo`. |
| `:Codec` | Typed encode/decode over a `Layout<T>`. |
| `:Reflect` | Compile-time member count, names, types and pointer-to-member lookup for aggregates. |
| `:Error` | `Error` and its subclasses. |

## Defining a table

```cpp
import Crafter.DB;
using namespace Crafter;

enum class Currency : std::uint16_t { Euro };

struct Transaction {
    std::string name;
    std::string bic;
    std::string iban;
    std::string description;
    Currency currency;                                   // enums, integers, floats: stored as is
    std::int64_t amount;
    std::chrono::system_clock::time_point timePoint;     // built-in formatter: rep + period

    static constexpr std::uint32_t version = 1;                  // required
    static constexpr auto orderedBy = &Transaction::timePoint;   // optional: enforced on writes, enables Range()
};
```

A table struct is an aggregate (public members, no base classes with members, no user-declared constructors) that is default constructible and declares `version`. Every member must be storable: arithmetic types, `bool`, `std::byte`, enums, `std::chrono::duration` and `time_point`, `std::string`, `std::vector<E>` for trivially copyable `E`, `std::optional<M>`, `std::array<E, N>`, `FixedString<N>`, nested aggregates of storable members (flattened into `outer.inner` columns), and anything with a `Formatter`. Pointers and references are rejected at compile time; each `static_assert` names the table and the member.

The row above is 88 bytes: a flags byte, `currency`, four 16-byte string handles, `amount`, `timePoint`.

## Storing your own types

Specialise `Crafter::DB::Formatter<T>` the way you would `std::formatter`. A fixed-width formatter maps to a storable `Stored` type:

```cpp
template<>
struct Crafter::DB::Formatter<Percent> {
    using Stored = std::int32_t;                                   // basis points
    static Stored Encode(const Percent& value) noexcept { return value.BasisPoints(); }
    static Percent Decode(const Stored& stored) noexcept { return Percent(stored / 100.0); }
};
```

A variable-length formatter writes into the heap. Inheriting the `std::string` one and overriding the ends is the common shape:

```cpp
template<>
struct Crafter::DB::Formatter<Tags> : Crafter::DB::Formatter<std::string> {
    static Var Encode(const Tags& value, HeapWriter& heap) { return Formatter<std::string>::Encode(value.Joined(), heap); }
    static Tags Materialize(StringRef view) { return Tags::Split(view.View()); }   // row[&T::tags] still hands out the raw StringRef
};
```

## Opening a table

```cpp
DB::Table<Transaction> table(cachePath / "transactions" / "bunq" / "main.cdb");          // created if missing, migrated if older
DB::Table<Transaction> frozen(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = true });
```

`OpenOptions` (designated initialisers): `mode` (`OpenOrCreate`, `OpenExisting`, `CreateNew`), `readOnly`, `createDirectories`, `filePermissions` (0600), `directoryPermissions` (0700), `durability` (`OnFlush`: durable after `Flush()` and on destruction; `EveryAppend`), `migrate`, `backupBeforeMigrate` (keeps `<file>.v<old>.bak`), `reserveBytes` (virtual address space per region, 32 GiB), `excludeFromCoreDumps`, `prefetch`.

One `Table` per file at a time: a writer takes an exclusive `flock`, read-only tables take shared locks. A conflicting open throws `LockedError`, in the same process as well as from another. Two processes creating the same table at once end up sharing one file (the create never replaces an existing file).

The destructor flushes but cannot report a failure; call `Close()` when a failed final flush must surface as an exception. After `fork()`, the child may read an inherited table (the mapping is shared) but every write, `Flush` and the destructor's flush are refused in the child, so its stale counters never reach the file.

## Writing

```cpp
DB::RowId id = table.Append(tr);                      // ids are slot indices
table.Append(std::span<const Transaction>(rows));     // one reservation, one publish
table.Emplace("Albert Heijn", "ABNANL2A", iban, "groceries", Currency::Euro, -2345, now);   // one argument per member, strings need no std::string
table.Update(id, tr);
table.Set<&Transaction::description>(id, "corrected");
table.Erase(id);                                       // tombstone; ids stay valid
table.EraseIf([](DB::Row<Transaction> r) { return r[&Transaction::amount] == 0; });
table.Flush();                                         // fdatasync + commit record
table.Compact();                                       // rewrite without tombstones and heap garbage; invalidates ids and views
```

On an `orderedBy` table, a write whose key would break the order throws `OrderError` before anything is written; a batch is all-or-nothing. `Update` and `Set` commit any new heap bytes to disk before the row starts pointing at them, so a crash leaves the old row or the new one, never a dangling reference; the row bytes themselves are not atomic against a power loss mid-write.

## Reading and querying

```cpp
table.Size();                                     // live rows
DB::Row<Transaction> last = table.Back();         // view, no allocation
auto when = last[&Transaction::timePoint];        // value
DB::StringRef iban = last[&Transaction::iban];    // view into the file; converts to std::string_view
Transaction copy = last.Materialize();

for (DB::Row<Transaction> row : table) ...        // live rows in id order
std::vector<Transaction> all = table.Materialize();

// predicates: a lambda, or column tags
const DB::Key key(ibanText);                      // encode the search key once, compare with one 8-byte load per row
auto rows = table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == key; }).Materialize();

inline constexpr DB::Column<&Transaction::iban> iban{};
inline constexpr DB::Column<&Transaction::amount> amount{};
inline constexpr DB::Column<&Transaction::timePoint> timePoint{};
auto big = table.Where(iban == ibanText && amount > 100'00).Count();

// composes with std::ranges
for (auto row : table | std::views::filter(...) | std::views::take(10)) ...
std::int64_t total = std::ranges::fold_left(table.Select<&Transaction::amount>(), 0LL, std::plus{});
```

`StringRef` compares against `std::string_view`, `std::string`, literals, `Key` and other refs without building a `std::string_view` first: the handle's length-and-prefix word rejects almost every non-match, the heap is read only for prefix-equal values longer than 12 bytes.

Predicates are template parameters: the lambda body inlines into the scan loop. There is no `std::function` and no per-row call.

## Ordered columns

```cpp
auto inRange = table.Range<&Transaction::timePoint>(from, to);     // from <= key <= to, binary search
auto since   = table.From<&Transaction::timePoint>(from);          // key >= from
auto newer   = table.After<&Transaction::timePoint>(last);         // key >  last
DB::RowId lb = table.LowerBound<&Transaction::timePoint>(from);

// range + property: Range narrows by binary search, Where filters inside it
auto rent = table.Range<&Transaction::timePoint>(from, to).Where(iban == x).Materialize();
// the same from one expression: bounds on the ordered column in a top-level && become the binary search
auto same = table.Where(timePoint >= from && timePoint <= to && iban == x).Materialize();
```

Keys may repeat; `Range` is inclusive. `orderedBy` must be an integer, enum or integral chrono member: floats have NaN, and strings have no binary search here. Tombstoned slots keep their key, so the search stays valid until `Compact()`. `std::ranges::lower_bound(table.Slots(), …)` is the escape hatch for anything else.

## Migrations

```cpp
struct TransactionV1 { /* as above */ static constexpr std::uint32_t version = 1; };

struct TransactionV2 {              // added `reference`: additive, no Migrate needed
    /* V1 members + */ std::string reference;
    static constexpr std::uint32_t version = 2;
    using Previous = TransactionV1;
};

struct Transaction {                // renamed description -> memo, merged amount + currency: needs Migrate
    std::string name, bic, iban, memo, reference;
    Money money;
    std::chrono::system_clock::time_point timePoint;

    static constexpr std::uint32_t version = 3;
    using Previous = TransactionV2;
    static Transaction Migrate(TransactionV2&& old) {
        return { .name = std::move(old.name), .bic = std::move(old.bic), .iban = std::move(old.iban), .memo = std::move(old.description),
                 .reference = std::move(old.reference), .money = { old.amount, old.currency }, .timePoint = old.timePoint };
    }
};
```

Opening a version-1 file with `Table<Transaction>` streams every live row through V1 → V2 (copied by member name, new members value-initialised) → `Migrate` → V3 into a new file that replaces the old one atomically; the original stays as `main.cdb.v1.bak`. When the new struct declares an `orderedBy` the old rows do not satisfy, the migration sorts them (stably) by the new key. The fingerprint covers column names, kinds, offsets, chrono periods and clocks, blob element types and the ordered column, so any of those changing without a version bump is refused. Without `Migrate`, every member of `Previous` must exist in the new struct with the same name and type (a removal or rename fails to compile with a message saying so). A file whose version equals the struct's but whose fingerprint differs throws `SchemaError` listing both column sets. `migrate = false` or `readOnly` on an older file throws `MigrationError`. `DB::Migrate<T>(path)` runs a migration without keeping the table open; `DB::PeekVersion(path)` reads the version alone.

## Thread safety and lifetimes

Like a standard container: any number of concurrent readers, writes need exclusive access from the caller. Exception: `Append`/`Emplace`/`Erase` are safe while other threads read (rows are published with a release store, tombstones with an atomic byte and a monotonic flag that switches scans to the checking path). A scan started before an `Erase` may still yield that row. `Row` views and `StringRef`s stay valid across `Append` and `Erase` and are invalidated by `Update`/`Set` of their row, `Compact`, `Clear`, migration and destruction. Extents are mapped into a reserved virtual range, so the mapping never moves while the table is open. A `Query` over a plain range (`Rows`, `Range`, `From`, …) can be iterated as a const object; one built with `Where` cannot, because `std::ranges::filter_view` has no const iteration.

## Errors

All exceptions derive from `Crafter::DB::Error` (a `std::runtime_error`): `IOError` (with `code` and `path`), `CorruptError`, `SchemaError`, `MigrationError`, `LockedError`, `OrderError`, `ReadOnlyError`. Out-of-range or erased ids throw `std::out_of_range` from `At()`; `operator[]` does not check.

## File format

One file per table: a 4 KiB header (magic, format and schema version, fingerprint, extent table, two CRC-32C commit records), schema pages (column descriptors and names, so `Inspect()` and `DynamicRow` can read any table without its struct), then extents. Rows and heap are two regions, each a chain of doubling extents appended at the end of the file and mapped back to back. Extents are aligned to the page size of the system that created the file (4 KiB on x86-64); a file created with smaller pages than the running kernel's cannot be mapped and is refused with a clear error. Filesystems without `fallocate` fall back to `posix_fallocate`. Appends publish rows to in-process readers immediately and become durable at `Flush()`: `fdatasync`, then the inactive commit record, then `msync`. On open the newest valid commit wins; rows and heap bytes beyond it, and uncommitted extents, are discarded. Deletes set a flag byte; `Compact()` rewrites. `Update` is safe against a process crash but not atomic against a power loss mid-row (documented limitation). Little-endian only. `Verify()` checks every handle, the ordered column and the live count.

## Performance

`tests/Benchmark` compares Crafter.DB with a cleaned-up copy of Nostro's format (iostream, 2-byte length-prefixed strings) on 100 000 transactions, warm cache, one run on a Xeon w7-3465X:

| Operation | Nostro | Crafter.DB | Note |
| --- | --- | --- | --- |
| append 100 000 rows | 16.5 ms | 11.1 ms | Crafter.DB includes `fdatasync`; Nostro syncs nothing |
| last timestamp, 1000 × open + read | 1.9 ms | 30.0 ms | opening a mapped table costs one `mmap` per extent (about 20 for this size); open once per process |
| load all, materialised | 36.0 ms | 10.4 ms | zero-copy scan of two columns: 0.8 ms |
| time range, 10 % of rows | 10.0 ms | 0.9 ms | binary search + contiguous slice |
| `iban == x` | 16.2 ms | 0.23–0.30 ms | 2–3 ns per row |
| time range + `iban == x` | 11.2 ms | 0.03 ms | |

File size for that table: 17.2 MB against Nostro's 9.3 MB (fixed-width handles plus the slack of the last extents; `Compact()` right-sizes).

## Building

This library uses [Crafter.Build](https://forgejo.catcrafts.net/Catcrafts/Crafter.Build), which ships clang 23 and libc++ 23:

```
crafter-build            # static library
crafter-build test       # build and run the tests
```

Linux only: the engine uses `flock`, `fallocate` and fixed-address `mmap`.

## Testing

Each test is `tests/<Name>/main.cpp` and is declared in `project.cpp`:

- `ShouldRoundTrip`, `ShouldKeepLayoutGolden`, `ShouldKeepViewsStableAcrossGrowth`, `ShouldCompareStringRefs`
- `ShouldQuery`, `ShouldEraseAndCompact`, `ShouldFormatCustomTypes`
- `ShouldMigrate`, `ShouldRejectSchemaChange`, `ShouldRecoverCommits`, `ShouldLockAcrossProcesses`, `ShouldVerify`, `ShouldReadWhileAppending`
- `Benchmark` (always passes, prints the table above)

## License

This library is licensed under the GNU Lesser General Public License, version 3 only (LGPL-3.0-only). See [LICENSE](LICENSE) for the license text. The LGPL supplements the GNU General Public License version 3, a copy of which is included as [LICENSE.GPL](LICENSE.GPL).

## Copyright

Copyright (C) 2026 Catcrafts®

catcrafts.net