Jorijn van der Graaf
38c9076e1d
AGENTS.template.md is what crafter-build writes AGENTS.md from on every build and test run: the partition map, the toolchain facts the code leans on, build and test commands, test conventions, code style, the design rules with their reasons, commit guidelines and the gotchas found in the storage and file layers. AGENTS.consumer.md is the section crafter-build copies into the AGENTS.md of every project that depends on this library: declaring the dependency, the table-struct rules, locking and durability, writing and querying, migrations, an errors table and how to test a consumer. Both were fact-checked claim by claim against the interfaces, the implementation units and the tests. AGENTS.md is build output and is now gitignored, as in the sibling libraries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
150 lines
31 KiB
Markdown
150 lines
31 KiB
Markdown
# AGENTS.md
|
|
|
|
Guidance for coding agents and contributors working on Crafter.DB itself. The [README](README.md) is the user-facing manual, and [AGENTS.consumer.md](AGENTS.consumer.md) is what agents working in a *consumer* of this library read: crafter-build copies it into the dependency-notes section of the consumer's `AGENTS.md` on every build. When any of the three disagrees with the code, the code wins; the interfaces under `interfaces/` are the contract, and `tests/ShouldKeepLayoutGolden/main.cpp` is the contract for the bytes on disk.
|
|
|
|
## Project overview
|
|
|
|
Crafter.DB is the Catcrafts embedded table store: one C++26 module, `Crafter.DB`, with twelve partitions behind the umbrella (thirteen interface files) and two implementation units, no dependencies, Linux only. A table is a plain struct with a `version`; a file is a memory-mapped array of fixed-width rows plus a heap for strings and blobs; reads are zero-copy views; queries are lambdas or column expressions; schema changes are C++ migrations run on open. It replaces a hand-rolled, iostream-based transaction log in the first Catcrafts project that needed one (that port is still to do) and serves any Crafter project that keeps rows in a file.
|
|
|
|
| Path | What lives there |
|
|
|---|---|
|
|
| `interfaces/Crafter.DB.cppm` | The umbrella: `export import`s every partition. |
|
|
| `interfaces/Crafter.DB-<Partition>.cppm` | One partition per file, each but `:Error` opening with a comment that says what the partition owns. All but `:File`, `:Storage` and two plain functions in `:Migration` (`DescribeColumns`, `PeekVersion`) are templates and inline code. |
|
|
| `implementations/Crafter.DB-File.cpp`, `-Storage.cpp` | The two implementation units, `Crafter.DB:File_impl` and `Crafter.DB:Storage_impl`. `File` is the only unit that includes a C header. |
|
|
| `project.cpp` | The build description: the static library `Crafter.DB` and, for `*-linux-gnu` targets, the fourteen tests. No dependencies, no lint rules. |
|
|
| `tests/<Name>/main.cpp` | One test program per directory, declared in `project.cpp`. |
|
|
| `AGENTS.template.md`, `AGENTS.consumer.md` | The source of this file, and the notes for consumers. Every build and test run writes `AGENTS.md` from the template; the library has no dependencies, so the dependency-notes section never appears here. `AGENTS.md` is gitignored: edit the template. |
|
|
| `bin/`, `build/` | Generated and gitignored. `bin/Crafter.DB-<target>-<march>-<mtune>-<hash>/` holds the PCMs, `std.pcm` and `libCrafter.DB.a`, `build/<same>/` the objects, and each test has a pair of its own. `build/test-logs/<Name>.log` keeps the output of failed tests. |
|
|
|
|
Partitions, from the bottom up (each imports only the ones above it):
|
|
|
|
- `:Error`: `Error : std::runtime_error` and its subclasses `IOError` (with `code` and `path`), `CorruptError`, `SchemaError`, `MigrationError`, `LockedError`, `OrderError`, `ReadOnlyError`. Everything the library throws derives from `Error`, except the `std::out_of_range` of `At()` and friends and the `std::length_error` of a `FixedString` that does not fit.
|
|
- `:Reflect`: compile-time introspection of aggregates without P2996: `memberCount`, `Member<I>`, `MemberType`, `MemberName<I, T>()` (parsed out of `std::source_location::function_name()`), `MemberIndex<T, Ptr>()`, `MemberIndexOf`, `TypeName<T>()`, `ForEachMember`, the `Aggregate` concept. The `static_assert`s at the bottom fail the library's own build if a compiler changes the shape of `function_name()`.
|
|
- `:Format`: the on-disk PODs (`FileHeader`, `CommitRecord`, `Extent`, `SchemaBlobHeader`, `ColumnDescriptor`, `Var`), `Kind`, the column flags, FNV-1a, CRC-32C, `SchemaInfo` (serialise, parse with range checks, fingerprint). Every struct size and the load-bearing offsets are pinned by `static_assert`s.
|
|
- `:Formatter`: `Formatter<T>`, the customisation point, with the `Scalar`, `FixedFormatter` and `VarFormatter` concepts; `Heap` and `HeapWriter`; `EncodeVar`/`VarBytes`; `Key`, `StringRef`, `BlobRef<E>`, `FixedString<N>`; the built-in formatters for `std::string`, `std::vector<Trivial>`, `std::chrono::duration` and `time_point`.
|
|
- `:Schema`: `Layout<T>`, computed `consteval`: members classified by `Shape`, nested aggregates flattened to dotted names, leaves placed by ascending alignment after the flags byte, the fingerprint; `TableType`, `HasOrderedBy`, `HasPrevious`, `HasMigrate`, `OrderableKey`; `CheckTable<T>()` with the diagnostics for the struct and its members (the size limits sit in `Layout` and `Arrange`); `MakeSchemaInfo<T>()`; the `SchemaTests` namespace of compile-time checks.
|
|
- `:File`: everything that talks to the OS: `OpenOptions`, `OpenMode`, `Durability`; `File` (descriptors, `flock`, `fallocate` with the `posix_fallocate` fallback, `pread`/`pwrite`, `fdatasync`/`fsync`); `MappedRegion` (a `PROT_NONE` reservation filled with `MAP_FIXED` extents); `MappedPages`; the `Os` namespace (private `mkdir -p`, rename without replace, hard link, directory sync, random bytes, page size, pid). Nothing here knows what a table is.
|
|
- `:Storage`: the type-erased engine. `TableFile` creates, opens, validates, recovers, grows, flushes, compacts and verifies a table knowing rows only as `rowSize` bytes and columns only through the file's descriptors; `Inspection` and `TableFile::Inspect`, `DynamicRow`, `TableStats`, `VerifyResult`, `CreateParams`.
|
|
- `:Codec`: typed encode/decode over `Layout<T>`: `EncodeRow`, `EmplaceRow`, `EncodeMember`, `DecodeRow`, `DecodeMember`, `View<M>`, `CopyLeaves`/`ZeroLeaves`, the ordered-column helpers (`LowerBoundSlot`, `UpperBoundSlot`, `CheckOrder`) and `AppendRows`.
|
|
- `:Row`: `RowId`, `Column<Ptr>`, `Row<T>` (the 16-byte view), `LiveIterator`/`LiveRange` (sentinel-terminated, skips tombstones), `SlotView` (random access over every slot), `DescribeRow`, `std::formatter<Row<T>>` and `std::formatter<StringRef>`.
|
|
- `:Query`: the expression types `Compare`, `And`, `Or`, `Not` and their operators on `Column` tags, `Bounds` and `CollectBounds` (bound extraction from a top-level `&&`), the chainable `Query<T, R>`.
|
|
- `:Migration`: `CopyByName`, `ConvertChain`, `RequireFingerprint`, `WriteConverted`/`RewriteFrom`/`WalkChain`, `MigrateTable`, and the exported `Migrate<T>(path)` and `PeekVersion(path)`.
|
|
- `:Table`: `Table<T>`, the user-facing handle: open/create/migrate, every write, iteration, `Where`/`Select`, the ordered-column searches, and `Compact`/`Clear` through `Rebuild`.
|
|
|
|
Numbers the code and the tests pin (`tests/ShouldKeepLayoutGolden`, the `SchemaTests` namespace): the README's `Transaction` is 88 bytes, 8-aligned, with `currency` at offset 2, `name` at 8, `bic` at 24, `iban` at 40, `description` at 56, `amount` at 72 and `timePoint` at 80; a `Var` handle is 16 bytes with 12 inline; the header is one 4 KiB page followed by at least one schema page; the first extent of each region is 16 KiB, extents double up to 1 GiB, and a region holds at most 32 of them (roughly 17 GiB); 32 GiB of address space is reserved per region; a row is at most 65535 bytes and a table at most 65535 columns; a value is below 4 GiB. A fresh table is 24 KiB without string or blob columns and 40 KiB with.
|
|
|
|
## Dev environment
|
|
|
|
crafter-build is the only tool, from its Flatpak (`alias crafter-build='flatpak run net.catcrafts.CrafterBuild'`; this machine has its `stable` and `beta` branches, both 4.0.0, and a build installed from a Crafter.Build checkout is branch `local`, chosen with `--branch=`). It compiles with its bundled clang 23.1.2 and libc++ 23.1.2, `-std=c++26`, `import std;`. The host's `/usr/bin/clang++` is not used and is not a supported compiler for this library. The project has no dependencies, so nothing is fetched and no network is needed. The host is Linux x86_64.
|
|
|
|
What that compiler does and does not have, checked on 2026-10-17 (the library leans on all of it):
|
|
|
|
- No P2996 reflection: `-freflection` is accepted and ignored, and there is no `<meta>`. `:Reflect` exists because of that and is the only partition to replace when a clang ships `std::meta`.
|
|
- Structured-binding packs (`auto&& [...m] = t;`), pack indexing (`m...[I]`), user-generated `static_assert` messages, deducing `this`, `[[clang::no_destroy]]`, `std::atomic_ref`, `std::ranges::fold_left`, `std::format` and `std::println`: all in use.
|
|
- libc++ 23 lacks `std::start_lifetime_as`, `std::inplace_vector` and `std::simd`, so row fields are read with constant-size `memcpy` (which folds to a load) and SIMD scans are not attempted.
|
|
- libc++'s `system_clock::duration` is microseconds (libstdc++'s is nanoseconds). The chrono period is part of the fingerprint for that reason, and the golden test pins `paramDen == 1000000`.
|
|
|
|
The Flatpak sandbox sees the host filesystem but has its own `/tmp` (`fs::temp_directory_path()` inside a test is a per-app directory under `$XDG_RUNTIME_DIR/.flatpak/net.catcrafts.CrafterBuild/tmp`, shared by instances running at the same time and gone once none is), keeps its caches under `~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/`, and a binary it built runs only inside it: `flatpak run --command=$PWD/bin/<Name>-<variant>/<Name> net.catcrafts.CrafterBuild`. That is how you see a passing test's output, which the runner does not keep, the Benchmark's table in particular. `touch tests/<Name>/main.cpp && crafter-build -v test <Name>` prints the exact compiler command, the model for a scratch program against the built PCMs.
|
|
|
|
`std.pcm` is built once per target, `--march` and compiler into `~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/<target>-<march>-<clang version>-<hash>/` and copied into each variant's directory; a toolchain update rebuilds it once, and a warm run of the whole suite takes about 2.5 seconds. The variant hash includes the compiler, so the directories of earlier compilers, and of the stub this repository started as (`bin/Crafter.DB-…-97c53ceb1aa99864/`, with colon-named PCMs), stay until `crafter-build clean`; glob for a directory rather than predicting its hash.
|
|
|
|
## Build, test and style commands
|
|
|
|
```bash
|
|
crafter-build # bin/Crafter.DB-<variant>/libCrafter.DB.a and the PCMs
|
|
crafter-build test # build and run the fourteen tests, Benchmark included
|
|
crafter-build test 'ShouldM*' # name glob
|
|
crafter-build test --list
|
|
crafter-build test --jobs=1 --timeout=120
|
|
crafter-build -v test ShouldQuery # with the compiler commands
|
|
crafter-build clean # delete bin/ and build/
|
|
```
|
|
|
|
`project.cpp` registers no lint rules and does not pull in Crafter.LintRules (a `#pragma crafter dependency` would `git pull` and rebuild it on every load of `project.cpp`), so `crafter-build lint` and `crafter-build format` exit 1 with "No lint rules defined." Style is checked by eye, see "Code style". There is no CI. Before a change is done, `crafter-build test` passes: 14 passed as of 2026-10-31. Failed, crashed and timed-out tests write `build/test-logs/<Name>.log`; a later passing run does not delete the file (three from the first day are still there), so check its timestamp before trusting it.
|
|
|
|
## Testing instructions
|
|
|
|
Tests are declared, not discovered: one `cfg.AddTest("<Name>").Dependencies({ &cfg });` line in `project.cpp`, inside the `if (cfg.target.ends_with("-linux-gnu"))` block (the engine uses `flock`, `fallocate` and fixed-address `mmap`; the musl target gets the library and no tests). The source is `tests/<Name>/main.cpp`, an `int main()` linked against the local `libCrafter.DB.a`: exit 0 passes, nonzero fails, 77 skips (unused here). The runner gives each test 60 s in its own process, in parallel.
|
|
|
|
Conventions the fourteen follow:
|
|
|
|
- Names are `Should<Verb><What>`; `Benchmark` is the exception. Open `main.cpp` with a comment that states the behaviour under test and the failure it guards against; `tests/ShouldRecoverCommits/main.cpp` and `tests/ShouldReadWhileAppending/main.cpp` show the shape, and a regression test names the bug.
|
|
- `Check(condition, what)` and `CheckEqual<T>(actual, expected, what)` in an anonymous namespace print `FAIL: …` (with both values for `CheckEqual`) and count; `main` prints `ok` and returns 0 only when nothing failed. Do not stop at the first failure unless later steps are meaningless without it.
|
|
- Each test works under `fs::temp_directory_path() / "crafter-db-<name>"`, `remove_all`ed first and last, with a name of its own because the runner is parallel. Table structs are declared per test: the README's `Transaction` where the reference layout matters, a one- or two-member struct where it does not (`Item`, `Entry`, `Tick`).
|
|
- Process-level behaviour is tested with `fork()`. `ShouldLockAcrossProcesses` opens the table in a child and maps the outcome to an exit code; `ShouldRecoverCommits` runs the writer in a child that ends with `_exit(0)` while its `Table` is still in scope, so no destructor and no `Flush` runs, which is the only honest way to simulate a crash. Those two `#include <sys/wait.h>` and `<unistd.h>` before their imports. Damage to a file is done with a `Patch(path, offset, bytes)` helper over `std::fstream` in `ShouldRecoverCommits`, `ShouldRejectSchemaChange` and `ShouldVerify`.
|
|
- Concurrency is tested with `std::jthread` readers scanning while the main thread appends and erases (`ShouldReadWhileAppending`), and such a run ends with `table.Verify().Ok()`.
|
|
- `Benchmark` never fails. It prints the legacy format (a cleaned-up copy of the hand-rolled iostream log Crafter.DB replaces) against Crafter.DB on 100 000 rows with a warm cache; raise `rowCount` for a serious run, and quote its numbers with the machine (the README's are a Xeon w7-3465X).
|
|
|
|
What is covered: layout and fingerprint (`ShouldKeepLayoutGolden`), encode/decode and reopen (`ShouldRoundTrip`), pointer stability across extents (`ShouldKeepViewsStableAcrossGrowth`), `StringRef`/`BlobRef` semantics (`ShouldCompareStringRefs`), every query form and the ordered column (`ShouldQuery`), tombstones and `Compact` (`ShouldEraseAndCompact`), every member shape and both formatter shapes (`ShouldFormatCustomTypes`), the migration chain and its refusals (`ShouldMigrate`), schema and corruption refusals (`ShouldRejectSchemaChange`), the commit protocol under crashes (`ShouldRecoverCommits`), locking (`ShouldLockAcrossProcesses`), `Verify` and `DynamicRow` (`ShouldVerify`), readers against a writer (`ShouldReadWhileAppending`). Not covered: a table on a 16 KiB-page kernel, `reserveBytes` exhaustion, the `posix_fallocate` fallback, `prefetch`.
|
|
|
|
A bug fix ships with a test that fails before it and passes after. A change to the bytes on disk makes `ShouldKeepLayoutGolden` fail: that is the point of it, so read design rule 4 before updating its expectations.
|
|
|
|
## Code style
|
|
|
|
Nothing is enforced by tooling; match the file you are in. The conventions are those of Crafter.LintRules' full set, applied by hand.
|
|
|
|
- The first two lines of every source are `// SPDX-License-Identifier: LGPL-3.0-only` and `// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®`, with a space after `//` (as Crafter.Build spells it; Crafter.Math, Network and Asset do not), then a blank line. Every interface but `:Error` then carries a comment block saying what the partition owns, before `export module Crafter.DB:<P>;`; keep it true when the partition changes.
|
|
- Four-space indentation, no tabs, LF, a final newline, K&R braces, `if (x) return;` on one line when it fits, one declaration per statement, `if (` and `for (` with a space and no space before a call's `(`.
|
|
- `import std;` everywhere. The only C headers are in the global module fragment of `implementations/Crafter.DB-File.cpp` (after `module;`, with `#define _GNU_SOURCE 1`) and the two POSIX headers the forking tests include before their imports. `__builtin_offsetof` rather than `<cstddef>`.
|
|
- `PascalCase` for types, functions and concepts; `camelCase` for variables, members and constants alike (`inlineVarBytes`, `fnvOffset`, `rowFlagDeleted`), which is why the table statics are `version` and `orderedBy` while `Previous` and `Migrate` are a type and a function. Internal namespaces are `Reflect`, `Detail`, `Os` and `SchemaTests`. `enum class` only. Fixed-width integers; plain `int` and `long` appear only where POSIX or the language demand them (`main`, `waitpid`'s `status`, `operator++(int)`, what `sysconf` returns) and in the tests' `failures` counters.
|
|
- `std::format` and `std::println`; iostream appears only in the Benchmark's copy of the legacy format and in the tests' `Patch` helpers and fixture writers. Exception messages start with the path where one is known (`{}: …`; `Heap::Read`, `EncodeVar`, `CompactInto`, the reservation and the `msync` failure carry none). The `static_assert`s of `CheckTable`, `CopyByName` and `CheckEmplaceArgument` start with `Crafter.DB: ` and name the table type and the member (`Message` and `MemberMessage` in `:Schema`); a handful of internal limits assert without the prefix.
|
|
- Scalars and handles are read with `std::memcpy` into a local, and the flags byte through `std::atomic_ref<std::uint8_t>` because readers race with `Erase`; the only reinterpreted pointers are the zero-copy views (`StringRef::View`, `BlobRef::View`, `DynamicRow::Text`) and the schema name pool.
|
|
- Hot paths carry no indirection: predicates are template parameters, column offsets are `constexpr`, and `HeapWriter::Grow` is the only virtual call, reached only when an extent fills. A change to the scan loop, `StringRef`'s comparisons, `Row::operator[]` or the encoders comes with evidence, the Benchmark before and after or the `-O3` assembly of the loop; the maintainer reviews performance claims at that level.
|
|
- Comments say why: the failure mode, the race, the rejected alternative (`MADV_DONTFORK` in `-File.cpp`, the `external<T>` object in `:Reflect`, the sentinel iterator in `:Row`). The code carries long comments on purpose; do not strip them, write them for new non-obvious code, and give commit bodies the same treatment.
|
|
|
|
## Design rules
|
|
|
|
Decided deliberately; change them only with the maintainer's say-so.
|
|
|
|
1. **Structs are the schema.** No macros, registration, traits to specialise or code generation. Every rule about a table struct's members is a `static_assert` in `CheckTable<T>()` with a message naming the table and member, the size limits are `static_assert`s and consteval throws in `Layout` and `Arrange`, and a type with no `Formatter` is a clean concept failure. Misuse is a compile error, never a runtime surprise.
|
|
2. **One file per table, two regions, fixed addresses.** Rows and heap are chains of doubling extents appended at the end of the file and mapped `MAP_FIXED` into a `PROT_NONE` reservation, so row `i` is `base + i * rowSize` and no pointer moves while the table is open. No free-space management: `Compact()` right-sizes. Region slots 2 and 3 are reserved for a persisted index and a WAL.
|
|
3. **Fixed-width rows, 16-byte handles, a bump-arena heap.** Flags byte at offset 0, leaves stable-sorted by alignment, row size rounded to `rowAlign`. Strings and blobs are Umbra handles (length, 4-byte prefix, then 8 inline bytes or a heap offset), so values of up to 12 bytes live inline (strings, and blobs whose elements are aligned to 4 or less). The heap has no per-entry header; `Update`, `Set` and `Erase` orphan bytes that only `Compact()` reclaims. Little-endian only.
|
|
4. **The file format is a contract pinned twice**: by the `static_assert`s on every struct size and the load-bearing offsets in `:Format`, and by `tests/ShouldKeepLayoutGolden` with an independent spelling of the fingerprint and the exact handle bytes. `Kind` is append-only. The fingerprint covers names, kinds, offsets, sizes, flags, chrono period, blob element type, clock and the ordered column, and what it covers is what a `version` bump is required for. Changing any byte an existing file holds, or the fingerprint's inputs, is a format change: bump `formatVersion` (1 today, exact match required) and say so; never reinterpret old files silently.
|
|
5. **Durability by two commit records, not a journal.** An append is visible to in-process readers at once (release stores of the published counts) and durable at `Flush()`: `fdatasync`, the inactive commit slot with its CRC, `msync`. On open the newest valid commit wins and everything past it, uncommitted extents included, is discarded; an unclean close recounts tombstones. `Update` and `Set` commit their new heap bytes before the row points at them, and the row bytes themselves are not atomic against a power loss; that limitation is documented, and the WAL region is the v2 answer, not an `fsync` per row.
|
|
6. **A type-erased engine under a thin typed layer.** `TableFile` knows rows as `rowSize` bytes and columns through the file's own descriptors, so `Inspect`, `Verify`, `Compact`, `DynamicRow` and a migration from a struct that no longer exists work without the C++ type; `Table<T>` adds compile-time offsets and encoders on top; `:File` knows nothing about tables at all. Keep each layer ignorant of the one above it.
|
|
7. **Queries are C++ and inline.** A predicate is a template parameter stored by value in the view: no `std::function`, no virtual call, no per-row call. Search keys are encoded once (`Key`, or automatically in a column expression). Bound extraction reads only comparisons on the `orderedBy` column in a top-level `&&` chain; `||` and `!` leave the whole predicate as a filter, which is always correct and only ever a narrowing. No SQL and no planner; hash indexes (`indexedBy`) are deferred, not refused.
|
|
8. **One writer, many in-process readers, a standard container's contract plus a little.** `flock` exclusive for a writer, shared for `readOnly`, between processes and between two `Table` objects in one. `Append`, `Emplace` and `Erase` are safe while other threads read (published counts, a monotonic tombstone flag, sentinel-terminated iteration that never compares against a moving end); `Update`, `Set`, `Compact` and `Clear` need exclusive access from the caller. `Row` views and `StringRef`s survive `Append` and `Erase`.
|
|
9. **Private by default, and never replace.** Files `0600` in `0700` directories, `O_NOFOLLOW`, `MADV_DONTDUMP`, creation through an `O_EXCL` temp file and a rename that fails when the target exists (racing creators share one file), rewrites through a temp file and `rename` (a visible table file is always valid). `MADV_DONTFORK` was tried and rejected: a forked child may read an inherited table, and its writes, `Flush` and destructor flush are refused by the owner-process check instead.
|
|
10. **Every byte from a file is checked before it is used**: header fields, both commit CRCs, extents against the committed length, every descriptor field in `SchemaInfo::Parse`, every heap handle against the published heap length (`Heap::Read`), the page-size granule against the running kernel. A new field read from a file gets its check and a case in `ShouldRejectSchemaChange` or `ShouldVerify`.
|
|
11. **Linux, no dependencies, no host split.** The library needs `flock`, `fallocate` and fixed-address `mmap` and says so; there is no `#ifdef` for another platform and no portability layer to grow. It depends on nothing, so a consumer's build fetches nothing for it.
|
|
|
|
## Commit and PR guidelines
|
|
|
|
- Branch off `master` (`claude/<topic>` is the pattern for agent work in the sibling repositories) and open the PR against `master` on the project's Forgejo. The repository is `Catcrafts/Crafter.DB` on the project's Forgejo: `forgejo@forgejo.catcrafts.net:Catcrafts/Crafter.DB.git` over SSH, `https://forgejo.catcrafts.net/Catcrafts/Crafter.DB.git` for consumers. A checkout whose `origin` still names the personal namespace the repository moved from can fetch (HTTPS redirects) but not push over SSH ("Push to create is not enabled"); repoint `origin`. Pushes go over SSH.
|
|
- The history has one commit, so take the voice from the sibling repositories: `type(scope): imperative summary` with `feat`, `fix`, `refactor`, `docs` or `test` and the partition as the scope (`storage`, `schema`, `query`, `migration`, `file`, `format`, `reflect`). Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns, with the numbers when something was measured. Add a `Co-Authored-By:` trailer when an agent wrote it.
|
|
- Separate commits for the fix, its test and the README change when each stands on its own. The README is the manual: keep its module table, examples and "Performance" table true to the code. Update `AGENTS.consumer.md` in the same change when a signature, a default, a limit or an error behaviour consumers lean on changes; consumers pick it up on their next build.
|
|
- A format change (design rule 4) updates `ShouldKeepLayoutGolden` in the same commit, and the commit message says it is one.
|
|
- `crafter-build test` green before pushing. There is no CI.
|
|
- Never commit `bin/`, `build/` or `AGENTS.md`.
|
|
|
|
## Security considerations
|
|
|
|
- A table file is untrusted input: anything on disk may have been truncated, bit-flipped or crafted. Design rule 10 is the defence, and `ShouldRejectSchemaChange` and `ShouldVerify` are its regression suite. `Verify()` is the deep check and returns problems rather than throwing, so a tool can print them all. There is no per-row checksum: a flipped byte inside a scalar column goes unnoticed.
|
|
- The defaults keep data private (`0600`, `0700`, `O_NOFOLLOW` against a swapped symlink, `MADV_DONTDUMP` against core dumps). Do not loosen a default; a consumer that wants a readable file sets `filePermissions`.
|
|
- The lock is advisory (`flock`). A process that ignores it, or `cp`, can read the file mid-write, and what it reads is valid up to the writer's last `Flush` because only committed rows count on open.
|
|
- `reserveBytes` (32 GiB per region, two regions per table) is address space, not memory, but a tight `ulimit -v` or hundreds of open tables turns it into an `IOError` at open.
|
|
- `project.cpp` has no dependencies, so loading it executes nothing fetched from the network; keep it that way, or pin a `commit` on anything added.
|
|
- Tests write only under `fs::temp_directory_path()`, fork only to exercise locking and crashes, and never touch the network.
|
|
|
|
## Gotchas
|
|
|
|
- Adding a partition is three edits: the interface file, the `ifaces` array in `project.cpp` and the umbrella `Crafter.DB.cppm`; an implementation unit also goes into the `impls` array. Keep the order of the "Project overview" list: a partition imports only what is above it.
|
|
- `MemberName` depends on clang spelling `function_name()` as `… [P = &external.iban]` and `TypeName` on `… [T = <qualified type name>]`; the `static_assert`s at the bottom of `:Reflect` catch a change at the library's build. `Reflect::external<T>` is `[[clang::no_destroy]] inline const T` on purpose: an undefined `extern` would be cheaper but warns at every instantiation.
|
|
- Leaf indices follow declaration order, depth first; column offsets do not. A `std::optional` contributes a presence leaf named `name?` before its payload, a nested aggregate flattens to `outer.inner`, a `std::array` of scalars to `name[i]`, and the flags byte is column 0 but no leaf. `Layout<T>::Offset(leaf)` and `Column(leaf)` translate.
|
|
- Reordering two members of the same alignment class changes offsets and therefore the fingerprint; the README's `Transaction` has `currency` at offset 2, ahead of every string. Both follow from rule 3, and a `SchemaError` is the symptom in a consumer that forgot the `version` bump.
|
|
- `char` is always `Kind::Int8` whatever the platform's signedness, `wchar_t` depends on its size, `long double` is not storable, `std::optional<std::optional<…>>` is rejected (no distinct column names), and a `Formatter` may override `kind` only when its `Stored` is a single column.
|
|
- The `Transaction` inside `SchemaTests` and the `Probe` inside `:Reflect` are compile-time tests that run on every build of the library, not examples to copy from.
|
|
- `TableFile::Inspect` is a static member; the README's module table calls it `Inspect(path)`. The tests spell it `DB::TableFile::Inspect(path)`.
|
|
- Extents are aligned to the larger of 4 KiB and the running kernel's page size at creation, and `header.pageSize` records it. A file created on a 4 KiB kernel is refused on a 16 KiB one with an `IOError` (`not_supported`) rather than mapped badly; nothing tests that path.
|
|
- A region tops out at 32 extents (`IOError`, `file_too_large`, "Compact() it"); `Compact` sizes the first extent of the new file to the live payload, which resets the chain. A reservation that fills throws `IOError` (`not_enough_memory`) naming `OpenOptions::reserveBytes`.
|
|
- `Table::Open` is a loop for a reason: the first open fails with `no_such_file_or_directory`, the create may lose a race and fail with `file_exists`, and the second pass opens the other process's file. Both halves of the race are needed, and `TableFile::Create` never replaces.
|
|
- `RemoveStaleTemps` runs on every writable open, after the exclusive lock is taken, and deletes the `<file>.tmp-*` siblings it can lock exclusively; the probe spares a temp another process is still writing. The twelve-hex-digit suffix comes from `Os::RandomBytes`.
|
|
- A migration's backup `<file>.v<old>.bak` is a hard link to the old inode taken before the rename, so it costs nothing until the original is replaced, and it fails on a filesystem without hard links.
|
|
- `Deleted()` reads the flags byte through a relaxed `atomic_ref`, `MarkDeleted` raises the monotonic `tombstones` flag before the byte, and `LiveRange::begin()` reads that flag when iteration starts rather than when the range was built, so a `Query` built before an `Erase` still skips the row. A scan that started before the `Erase` may yield it; that is documented, not a bug.
|
|
- `LowerBound`/`UpperBound` search every slot, tombstones included, because an erased row keeps its key; only `Compact` drops keys. A `Where` with bounds on the ordered column does the same.
|
|
- A `Query` over `Rows`, `Range`, `From` and the like can be iterated as a `const` object; one built with `Where` cannot, because `std::ranges::filter_view` has no const iteration (the deducing-`this` `requires` clauses in `Query` are what makes the first case work). `Query::Select` is rvalue-only: `std::move` a named query first.
|
|
- `Emplace` converts arguments implicitly, never with `static_cast`, so an `int` for an enum member or swapped arguments fail to compile with a message naming the member. `Append` of a range that is not both sized and random-access buffers it first, so the order check covers the whole batch before anything is written.
|
|
- `Set<&T::member>` copies only that member's leaves back into the slot (`CopyLeaves`) and leaves the flags byte alone, so an `Erase` that raced with it is not undone even though the contract gives `Set` exclusive access. Do not "simplify" it to a whole-row `memcpy`.
|
|
- After `fork()` the child's `Table` reads fine and refuses every write, `Flush` and its destructor's flush (`OwnedByThisProcess`): the parent's counters in the child are stale and must never reach the file. Do not make the destructor flush there.
|
|
- The Benchmark's `last timestamp (1000x open+read)` row is the one operation where the legacy format wins (about 2 ms against 30 ms): a mapped open costs one `mmap` per extent. The answer is to open once per process, which the README and the consumer notes say; do not try to win that row by changing the format.
|
|
- `crafter-build test` prints "Building std PCM" once per configuration (the library and each test) on every run; the step returns at once when the cached PCM is current, and only a toolchain update rebuilds it.
|