Catcrafts/Crafter.DB
Watch
2
0
Fork
You've already forked Crafter.DB
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit: Crafter.DB

Browse source 
Header-only C++26 module library for typed, file-backed tables with
schema reflection, migrations, queries, and crash-safe commits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-17 04:16:51 +02:00
commit bfd0cc261b
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
34 changed files with 7506 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
build/
bin/

65
LICENSE Normal file
View file

@ -0,0 +1,65 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates the terms and conditions of version 3 of the GNU General Public License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, “this License” refers to version 3 of the GNU Lesser General Public License, and the “GNU GPL” refers to version 3 of the GNU General Public License.
“The Library” refers to a covered work governed by this License, other than an Application or a Combined Work as defined below.
An “Application” is any work that makes use of an interface provided by the Library, but which is not otherwise based on the Library. Defining a subclass of a class defined by the Library is deemed a mode of using an interface provided by the Library.
A “Combined Work” is a work produced by combining or linking an Application with the Library. The particular version of the Library with which the Combined Work was made is also called the “Linked Version”.
The “Minimal Corresponding Source” for a Combined Work means the Corresponding Source for the Combined Work, excluding any source code for portions of the Combined Work that, considered in isolation, are based on the Application, and not on the Linked Version.
The “Corresponding Application Code” for a Combined Work means the object code and/or source code for the Application, including any data and utility programs needed for reproducing the Combined Work from the Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a facility refers to a function or data to be supplied by an Application that uses the facility (other than as an argument passed when the facility is invoked), then you may convey a copy of the modified version:
a) under this License, provided that you make a good faith effort to ensure that, in the event an Application does not supply the function or data, the facility still operates, and performs whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from a header file that is part of the Library. You may convey such object code under terms of your choice, provided that, if the incorporated material is not limited to numerical parameters, data structure layouts and accessors, or small macros, inline functions and templates (ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the Library is used in it and that the Library and its use are covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that, taken together, effectively do not restrict modification of the portions of the Library contained in the Combined Work and reverse engineering for debugging such modifications, if you also do each of the following:
a) Give prominent notice with each copy of the Combined Work that the Library is used in it and that the Library and its use are covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license document.
c) For a Combined Work that displays copyright notices during execution, include the copyright notice for the Library among these notices, as well as a reference directing the user to the copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.
1) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (a) uses at run time a copy of the Library already present on the user's computer system, and (b) will operate properly with a modified version of the Library that is interface-compatible with the Linked Version.
e) Provide Installation Information, but only if you would otherwise be required to provide such information under section 6 of the GNU GPL, and only to the extent that such information is necessary to install and execute a modified version of the Combined Work produced by recombining or relinking the Application with a modified version of the Linked Version. (If you use option 4d0, the Installation Information must accompany the Minimal Corresponding Source and Corresponding Application Code. If you use option 4d1, you must provide the Installation Information in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the Library side by side in a single library together with other library facilities that are not Applications and are not covered by this License, and convey such a combined library under terms of your choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities, conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions of the GNU Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Library as you received it specifies that a certain numbered version of the GNU Lesser General Public License “or any later version” applies to it, you have the option of following the terms and conditions either of that published version or of any later version published by the Free Software Foundation. If the Library as you received it does not specify a version number of the GNU Lesser General Public License, you may choose any version of the GNU Lesser General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide whether future versions of the GNU Lesser General Public License shall apply, that proxy's public statement of acceptance of any version is permanent authorization for you to choose that version for the Library.

674
LICENSE.GPL Normal file
View file

@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.

238
README.md Normal file
View file

@ -0,0 +1,238 @@
# Crafter.DB
An embedded, typed table store for C++26: a table is a plain struct, a file is a memory-mapped array of fixed-width rows, queries are C++. Built for the Catcrafts projects (it replaces Nostro's hand-rolled transaction log) and distributed as one C++ module, `Crafter.DB`.
- **Structs are the schema.** Any aggregate with a `version` is a table. No macros, no registration, no code generation: member names, types and offsets are derived at compile time.
- **Fixed-width rows, out-of-line strings.** Every row is `rowSize` bytes. Strings and blobs live in a heap region; the row holds a 16-byte handle that keeps values of up to 12 bytes, and the first 4 bytes of longer ones, inline, so `iban == x` is one 8-byte compare per row.
- **Zero-copy reads.** `Row<T>` is a 16-byte view; `row[&T::iban]` reads one column straight from the mapping. `Materialize()` copies only when you ask.
- **Queries are C++.** Lambdas over rows, or column expressions (`iban == x && amount > 100`), composed with `std::ranges`. A declared `orderedBy` column gives binary search (`Range`, `From`, `After`), also from a combined expression.
- **Versioned, with C++ migrations.** The file carries the struct's `version` and a layout fingerprint. A struct that changed without a version bump is refused; an older file is migrated through the `Previous` chain on open.
- **Crash safe appends.** Two CRC-protected commit records; a crash loses at most what was not yet flushed, never committed rows.
- **Private by default.** Files are created `0600` in `0700` directories and excluded from core dumps.
## Modules
`import Crafter.DB;` re-exports these partitions, all in `namespace Crafter::DB`:
| Partition | Contents |
| --- | --- |
| `:Table` | `Table<T>`: open/create/migrate, `Append`, `Emplace`, `Update`, `Set`, `Erase`, `EraseIf`, `Compact`, `Clear`, `Flush`, iteration, `Where`, `Select`, `Range`/`From`/`After`/`Until`/`Before`, `LowerBound`/`UpperBound`, `Verify`. |
| `:Row` | `Row<T>` (column views, `Materialize`), `RowId`, `Column<&T::m>` tags, `LiveRange`, `SlotView`, `std::formatter<Row<T>>`. |
| `:Query` | Column expressions (`==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `\|\|`, `!`), bound extraction, the chainable `Query` result (`Materialize`, `Count`, `First`, `Any`, `Where`, `Select`). |
| `:Formatter` | `Formatter<T>`, the customisation point for member types; `StringRef`, `BlobRef<E>`, `Key`, `FixedString<N>`; built-in formatters for `std::string`, `std::vector<Trivial>`, `std::chrono`. |
| `:Schema` | `Layout<T>`: the compile-time row layout, fingerprint and diagnostics; `TableType`, `HasOrderedBy`, `HasPrevious`, `HasMigrate`. |
| `:Migration` | `Migrate<T>(path)`, `PeekVersion(path)`, the chain walk and the by-name copy. |
| `:Storage` | The type-erased engine: `TableFile`, `Inspect(path)`, `DynamicRow`, `VerifyResult`, `TableStats`. |
| `:File` | `OpenOptions`, `OpenMode`, `Durability`; file descriptors, locks, reserved mappings. |
| `:Format` | The on-disk structures (`FileHeader`, `CommitRecord`, `ColumnDescriptor`, `Var`), `Kind`, FNV-1a, CRC-32C, `SchemaInfo`. |
| `:Codec` | Typed encode/decode over a `Layout<T>`. |
| `:Reflect` | Compile-time member count, names, types and pointer-to-member lookup for aggregates. |
| `:Error` | `Error` and its subclasses. |
## Defining a table
```cpp
import Crafter.DB;
using namespace Crafter;
enum class Currency : std::uint16_t { Euro };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency; // enums, integers, floats: stored as is
std::int64_t amount;
std::chrono::system_clock::time_point timePoint; // built-in formatter: rep + period
static constexpr std::uint32_t version = 1; // required
static constexpr auto orderedBy = &Transaction::timePoint; // optional: enforced on writes, enables Range()
};
```
A table struct is an aggregate (public members, no base classes with members, no user-declared constructors) that is default constructible and declares `version`. Every member must be storable: arithmetic types, `bool`, `std::byte`, enums, `std::chrono::duration` and `time_point`, `std::string`, `std::vector<E>` for trivially copyable `E`, `std::optional<M>`, `std::array<E, N>`, `FixedString<N>`, nested aggregates of storable members (flattened into `outer.inner` columns), and anything with a `Formatter`. Pointers and references are rejected at compile time; each `static_assert` names the table and the member.
The row above is 88 bytes: a flags byte, `currency`, four 16-byte string handles, `amount`, `timePoint`.
## Storing your own types
Specialise `Crafter::DB::Formatter<T>` the way you would `std::formatter`. A fixed-width formatter maps to a storable `Stored` type:
```cpp
template<>
struct Crafter::DB::Formatter<Percent> {
using Stored = std::int32_t; // basis points
static Stored Encode(const Percent& value) noexcept { return value.BasisPoints(); }
static Percent Decode(const Stored& stored) noexcept { return Percent(stored / 100.0); }
};
```
A variable-length formatter writes into the heap. Inheriting the `std::string` one and overriding the ends is the common shape:
```cpp
template<>
struct Crafter::DB::Formatter<Tags> : Crafter::DB::Formatter<std::string> {
static Var Encode(const Tags& value, HeapWriter& heap) { return Formatter<std::string>::Encode(value.Joined(), heap); }
static Tags Materialize(StringRef view) { return Tags::Split(view.View()); } // row[&T::tags] still hands out the raw StringRef
};
```
## Opening a table
```cpp
DB::Table<Transaction> table(cachePath / "transactions" / "bunq" / "main.cdb"); // created if missing, migrated if older
DB::Table<Transaction> frozen(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = true });
```
`OpenOptions` (designated initialisers): `mode` (`OpenOrCreate`, `OpenExisting`, `CreateNew`), `readOnly`, `createDirectories`, `filePermissions` (0600), `directoryPermissions` (0700), `durability` (`OnFlush`: durable after `Flush()` and on destruction; `EveryAppend`), `migrate`, `backupBeforeMigrate` (keeps `<file>.v<old>.bak`), `reserveBytes` (virtual address space per region, 32 GiB), `excludeFromCoreDumps`, `prefetch`.
One `Table` per file at a time: a writer takes an exclusive `flock`, read-only tables take shared locks. A conflicting open throws `LockedError`, in the same process as well as from another. Two processes creating the same table at once end up sharing one file (the create never replaces an existing file).
The destructor flushes but cannot report a failure; call `Close()` when a failed final flush must surface as an exception. After `fork()`, the child may read an inherited table (the mapping is shared) but every write, `Flush` and the destructor's flush are refused in the child, so its stale counters never reach the file.
## Writing
```cpp
DB::RowId id = table.Append(tr); // ids are slot indices
table.Append(std::span<const Transaction>(rows)); // one reservation, one publish
table.Emplace("Albert Heijn", "ABNANL2A", iban, "groceries", Currency::Euro, -2345, now); // one argument per member, strings need no std::string
table.Update(id, tr);
table.Set<&Transaction::description>(id, "corrected");
table.Erase(id); // tombstone; ids stay valid
table.EraseIf([](DB::Row<Transaction> r) { return r[&Transaction::amount] == 0; });
table.Flush(); // fdatasync + commit record
table.Compact(); // rewrite without tombstones and heap garbage; invalidates ids and views
```
On an `orderedBy` table, a write whose key would break the order throws `OrderError` before anything is written; a batch is all-or-nothing. `Update` and `Set` commit any new heap bytes to disk before the row starts pointing at them, so a crash leaves the old row or the new one, never a dangling reference; the row bytes themselves are not atomic against a power loss mid-write.
## Reading and querying
```cpp
table.Size(); // live rows
DB::Row<Transaction> last = table.Back(); // view, no allocation
auto when = last[&Transaction::timePoint]; // value
DB::StringRef iban = last[&Transaction::iban]; // view into the file; converts to std::string_view
Transaction copy = last.Materialize();
for (DB::Row<Transaction> row : table) ... // live rows in id order
std::vector<Transaction> all = table.Materialize();
// predicates: a lambda, or column tags
const DB::Key key(ibanText); // encode the search key once, compare with one 8-byte load per row
auto rows = table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == key; }).Materialize();
inline constexpr DB::Column<&Transaction::iban> iban{};
inline constexpr DB::Column<&Transaction::amount> amount{};
inline constexpr DB::Column<&Transaction::timePoint> timePoint{};
auto big = table.Where(iban == ibanText && amount > 100'00).Count();
// composes with std::ranges
for (auto row : table | std::views::filter(...) | std::views::take(10)) ...
std::int64_t total = std::ranges::fold_left(table.Select<&Transaction::amount>(), 0LL, std::plus{});
```
`StringRef` compares against `std::string_view`, `std::string`, literals, `Key` and other refs without building a `std::string_view` first: the handle's length-and-prefix word rejects almost every non-match, the heap is read only for prefix-equal values longer than 12 bytes.
Predicates are template parameters: the lambda body inlines into the scan loop. There is no `std::function` and no per-row call.
## Ordered columns
```cpp
auto inRange = table.Range<&Transaction::timePoint>(from, to); // from <= key <= to, binary search
auto since = table.From<&Transaction::timePoint>(from); // key >= from
auto newer = table.After<&Transaction::timePoint>(last); // key > last
DB::RowId lb = table.LowerBound<&Transaction::timePoint>(from);
// range + property: Range narrows by binary search, Where filters inside it
auto rent = table.Range<&Transaction::timePoint>(from, to).Where(iban == x).Materialize();
// the same from one expression: bounds on the ordered column in a top-level && become the binary search
auto same = table.Where(timePoint >= from && timePoint <= to && iban == x).Materialize();
```
Keys may repeat; `Range` is inclusive. `orderedBy` must be an integer, enum or integral chrono member: floats have NaN, and strings have no binary search here. Tombstoned slots keep their key, so the search stays valid until `Compact()`. `std::ranges::lower_bound(table.Slots(), …)` is the escape hatch for anything else.
## Migrations
```cpp
struct TransactionV1 { /* as above */ static constexpr std::uint32_t version = 1; };
struct TransactionV2 { // added `reference`: additive, no Migrate needed
/* V1 members + */ std::string reference;
static constexpr std::uint32_t version = 2;
using Previous = TransactionV1;
};
struct Transaction { // renamed description -> memo, merged amount + currency: needs Migrate
std::string name, bic, iban, memo, reference;
Money money;
std::chrono::system_clock::time_point timePoint;
static constexpr std::uint32_t version = 3;
using Previous = TransactionV2;
static Transaction Migrate(TransactionV2&& old) {
return { .name = std::move(old.name), .bic = std::move(old.bic), .iban = std::move(old.iban), .memo = std::move(old.description),
.reference = std::move(old.reference), .money = { old.amount, old.currency }, .timePoint = old.timePoint };
}
};
```
Opening a version-1 file with `Table<Transaction>` streams every live row through V1 → V2 (copied by member name, new members value-initialised) → `Migrate` → V3 into a new file that replaces the old one atomically; the original stays as `main.cdb.v1.bak`. When the new struct declares an `orderedBy` the old rows do not satisfy, the migration sorts them (stably) by the new key. The fingerprint covers column names, kinds, offsets, chrono periods and clocks, blob element types and the ordered column, so any of those changing without a version bump is refused. Without `Migrate`, every member of `Previous` must exist in the new struct with the same name and type (a removal or rename fails to compile with a message saying so). A file whose version equals the struct's but whose fingerprint differs throws `SchemaError` listing both column sets. `migrate = false` or `readOnly` on an older file throws `MigrationError`. `DB::Migrate<T>(path)` runs a migration without keeping the table open; `DB::PeekVersion(path)` reads the version alone.
## Thread safety and lifetimes
Like a standard container: any number of concurrent readers, writes need exclusive access from the caller. Exception: `Append`/`Emplace`/`Erase` are safe while other threads read (rows are published with a release store, tombstones with an atomic byte and a monotonic flag that switches scans to the checking path). A scan started before an `Erase` may still yield that row. `Row` views and `StringRef`s stay valid across `Append` and `Erase` and are invalidated by `Update`/`Set` of their row, `Compact`, `Clear`, migration and destruction. Extents are mapped into a reserved virtual range, so the mapping never moves while the table is open. A `Query` over a plain range (`Rows`, `Range`, `From`, …) can be iterated as a const object; one built with `Where` cannot, because `std::ranges::filter_view` has no const iteration.
## Errors
All exceptions derive from `Crafter::DB::Error` (a `std::runtime_error`): `IOError` (with `code` and `path`), `CorruptError`, `SchemaError`, `MigrationError`, `LockedError`, `OrderError`, `ReadOnlyError`. Out-of-range or erased ids throw `std::out_of_range` from `At()`; `operator[]` does not check.
## File format
One file per table: a 4 KiB header (magic, format and schema version, fingerprint, extent table, two CRC-32C commit records), schema pages (column descriptors and names, so `Inspect()` and `DynamicRow` can read any table without its struct), then extents. Rows and heap are two regions, each a chain of doubling extents appended at the end of the file and mapped back to back. Extents are aligned to the page size of the system that created the file (4 KiB on x86-64); a file created with smaller pages than the running kernel's cannot be mapped and is refused with a clear error. Filesystems without `fallocate` fall back to `posix_fallocate`. Appends publish rows to in-process readers immediately and become durable at `Flush()`: `fdatasync`, then the inactive commit record, then `msync`. On open the newest valid commit wins; rows and heap bytes beyond it, and uncommitted extents, are discarded. Deletes set a flag byte; `Compact()` rewrites. `Update` is safe against a process crash but not atomic against a power loss mid-row (documented limitation). Little-endian only. `Verify()` checks every handle, the ordered column and the live count.
## Performance
`tests/Benchmark` compares Crafter.DB with a cleaned-up copy of Nostro's format (iostream, 2-byte length-prefixed strings) on 100 000 transactions, warm cache, one run on a Xeon w7-3465X:
| Operation | Nostro | Crafter.DB | Note |
| --- | --- | --- | --- |
| append 100 000 rows | 16.5 ms | 11.1 ms | Crafter.DB includes `fdatasync`; Nostro syncs nothing |
| last timestamp, 1000 × open + read | 1.9 ms | 30.0 ms | opening a mapped table costs one `mmap` per extent (about 20 for this size); open once per process |
| load all, materialised | 36.0 ms | 10.4 ms | zero-copy scan of two columns: 0.8 ms |
| time range, 10 % of rows | 10.0 ms | 0.9 ms | binary search + contiguous slice |
| `iban == x` | 16.2 ms | 0.23–0.30 ms | 2–3 ns per row |
| time range + `iban == x` | 11.2 ms | 0.03 ms | |
File size for that table: 17.2 MB against Nostro's 9.3 MB (fixed-width handles plus the slack of the last extents; `Compact()` right-sizes).
## Building
This library uses [Crafter.Build](https://forgejo.catcrafts.net/Catcrafts/Crafter.Build), which ships clang 23 and libc++ 23:
```
crafter-build # static library
crafter-build test # build and run the tests
```
Linux only: the engine uses `flock`, `fallocate` and fixed-address `mmap`.
## Testing
Each test is `tests/<Name>/main.cpp` and is declared in `project.cpp`:
- `ShouldRoundTrip`, `ShouldKeepLayoutGolden`, `ShouldKeepViewsStableAcrossGrowth`, `ShouldCompareStringRefs`
- `ShouldQuery`, `ShouldEraseAndCompact`, `ShouldFormatCustomTypes`
- `ShouldMigrate`, `ShouldRejectSchemaChange`, `ShouldRecoverCommits`, `ShouldLockAcrossProcesses`, `ShouldVerify`, `ShouldReadWhileAppending`
- `Benchmark` (always passes, prints the table above)
## License
This library is licensed under the GNU Lesser General Public License, version 3 only (LGPL-3.0-only). See [LICENSE](LICENSE) for the license text. The LGPL supplements the GNU General Public License version 3, a copy of which is included as [LICENSE.GPL](LICENSE.GPL).
## Copyright
Copyright (C) 2026 Catcrafts®
catcrafts.net

317
implementations/Crafter.DB-File.cpp Normal file
View file

@ -0,0 +1,317 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
module;
#define _GNU_SOURCE 1
#include <cerrno>
#include <cstdio>
#include <cstring>
#include <fcntl.h>
#include <sys/file.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
module Crafter.DB:File_impl;
import :File;
import :Error;
import std;
using namespace Crafter::DB;
namespace fs = std::filesystem;
namespace {
std::error_code Errno() noexcept {
return std::error_code(errno, std::generic_category());
}
[[noreturn]] void Fail(std::string_view what, const fs::path& path) {
throw IOError(what, Errno(), path);
}
}
// --- File -------------------------------------------------------------------
File::File(File&& other) noexcept : descriptor(other.descriptor), path(std::move(other.path)) {
other.descriptor = -1;
}
File& File::operator=(File&& other) noexcept {
if (this != &other) {
Close();
descriptor = other.descriptor;
path = std::move(other.path);
other.descriptor = -1;
}
return *this;
}
File::~File() {
Close();
}
File File::Open(const fs::path& path, bool writable) {
File file;
file.path = path;
// O_NOFOLLOW: a private cache file must not be swapped for a symlink to
// somewhere we would then happily write 0600 data into.
const std::int32_t flags = (writable ? O_RDWR : O_RDONLY) | O_CLOEXEC | O_NOFOLLOW;
file.descriptor = ::open(path.c_str(), flags);
if (file.descriptor == -1) Fail("cannot open", path);
return file;
}
File File::CreateExclusive(const fs::path& path, fs::perms permissions) {
File file;
file.path = path;
file.descriptor = ::open(path.c_str(), O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, static_cast<mode_t>(permissions));
if (file.descriptor == -1) Fail("cannot create", path);
return file;
}
void File::LockExclusive() {
if (::flock(descriptor, LOCK_EX | LOCK_NB) == -1) {
if (errno == EWOULDBLOCK) throw LockedError(std::format("{}: another Table holds this file open for writing", path.string()));
Fail("cannot lock", path);
}
}
void File::LockShared() {
if (::flock(descriptor, LOCK_SH | LOCK_NB) == -1) {
if (errno == EWOULDBLOCK) throw LockedError(std::format("{}: another Table holds this file open for writing", path.string()));
Fail("cannot lock", path);
}
}
std::uint64_t File::Size() const {
struct stat info{};
if (::fstat(descriptor, &info) == -1) Fail("cannot stat", path);
return static_cast<std::uint64_t>(info.st_size);
}
void File::Truncate(std::uint64_t length) {
if (::ftruncate(descriptor, static_cast<off_t>(length)) == -1) Fail("cannot truncate", path);
}
void File::Allocate(std::uint64_t offset, std::uint64_t length) {
if (::fallocate(descriptor, 0, static_cast<off_t>(offset), static_cast<off_t>(length)) == 0) return;
if (errno != EOPNOTSUPP && errno != ENOSYS) Fail("cannot allocate space", path);
// Filesystems without fallocate (FUSE, 9p, exFAT): posix_fallocate falls
// back to writing zeros, which equally guarantees the blocks exist.
const std::int32_t result = ::posix_fallocate(descriptor, static_cast<off_t>(offset), static_cast<off_t>(length));
if (result != 0) throw IOError("cannot allocate space", std::error_code(result, std::generic_category()), path);
}
void File::ReadAt(std::uint64_t offset, std::span<std::byte> out) const {
while (!out.empty()) {
const ssize_t got = ::pread(descriptor, out.data(), out.size(), static_cast<off_t>(offset));
if (got == -1) {
if (errno == EINTR) continue;
Fail("read failed", path);
}
if (got == 0) throw CorruptError(std::format("{}: file ends {} bytes short", path.string(), out.size()));
out = out.subspan(static_cast<std::size_t>(got));
offset += static_cast<std::uint64_t>(got);
}
}
void File::WriteAt(std::uint64_t offset, std::span<const std::byte> in) {
while (!in.empty()) {
const ssize_t put = ::pwrite(descriptor, in.data(), in.size(), static_cast<off_t>(offset));
if (put == -1) {
if (errno == EINTR) continue;
Fail("write failed", path);
}
in = in.subspan(static_cast<std::size_t>(put));
offset += static_cast<std::uint64_t>(put);
}
}
void File::SyncData() {
if (::fdatasync(descriptor) == -1) Fail("fdatasync failed", path);
}
void File::Sync() {
if (::fsync(descriptor) == -1) Fail("fsync failed", path);
}
void File::Close() noexcept {
if (descriptor != -1) {
::close(descriptor);
descriptor = -1;
}
}
// --- MappedRegion -----------------------------------------------------------
MappedRegion::MappedRegion(MappedRegion&& other) noexcept : base(other.base), reserved(other.reserved), mapped(other.mapped), dontDump(other.dontDump) {
other.base = nullptr;
other.reserved = 0;
other.mapped = 0;
}
MappedRegion& MappedRegion::operator=(MappedRegion&& other) noexcept {
if (this != &other) {
Release();
base = other.base;
reserved = other.reserved;
mapped = other.mapped;
dontDump = other.dontDump;
other.base = nullptr;
other.reserved = 0;
other.mapped = 0;
}
return *this;
}
MappedRegion::~MappedRegion() {
Release();
}
void MappedRegion::Reserve(std::uint64_t bytes) {
Release();
// PROT_NONE + MAP_NORESERVE costs no memory and no commit charge: it only
// claims the addresses, so later MAP_FIXED calls cannot collide with
// anything else the process maps.
void* address = ::mmap(nullptr, bytes, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
if (address == MAP_FAILED) throw IOError(std::format("cannot reserve {} bytes of address space", bytes), Errno(), fs::path());
base = static_cast<std::byte*>(address);
reserved = bytes;
mapped = 0;
}
void MappedRegion::Map(const File& file, std::uint64_t fileOffset, std::uint64_t length, bool writable) {
if (mapped + length > reserved) {
throw IOError(std::format("table needs more than the {} bytes of address space reserved per region; reopen with a larger OpenOptions::reserveBytes", reserved), std::make_error_code(std::errc::not_enough_memory), file.Path());
}
void* address = ::mmap(base + mapped, length, writable ? PROT_READ | PROT_WRITE : PROT_READ, MAP_SHARED | MAP_FIXED, file.Descriptor(), static_cast<off_t>(fileOffset));
if (address == MAP_FAILED) Fail("cannot map", file.Path());
if (dontDump) ::madvise(address, length, MADV_DONTDUMP);
mapped += length;
}
// MADV_DONTDUMP only. MADV_DONTFORK was considered and rejected: it turns any
// touch of an inherited table in a forked child into SIGSEGV. The mapping is
// MAP_SHARED, so a child reads consistently; writes are refused by the
// owner-process check in TableFile.
void MappedRegion::ExcludeFromCoreDumps() {
dontDump = true;
if (mapped != 0) ::madvise(base, mapped, MADV_DONTDUMP);
}
void MappedRegion::WillNeed(std::uint64_t length) {
if (length != 0) ::madvise(base, std::min(length, mapped), MADV_WILLNEED);
}
void MappedRegion::Release() noexcept {
if (base != nullptr) {
::munmap(base, reserved);
base = nullptr;
reserved = 0;
mapped = 0;
}
}
// --- MappedPages ------------------------------------------------------------
MappedPages::MappedPages(MappedPages&& other) noexcept : data(other.data), length(other.length) {
other.data = nullptr;
other.length = 0;
}
MappedPages& MappedPages::operator=(MappedPages&& other) noexcept {
if (this != &other) {
Release();
data = other.data;
length = other.length;
other.data = nullptr;
other.length = 0;
}
return *this;
}
MappedPages::~MappedPages() {
Release();
}
void MappedPages::Map(const File& file, std::uint64_t fileOffset, std::uint64_t mapLength, bool writable) {
Release();
void* address = ::mmap(nullptr, mapLength, writable ? PROT_READ | PROT_WRITE : PROT_READ, MAP_SHARED, file.Descriptor(), static_cast<off_t>(fileOffset));
if (address == MAP_FAILED) Fail("cannot map header", file.Path());
data = static_cast<std::byte*>(address);
length = mapLength;
}
void MappedPages::Sync() {
if (data != nullptr && ::msync(data, length, MS_SYNC) == -1) throw IOError("msync failed", Errno(), fs::path());
}
void MappedPages::Release() noexcept {
if (data != nullptr) {
::munmap(data, length);
data = nullptr;
length = 0;
}
}
// --- Os ---------------------------------------------------------------------
void Os::CreateDirectories(const fs::path& directory, fs::perms permissions) {
if (directory.empty()) return;
fs::path current;
for (const fs::path& part : directory) {
current /= part;
if (current == directory.root_path() || current.empty()) continue;
if (::mkdir(current.c_str(), static_cast<mode_t>(permissions)) == -1 && errno != EEXIST) Fail("cannot create directory", current);
}
}
void Os::Rename(const fs::path& from, const fs::path& to) {
if (::rename(from.c_str(), to.c_str()) == -1) Fail(std::format("cannot rename to {}", to.string()), from);
}
bool Os::RenameNoReplace(const fs::path& from, const fs::path& to) {
if (::renameat2(AT_FDCWD, from.c_str(), AT_FDCWD, to.c_str(), RENAME_NOREPLACE) == 0) return true;
if (errno == EEXIST) return false;
Fail(std::format("cannot rename to {}", to.string()), from);
}
void Os::HardLink(const fs::path& from, const fs::path& to) {
if (::link(from.c_str(), to.c_str()) == -1) Fail(std::format("cannot link to {}", to.string()), from);
}
bool Os::Remove(const fs::path& path) noexcept {
return ::unlink(path.c_str()) == 0;
}
void Os::SyncDirectory(const fs::path& directory) {
const fs::path dir = directory.empty() ? fs::path(".") : directory;
const std::int32_t fd = ::open(dir.c_str(), O_RDONLY | O_DIRECTORY | O_CLOEXEC);
if (fd == -1) Fail("cannot open directory", dir);
const std::int32_t result = ::fsync(fd);
::close(fd);
if (result == -1) Fail("fsync of directory failed", dir);
}
void Os::RandomBytes(std::span<std::byte> out) {
std::random_device device;
for (std::size_t i = 0; i < out.size(); i += 4) {
const std::uint32_t value = device();
std::memcpy(out.data() + i, &value, std::min<std::size_t>(4, out.size() - i));
}
}
std::uint64_t Os::UnixNanos() noexcept {
return static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::system_clock::now().time_since_epoch()).count());
}
std::uint64_t Os::PageSize() noexcept {
const long size = ::sysconf(_SC_PAGESIZE);
return size > 0 ? static_cast<std::uint64_t>(size) : 4096;
}
std::int64_t Os::ProcessId() noexcept {
return static_cast<std::int64_t>(::getpid());
}

615
implementations/Crafter.DB-Storage.cpp Normal file
View file

@ -0,0 +1,615 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
module Crafter.DB:Storage_impl;
import :Storage;
import :Error;
import :Format;
import :Formatter;
import :File;
import std;
using namespace Crafter::DB;
namespace fs = std::filesystem;
namespace {
constexpr std::uint64_t maxExtentBytes = std::uint64_t(1) << 30;
constexpr std::uint64_t willNeedLimit = std::uint64_t(256) << 20;
constexpr std::uint32_t regionRows = 0;
constexpr std::uint32_t regionHeap = 1;
constexpr std::uint64_t RoundUp(std::uint64_t bytes, std::uint64_t granule) noexcept {
return (bytes + granule - 1) / granule * granule;
}
// Extents are aligned to the larger of the format page and the running
// kernel's page size, so they can be mmapped on 16K and 64K kernels too.
std::uint64_t Granule() noexcept {
return std::max<std::uint64_t>(pageSize, Os::PageSize());
}
std::span<const std::byte> Bytes(const auto& object) noexcept {
return std::as_bytes(std::span(&object, 1));
}
void FillCommit(CommitRecord& record, std::uint64_t seq, std::uint64_t rowCount, std::uint64_t liveCount, std::uint64_t heapUsed, std::uint64_t heapGarbage, const std::array<std::uint32_t, regionSlots>& extentCount, std::uint64_t fileLength, std::uint64_t flags) noexcept {
record = CommitRecord{};
record.commitSeq = seq;
record.rowCount = rowCount;
record.liveCount = liveCount;
record.heapUsed = heapUsed;
record.heapGarbage = heapGarbage;
record.extentCount = extentCount;
record.fileLength = fileLength;
record.modifiedUnixNanos = Os::UnixNanos();
record.flags = flags;
record.crc32c = CommitCrc(record);
}
// Picks the newest valid commit slot; nullopt when both are damaged.
std::optional<std::size_t> ChooseCommit(const FileHeader& header) noexcept {
const bool valid0 = CommitValid(header.commits[0]);
const bool valid1 = CommitValid(header.commits[1]);
if (valid0 && valid1) return header.commits[0].commitSeq >= header.commits[1].commitSeq ? 0 : 1;
if (valid0) return 0;
if (valid1) return 1;
return std::nullopt;
}
void ValidateHeader(const FileHeader& header, std::uint64_t fileSize, const fs::path& path) {
if (header.magic != fileMagic) throw CorruptError(std::format("{}: not a Crafter.DB table (bad magic)", path.string()));
if (header.formatVersion != formatVersion) throw CorruptError(std::format("{}: file format {} is not supported by this build (format {})", path.string(), header.formatVersion, formatVersion));
if ((header.formatFlags & formatFlagLittleEndian) == 0) throw CorruptError(std::format("{}: unsupported byte order", path.string()));
if (header.pageSize < pageSize || header.pageSize % pageSize != 0) throw CorruptError(std::format("{}: extent granule {} is invalid", path.string(), header.pageSize));
if (header.pageSize % Os::PageSize() != 0) throw IOError(std::format("table was created with {}-byte extents, which cannot be mapped on a system with {}-byte pages", header.pageSize, Os::PageSize()), std::make_error_code(std::errc::not_supported), path);
if (header.columnCount == 0 || (header.orderedColumn != noColumn && header.orderedColumn >= header.columnCount)) throw CorruptError(std::format("{}: column count or ordered column is invalid", path.string()));
if (header.headerPages < 2 || std::uint64_t(header.headerPages) * pageSize > fileSize) throw CorruptError(std::format("{}: header page count {} is invalid", path.string(), header.headerPages));
if (header.regionCount == 0 || header.regionCount > regionSlots) throw CorruptError(std::format("{}: region count {} is invalid", path.string(), header.regionCount));
if (header.rowSize == 0 || header.rowAlign == 0 || header.rowAlign > 16 || header.rowSize % header.rowAlign != 0) throw CorruptError(std::format("{}: row size {} / alignment {} is invalid", path.string(), header.rowSize, header.rowAlign));
if (std::uint64_t(header.schemaBlobLength) + pageSize > std::uint64_t(header.headerPages) * pageSize) throw CorruptError(std::format("{}: schema blob does not fit the header pages", path.string()));
}
// Reads header, commit and schema blob; shared by Inspect and open. The
// header and the first schema page are fetched in one pread, which is the
// whole description for every table with fewer than ~170 columns.
void ReadDescription(const File& file, std::uint64_t size, FileHeader& header, std::size_t& chosen, SchemaInfo& schema) {
if (size < 2 * pageSize) throw CorruptError(std::format("{}: file is smaller than the header", file.Path().string()));
std::vector<std::byte> pages(2 * pageSize);
file.ReadAt(0, pages);
std::memcpy(&header, pages.data(), sizeof(header));
ValidateHeader(header, size, file.Path());
const std::optional<std::size_t> slot = ChooseCommit(header);
if (!slot) throw CorruptError(std::format("{}: both commit records are damaged", file.Path().string()));
chosen = *slot;
std::span<const std::byte> blob;
std::vector<std::byte> rest;
if (header.schemaBlobLength <= pageSize) {
blob = std::span(pages).subspan(pageSize, header.schemaBlobLength);
} else {
rest.resize(header.schemaBlobLength);
file.ReadAt(pageSize, rest);
blob = rest;
}
if (!SchemaInfo::Parse(blob, header.rowAlign, schema)) throw CorruptError(std::format("{}: schema blob is malformed", file.Path().string()));
if (schema.rowSize != header.rowSize || schema.columns.size() != header.columnCount) throw CorruptError(std::format("{}: schema blob disagrees with the header", file.Path().string()));
}
// Reads a scalar column as the variant alternative its kind implies.
std::optional<ScalarValue> ReadScalar(const std::byte* row, const ColumnDescriptor& column) noexcept {
const std::byte* at = row + column.offset;
switch (column.kind) {
case Kind::Flags: case Kind::Bool: case Kind::UInt8: { std::uint8_t v; std::memcpy(&v, at, 1); return ScalarValue(std::uint64_t(v)); }
case Kind::UInt16: { std::uint16_t v; std::memcpy(&v, at, 2); return ScalarValue(std::uint64_t(v)); }
case Kind::UInt32: { std::uint32_t v; std::memcpy(&v, at, 4); return ScalarValue(std::uint64_t(v)); }
case Kind::UInt64: { std::uint64_t v; std::memcpy(&v, at, 8); return ScalarValue(v); }
case Kind::Int8: { std::int8_t v; std::memcpy(&v, at, 1); return ScalarValue(std::int64_t(v)); }
case Kind::Int16: { std::int16_t v; std::memcpy(&v, at, 2); return ScalarValue(std::int64_t(v)); }
case Kind::Int32: { std::int32_t v; std::memcpy(&v, at, 4); return ScalarValue(std::int64_t(v)); }
case Kind::Int64: case Kind::TimePoint: case Kind::Duration: {
if (column.size == 8) { std::int64_t v; std::memcpy(&v, at, 8); return ScalarValue(v); }
if (column.size == 4) { std::int32_t v; std::memcpy(&v, at, 4); return ScalarValue(std::int64_t(v)); }
return std::nullopt;
}
case Kind::Float32: { float v; std::memcpy(&v, at, 4); return ScalarValue(double(v)); }
case Kind::Float64: { double v; std::memcpy(&v, at, 8); return ScalarValue(v); }
case Kind::String: case Kind::Blob: case Kind::FixedBytes: return std::nullopt;
}
return std::nullopt;
}
// cmp_less that also accepts doubles (std::cmp_less is integers only).
template<class A, class B>
bool CmpLessPartial(const A& x, const B& y) noexcept {
if constexpr (std::is_floating_point_v<A> || std::is_floating_point_v<B>) return static_cast<double>(x) < static_cast<double>(y);
else return std::cmp_less(x, y);
}
bool ScalarLess(const ScalarValue& a, const ScalarValue& b) noexcept {
return std::visit([](const auto& x, const auto& y) { return CmpLessPartial(x, y); }, a, b);
}
std::uint32_t VarAlign(const ColumnDescriptor& column) noexcept {
return column.kind == Kind::Blob && column.paramNum != 0 ? column.paramNum : 1;
}
}
// --- creation ---------------------------------------------------------------
fs::path TableFile::TempPath(const fs::path& path) {
std::array<std::byte, 6> random{};
Os::RandomBytes(random);
std::string suffix;
for (const std::byte b : random) suffix += std::format("{:02x}", static_cast<std::uint8_t>(b));
return fs::path(std::format("{}.tmp-{}", path.string(), suffix));
}
fs::path TableFile::BackupPath(const fs::path& path, std::uint32_t version) {
return fs::path(std::format("{}.v{}.bak", path.string(), version));
}
void TableFile::Replace(const fs::path& path, const fs::path& temp) {
Os::Rename(temp, path);
Os::SyncDirectory(path.parent_path());
}
void TableFile::Create(const fs::path& path, const CreateParams& params, const OpenOptions& options) {
const std::vector<std::byte> blob = params.schema.Serialize();
const std::uint64_t granule = Granule();
FileHeader header{};
header.magic = fileMagic;
header.formatVersion = formatVersion;
header.headerPages = static_cast<std::uint16_t>(1 + RoundUp(blob.size(), pageSize) / pageSize);
header.formatFlags = formatFlagLittleEndian;
header.pageSize = static_cast<std::uint32_t>(granule);
header.rowSize = params.schema.rowSize;
header.rowAlign = params.schema.rowAlign;
header.columnCount = static_cast<std::uint16_t>(params.schema.columns.size());
header.orderedColumn = params.orderedColumn;
header.regionCount = regionCount;
header.schemaVersion = params.schemaVersion;
header.schemaFingerprint = params.fingerprint;
header.schemaBlobLength = static_cast<std::uint32_t>(blob.size());
header.minExtentBytes = minExtentBytes;
header.createdUnixNanos = Os::UnixNanos();
Os::RandomBytes(std::as_writable_bytes(std::span(header.fileUuid)));
header.migratedFromVersion = params.migratedFromVersion;
std::array<std::uint32_t, regionSlots> extentCount{};
const std::uint64_t headerBytes = std::uint64_t(header.headerPages) * pageSize;
std::uint64_t fileLength = RoundUp(headerBytes, granule);
header.extents[regionRows][0] = { fileLength, std::max<std::uint64_t>(RoundUp(minExtentBytes, granule), RoundUp(params.initialRowBytes, granule)) };
extentCount[regionRows] = 1;
fileLength += header.extents[regionRows][0].length;
if (params.schema.HasHeap()) {
header.extents[regionHeap][0] = { fileLength, std::max<std::uint64_t>(RoundUp(minExtentBytes, granule), RoundUp(params.initialHeapBytes, granule)) };
extentCount[regionHeap] = 1;
fileLength += header.extents[regionHeap][0].length;
}
FillCommit(header.commits[0], 1, 0, 0, 0, 0, extentCount, fileLength, commitFlagCleanClose);
if (options.createDirectories && !path.parent_path().empty()) Os::CreateDirectories(path.parent_path(), options.directoryPermissions);
const fs::path temp = TempPath(path);
try {
File file = File::CreateExclusive(temp, options.filePermissions);
file.WriteAt(0, Bytes(header));
file.WriteAt(pageSize, blob);
file.Allocate(headerBytes, fileLength - headerBytes);
file.Sync();
file.Close();
// Never replace: two processes creating the same table at once must
// end up sharing one file, not each writing to its own inode.
if (!Os::RenameNoReplace(temp, path)) {
Os::Remove(temp);
throw IOError("table already exists", std::make_error_code(std::errc::file_exists), path);
}
Os::SyncDirectory(path.parent_path());
} catch (...) {
Os::Remove(temp);
throw;
}
}
Inspection TableFile::Inspect(const fs::path& path) {
const File file = File::Open(path, false);
Inspection result;
std::size_t chosen = 0;
ReadDescription(file, file.Size(), result.header, chosen, result.schema);
result.commit = result.header.commits[chosen];
return result;
}
void TableFile::RemoveStaleTemps(const fs::path& path) noexcept {
try {
const std::string prefix = std::format("{}.tmp-", path.filename().string());
const fs::path directory = path.parent_path().empty() ? fs::path(".") : path.parent_path();
for (const fs::directory_entry& entry : fs::directory_iterator(directory)) {
if (!entry.path().filename().string().starts_with(prefix)) continue;
try {
File probe = File::Open(entry.path(), true);
probe.LockExclusive(); // held by a live writer -> LockedError -> keep it
Os::Remove(entry.path());
} catch (...) {
}
}
} catch (...) {
}
}
// --- open -------------------------------------------------------------------
TableFile::TableFile(const fs::path& path, const OpenOptions& openOptions) : options(openOptions), writable(!openOptions.readOnly), ownerProcess(Os::ProcessId()) {
file = File::Open(path, writable);
if (writable) file.LockExclusive();
else file.LockShared();
const std::uint64_t size = file.Size();
ReadDescription(file, size, header, activeSlot, schema);
current = header.commits[activeSlot];
if (current.fileLength > size) throw CorruptError(std::format("{}: committed length {} exceeds the file size {}", path.string(), current.fileLength, size));
if (current.fileLength < std::uint64_t(header.headerPages) * pageSize) throw CorruptError(std::format("{}: committed length is inside the header", path.string()));
for (std::size_t r = 0; r < regionSlots; r++) {
if (current.extentCount[r] > maxExtents || (r >= header.regionCount && current.extentCount[r] != 0)) throw CorruptError(std::format("{}: extent count of region {} is invalid", path.string(), r));
}
rows.Reserve(options.reserveBytes);
heapRegion.Reserve(options.reserveBytes);
const auto mapRegion = [&](MappedRegion& region, std::uint32_t index) {
for (std::uint32_t e = 0; e < current.extentCount[index]; e++) {
const Extent extent = header.extents[index][e];
if (extent.length == 0 || extent.length % header.pageSize != 0 || extent.fileOffset % header.pageSize != 0 || extent.fileOffset < std::uint64_t(header.headerPages) * pageSize || extent.length > current.fileLength || extent.fileOffset > current.fileLength - extent.length) {
throw CorruptError(std::format("{}: extent {} of region {} lies outside the committed file", path.string(), e, index));
}
region.Map(file, extent.fileOffset, extent.length, writable);
}
};
mapRegion(rows, regionRows);
mapRegion(heapRegion, regionHeap);
// One madvise pair per region now; extents added later are advised as
// they are mapped.
if (options.excludeFromCoreDumps) {
rows.ExcludeFromCoreDumps();
heapRegion.ExcludeFromCoreDumps();
}
if (current.rowCount > rows.Mapped() / header.rowSize) throw CorruptError(std::format("{}: {} committed rows do not fit the mapped row extents", path.string(), current.rowCount));
if (current.heapUsed > heapRegion.Mapped()) throw CorruptError(std::format("{}: committed heap length exceeds the mapped heap extents", path.string()));
extentCount = current.extentCount;
fileLength = current.fileLength;
publishedRows.store(current.rowCount, std::memory_order_relaxed);
publishedLive.store(current.liveCount, std::memory_order_relaxed);
publishedHeap.store(current.heapUsed, std::memory_order_relaxed);
heapGarbage = current.heapGarbage;
heap = Heap{ heapRegion.Base(), &publishedHeap };
writer.Attach(this, heapRegion.Base(), current.heapUsed, heapRegion.Mapped());
for (std::size_t c = 0; c < schema.columns.size(); c++) {
if (schema.columns[c].kind == Kind::String || schema.columns[c].kind == Kind::Blob) varColumns.push_back(c);
}
// An unclean close means tombstones (and orphaned heap bytes) may have
// reached the disk after the last commit; recount so the "no tombstones"
// scan fast path and the garbage figure stay honest.
if ((current.flags & commitFlagCleanClose) == 0) RecountLive();
tombstones.store(publishedLive.load(std::memory_order_relaxed) != publishedRows.load(std::memory_order_relaxed), std::memory_order_relaxed);
if (options.prefetch) {
const std::uint64_t rowBytes = current.rowCount * header.rowSize;
if (rowBytes < willNeedLimit) rows.WillNeed(rowBytes);
if (current.heapUsed < willNeedLimit) heapRegion.WillNeed(current.heapUsed);
}
if (writable) {
// Uncommitted growth from an earlier crash: drop it so extents we add
// start exactly at the committed end.
if (size > current.fileLength) file.Truncate(current.fileLength);
headerPages.Map(file, 0, pageSize, true);
// Mark the file open: a crash from here on must recount on next open.
WriteCommit(false);
RemoveStaleTemps(path);
}
}
TableFile::~TableFile() {
try {
Close();
} catch (...) {
}
}
void TableFile::Close() {
if (closed) return;
closed = true;
// A forked child inherited this object; its counters describe the parent's
// state at fork time and must never reach the file.
if (writable && file.IsOpen() && OwnedByThisProcess()) Flush(true);
headerPages.Release();
rows.Release();
heapRegion.Release();
file.Close();
}
bool TableFile::OwnedByThisProcess() const noexcept {
return Os::ProcessId() == ownerProcess;
}
void TableFile::RequireWritable() const {
if (!writable) throw ReadOnlyError(std::format("{}: table is open read-only", file.Path().string()));
if (closed) throw Error(std::format("{}: table is closed", file.Path().string()));
if (!OwnedByThisProcess()) throw Error(std::format("{}: table was opened by another process (inherited through fork)", file.Path().string()));
}
TableStats TableFile::Stats() const noexcept {
return { SlotCount(), LiveCount(), HeapUsed(), heapGarbage, fileLength, current.commitSeq };
}
bool TableFile::Deleted(const std::byte* row) noexcept {
return (std::atomic_ref<std::uint8_t>(*const_cast<std::uint8_t*>(reinterpret_cast<const std::uint8_t*>(row))).load(std::memory_order_relaxed) & rowFlagDeleted) != 0;
}
void TableFile::RecountLive() noexcept {
const std::uint64_t count = publishedRows.load(std::memory_order_relaxed);
std::uint64_t live = 0;
std::uint64_t referenced = 0;
for (std::uint64_t i = 0; i < count; i++) {
const std::byte* row = Slot(i);
if (Deleted(row)) continue;
live++;
referenced += HeapBytesOf(row);
}
publishedLive.store(live, std::memory_order_relaxed);
// Everything in the heap that no live row references is garbage: this
// also covers bytes orphaned by Update/Set before the crash.
const std::uint64_t used = publishedHeap.load(std::memory_order_relaxed);
heapGarbage = used > referenced ? used - referenced : 0;
}
// --- growth -----------------------------------------------------------------
void TableFile::AddExtent(std::uint32_t region, std::uint64_t minimumBytes) {
RequireWritable();
const std::uint32_t count = extentCount[region];
if (count >= maxExtents) throw IOError("table has reached its maximum number of extents; Compact() it", std::make_error_code(std::errc::file_too_large), file.Path());
const std::uint64_t granule = header.pageSize;
const std::uint64_t previous = count == 0 ? 0 : header.extents[region][count - 1].length;
const std::uint64_t length = std::max({ RoundUp(minExtentBytes, granule), std::min(previous * 2, maxExtentBytes), RoundUp(minimumBytes, granule) });
const Extent extent{ fileLength, length };
file.Allocate(extent.fileOffset, extent.length);
MappedRegion& target = region == regionRows ? rows : heapRegion;
target.Map(file, extent.fileOffset, extent.length, true);
header.extents[region][count] = extent;
std::memcpy(headerPages.Data() + __builtin_offsetof(FileHeader, extents) + (region * maxExtents + count) * sizeof(Extent), &extent, sizeof(extent));
extentCount[region] = count + 1;
fileLength = extent.fileOffset + extent.length;
}
void TableFile::EnsureHeapCapacity(std::uint64_t bytes) {
while (heapRegion.Mapped() < bytes) AddExtent(regionHeap, bytes - heapRegion.Mapped());
writer.SetCapacity(heapRegion.Mapped());
}
void TableFile::Arena::Grow(std::uint64_t neededCapacity) {
owner->EnsureHeapCapacity(neededCapacity);
}
// --- writes -----------------------------------------------------------------
std::byte* TableFile::BeginAppend(std::uint64_t count) {
RequireWritable();
const std::uint64_t slots = publishedRows.load(std::memory_order_relaxed);
const std::uint64_t needed = (slots + count) * header.rowSize;
while (rows.Mapped() < needed) AddExtent(regionRows, needed - rows.Mapped());
return rows.Base() + slots * header.rowSize;
}
void TableFile::CommitAppend(std::uint64_t count) noexcept {
publishedHeap.store(writer.Cursor(), std::memory_order_release);
publishedLive.fetch_add(count, std::memory_order_relaxed);
publishedRows.fetch_add(count, std::memory_order_release);
}
void TableFile::AbortAppend() noexcept {
writer.Rewind(publishedHeap.load(std::memory_order_relaxed));
}
std::byte* TableFile::MutableSlot(std::uint64_t index) {
RequireWritable();
return rows.Base() + index * header.rowSize;
}
bool TableFile::MarkDeleted(std::uint64_t index) {
RequireWritable();
std::byte* row = rows.Base() + index * header.rowSize;
// The flag goes up before the byte, so a scan that starts after this call
// checks every row.
tombstones.store(true, std::memory_order_release);
const std::uint8_t previous = std::atomic_ref<std::uint8_t>(*reinterpret_cast<std::uint8_t*>(row)).fetch_or(rowFlagDeleted, std::memory_order_relaxed);
if ((previous & rowFlagDeleted) != 0) return false;
publishedLive.fetch_sub(1, std::memory_order_relaxed);
heapGarbage += HeapBytesOf(row);
return true;
}
std::uint64_t TableFile::HeapBytesOf(const std::byte* row) const noexcept {
std::uint64_t bytes = 0;
for (const std::size_t c : varColumns) {
const ColumnDescriptor& column = schema.columns[c];
Var var;
std::memcpy(&var, row + column.offset, sizeof(var));
if (!VarInline(var.length, VarAlign(column))) bytes += var.length;
}
return bytes;
}
void TableFile::WriteCommit(bool cleanClose) {
const std::size_t slot = activeSlot == 0 ? 1 : 0;
CommitRecord record;
FillCommit(record, current.commitSeq + 1, publishedRows.load(std::memory_order_relaxed), publishedLive.load(std::memory_order_relaxed), publishedHeap.load(std::memory_order_relaxed), heapGarbage, extentCount, fileLength, cleanClose ? commitFlagCleanClose : 0);
std::memcpy(headerPages.Data() + __builtin_offsetof(FileHeader, commits) + slot * sizeof(CommitRecord), &record, sizeof(record));
headerPages.Sync();
current = record;
header.commits[slot] = record;
activeSlot = slot;
}
void TableFile::Flush(bool cleanClose) {
if (!writable || (closed && !file.IsOpen())) return;
if (!OwnedByThisProcess()) throw Error(std::format("{}: table was opened by another process (inherited through fork)", file.Path().string()));
// Data first: the commit record must never describe rows or heap bytes
// that are not yet on disk.
file.SyncData();
WriteCommit(cleanClose);
}
// --- compaction -------------------------------------------------------------
void TableFile::CompactInto(TableFile& target) const {
target.RequireWritable();
if (target.RowSize() != RowSize() || target.Schema().columns.size() != schema.columns.size()) throw Error("CompactInto: target table has a different schema");
const std::uint64_t slots = SlotCount();
const std::uint64_t live = LiveCount();
if (live == 0) return;
std::byte* out = target.BeginAppend(live);
HeapWriter& arena = target.Writer();
std::uint64_t written = 0;
for (std::uint64_t i = 0; i < slots && written < live; i++) {
const std::byte* row = Slot(i);
if (Deleted(row)) continue;
std::memcpy(out, row, header.rowSize);
for (const std::size_t c : varColumns) {
const ColumnDescriptor& column = schema.columns[c];
if ((column.flags & columnFlagOptional) != 0 && column.presenceOffset != noColumn) {
// The payload of a disengaged optional carries no value: write
// an empty handle rather than carrying stale bytes forward.
std::uint8_t present;
std::memcpy(&present, out + column.presenceOffset, 1);
if (present == 0) {
const Var empty{};
std::memcpy(out + column.offset, &empty, sizeof(empty));
continue;
}
}
Var var;
std::memcpy(&var, out + column.offset, sizeof(var));
if (VarInline(var.length, VarAlign(column))) continue;
const std::span<const std::byte> bytes = heap.Read(var.tailOrOffset, var.length);
var.tailOrOffset = arena.Write(bytes, VarAlign(column));
std::memcpy(out + column.offset, &var, sizeof(var));
}
out += header.rowSize;
written++;
}
target.CommitAppend(written);
}
// --- verification -----------------------------------------------------------
VerifyResult TableFile::Verify() const {
VerifyResult result;
const std::uint64_t slots = SlotCount();
const std::uint64_t heapLimit = HeapUsed();
const std::uint16_t ordered = header.orderedColumn;
std::optional<ScalarValue> previousKey;
std::uint64_t live = 0;
for (std::uint64_t i = 0; i < slots; i++) {
const std::byte* row = Slot(i);
std::uint8_t flags;
std::memcpy(&flags, row, 1);
if ((flags & ~rowFlagDeleted) != 0) result.problems.push_back(std::format("row {}: unknown flag bits {:#x}", i, flags));
if ((flags & rowFlagDeleted) == 0) live++;
for (std::size_t c = 1; c < schema.columns.size(); c++) {
const ColumnDescriptor& column = schema.columns[c];
if ((column.flags & columnFlagPresence) != 0) {
std::uint8_t present;
std::memcpy(&present, row + column.offset, 1);
if (present > 1) result.problems.push_back(std::format("row {}: presence byte of '{}' is {}", i, schema.Name(c), present));
}
if (column.kind == Kind::String || column.kind == Kind::Blob) {
Var var;
std::memcpy(&var, row + column.offset, sizeof(var));
if (!VarInline(var.length, VarAlign(column)) && (var.tailOrOffset > heapLimit || var.length > heapLimit - var.tailOrOffset)) {
result.problems.push_back(std::format("row {}: '{}' points at heap [{}, +{}) beyond the committed {} bytes", i, schema.Name(c), var.tailOrOffset, var.length, heapLimit));
}
}
}
if (ordered != noColumn && ordered < schema.columns.size()) {
const std::optional<ScalarValue> key = ReadScalar(row, schema.columns[ordered]);
if (key && previousKey && ScalarLess(*key, *previousKey)) result.problems.push_back(std::format("row {}: '{}' is out of order", i, schema.Name(ordered)));
if (key) previousKey = key;
}
if (result.problems.size() > 1000) {
result.problems.push_back("... further problems omitted");
return result;
}
}
if (live != LiveCount()) result.problems.push_back(std::format("live count {} does not match the {} rows without a tombstone", LiveCount(), live));
return result;
}
// --- DynamicRow -------------------------------------------------------------
bool DynamicRow::Deleted() const noexcept {
return TableFile::Deleted(row);
}
std::optional<std::size_t> DynamicRow::Find(std::string_view name) const noexcept {
for (std::size_t i = 0; i < schema->columns.size(); i++) {
if (schema->Name(i) == name) return i;
}
return std::nullopt;
}
bool DynamicRow::Present(std::size_t index) const noexcept {
const ColumnDescriptor& column = schema->columns[index];
if ((column.flags & columnFlagOptional) == 0 || column.presenceOffset == noColumn) return true;
std::uint8_t present;
std::memcpy(&present, row + column.presenceOffset, 1);
return present != 0;
}
std::optional<ScalarValue> DynamicRow::Scalar(std::size_t index) const noexcept {
return ReadScalar(row, schema->columns[index]);
}
std::optional<std::span<const std::byte>> DynamicRow::Bytes(std::size_t index) const {
const ColumnDescriptor& column = schema->columns[index];
if (column.kind == Kind::FixedBytes) return std::span<const std::byte>(row + column.offset, column.size);
if (column.kind != Kind::String && column.kind != Kind::Blob) return std::nullopt;
Var var;
std::memcpy(&var, row + column.offset, sizeof(var));
if (VarInline(var.length, VarAlign(column))) return std::span<const std::byte>(row + column.offset + 4, var.length);
return heap->Read(var.tailOrOffset, var.length);
}
std::optional<std::string_view> DynamicRow::Text(std::size_t index) const {
const ColumnDescriptor& column = schema->columns[index];
if (column.kind != Kind::String && column.kind != Kind::FixedBytes) return std::nullopt;
const std::optional<std::span<const std::byte>> bytes = Bytes(index);
if (!bytes) return std::nullopt;
std::string_view text(reinterpret_cast<const char*>(bytes->data()), bytes->size());
if (column.kind == Kind::FixedBytes) text = text.substr(0, text.find('\0'));
return text;
}
std::string DynamicRow::Describe() const {
std::string out;
for (std::size_t i = 1; i < schema->columns.size(); i++) {
const ColumnDescriptor& column = schema->columns[i];
if ((column.flags & columnFlagPresence) != 0) continue;
if (!out.empty()) out += ", ";
out += schema->Name(i);
out += '=';
if (!Present(i)) {
out += "null";
continue;
}
if (const std::optional<std::string_view> text = Text(i)) {
out += std::format("\"{}\"", *text);
} else if (const std::optional<ScalarValue> scalar = Scalar(i)) {
std::visit([&](const auto& v) { out += std::format("{}", v); }, *scalar);
} else if (const std::optional<std::span<const std::byte>> bytes = Bytes(i)) {
out += std::format("<{} bytes>", bytes->size());
} else {
out += '?';
}
}
return out;
}

335
interfaces/Crafter.DB-Codec.cppm Normal file
View file

@ -0,0 +1,335 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Typed encode/decode against a Layout<T>. Every function here is a template
// over (table type, first leaf index, member type), so column offsets are
// compile-time constants and a scalar read is one load. Leaf indices follow
// declaration order, depth first, exactly as Schema enumerates them.
export module Crafter.DB:Codec;
import std;
import :Reflect;
import :Format;
import :Formatter;
import :Schema;
import :Storage;
import :Error;
export namespace Crafter::DB {
// What a Row hands out for a member of type M: the formatter's View for
// variable-length members, an optional of that for std::optional, and the
// value itself for everything else.
template<class M>
consteval auto ViewTypeOf() {
constexpr Shape shape = ShapeOf<M>();
if constexpr (shape == Shape::Var) return std::type_identity<typename Formatter<M>::View>{};
else if constexpr (shape == Shape::Optional) return std::type_identity<std::optional<typename decltype(ViewTypeOf<typename OptionalTraits<M>::Type>())::type>>{};
else return std::type_identity<M>{};
}
template<class M>
using View = typename decltype(ViewTypeOf<M>())::type;
// Zeroes the columns of leaves [First, First + Count): the payload of a
// disengaged std::optional must not keep stale bytes (or stale heap
// handles that Compact would then preserve).
template<class T, std::size_t First, std::size_t Count>
void ZeroLeaves(std::byte* row) noexcept {
[&]<std::size_t... I>(std::index_sequence<I...>) {
(std::memset(row + Layout<T>::Offset(First + I), 0, Layout<T>::columns[Layout<T>::Column(First + I)].size), ...);
}(std::make_index_sequence<Count>{});
}
// Copies the columns of leaves [First, First + Count) from one row image
// to another, leaving every other byte (the flags byte above all) alone.
template<class T, std::size_t First, std::size_t Count>
void CopyLeaves(std::byte* destination, const std::byte* source) noexcept {
[&]<std::size_t... I>(std::index_sequence<I...>) {
(std::memcpy(destination + Layout<T>::Offset(First + I), source + Layout<T>::Offset(First + I), Layout<T>::columns[Layout<T>::Column(First + I)].size), ...);
}(std::make_index_sequence<Count>{});
}
// Values are converted implicitly, never static_cast: an int does not
// silently become an enum, and swapped Emplace arguments do not compile.
template<class T, std::size_t Leaf, class M, class V>
void EncodeInto(std::byte* row, HeapWriter& heap, const V& value) {
constexpr Shape shape = ShapeOf<M>();
constexpr std::uint16_t offset = Layout<T>::Offset(Leaf);
if constexpr (shape == Shape::Scalar) {
const M scalar = value;
std::memcpy(row + offset, &scalar, sizeof(M));
} else if constexpr (shape == Shape::FixedBytes) {
if constexpr (FixedStringTraits<M>::value) {
const M fixed(value);
std::memcpy(row + offset, fixed.chars.data(), fixed.chars.size());
} else {
const M& array = value;
std::memcpy(row + offset, array.data(), sizeof(M));
}
} else if constexpr (shape == Shape::ScalarArray) {
using E = typename StdArrayTraits<M>::Element;
[&]<std::size_t... I>(std::index_sequence<I...>) {
(EncodeInto<T, Leaf + I, E>(row, heap, value[I]), ...);
}(std::make_index_sequence<StdArrayTraits<M>::size>{});
} else if constexpr (shape == Shape::Optional) {
using P = typename OptionalTraits<M>::Type;
const std::uint8_t present = value.has_value() ? 1 : 0;
std::memcpy(row + offset, &present, 1);
if (value.has_value()) EncodeInto<T, Leaf + 1, P>(row, heap, *value);
else ZeroLeaves<T, Leaf + 1, LeafCount<P>()>(row);
} else if constexpr (shape == Shape::Var) {
const Var var = Formatter<M>::Encode(value, heap);
std::memcpy(row + offset, &var, sizeof(var));
} else if constexpr (shape == Shape::Fixed) {
using S = typename Formatter<M>::Stored;
if constexpr (std::is_same_v<V, M>) {
const S stored = Formatter<M>::Encode(value);
EncodeInto<T, Leaf, S>(row, heap, stored);
} else {
const M converted = value;
const S stored = Formatter<M>::Encode(converted);
EncodeInto<T, Leaf, S>(row, heap, stored);
}
} else if constexpr (shape == Shape::Aggregate) {
if constexpr (std::is_same_v<V, M>) {
[&]<std::size_t... I>(std::index_sequence<I...>) {
(EncodeInto<T, Leaf + LeafStart<M, I>(), MemberType<I, M>>(row, heap, Member<I>(value)), ...);
}(std::make_index_sequence<memberCount<M>>{});
} else {
const M converted = value;
EncodeInto<T, Leaf, M>(row, heap, converted);
}
} else {
static_assert(shape == Shape::Scalar, "unsupported member shape reached the encoder; CheckTable should have rejected it");
}
}
template<class T, std::size_t Leaf, class M>
M DecodeValue(const std::byte* row, const Heap& heap);
template<class T, std::size_t Leaf, class M>
View<M> DecodeView(const std::byte* row, const Heap& heap) {
constexpr Shape shape = ShapeOf<M>();
constexpr std::uint16_t offset = Layout<T>::Offset(Leaf);
if constexpr (shape == Shape::Var) {
return Formatter<M>::Decode(row + offset, heap);
} else if constexpr (shape == Shape::Optional) {
using P = typename OptionalTraits<M>::Type;
std::uint8_t present;
std::memcpy(&present, row + offset, 1);
if (present == 0) return std::nullopt;
return View<M>(DecodeView<T, Leaf + 1, P>(row, heap));
} else {
return DecodeValue<T, Leaf, M>(row, heap);
}
}
template<class T, std::size_t Leaf, class M>
M DecodeValue(const std::byte* row, const Heap& heap) {
constexpr Shape shape = ShapeOf<M>();
constexpr std::uint16_t offset = Layout<T>::Offset(Leaf);
if constexpr (shape == Shape::Scalar) {
M scalar;
std::memcpy(&scalar, row + offset, sizeof(M));
return scalar;
} else if constexpr (shape == Shape::FixedBytes) {
M fixed{};
if constexpr (FixedStringTraits<M>::value) std::memcpy(fixed.chars.data(), row + offset, fixed.chars.size());
else std::memcpy(fixed.data(), row + offset, sizeof(M));
return fixed;
} else if constexpr (shape == Shape::ScalarArray) {
using E = typename StdArrayTraits<M>::Element;
M array{};
[&]<std::size_t... I>(std::index_sequence<I...>) {
((array[I] = DecodeValue<T, Leaf + I, E>(row, heap)), ...);
}(std::make_index_sequence<StdArrayTraits<M>::size>{});
return array;
} else if constexpr (shape == Shape::Optional) {
using P = typename OptionalTraits<M>::Type;
std::uint8_t present;
std::memcpy(&present, row + offset, 1);
if (present == 0) return std::nullopt;
return M(DecodeValue<T, Leaf + 1, P>(row, heap));
} else if constexpr (shape == Shape::Var) {
return MaterializeVar<M>(Formatter<M>::Decode(row + offset, heap));
} else if constexpr (shape == Shape::Fixed) {
using S = typename Formatter<M>::Stored;
return Formatter<M>::Decode(DecodeValue<T, Leaf, S>(row, heap));
} else if constexpr (shape == Shape::Aggregate) {
M aggregate{};
[&]<std::size_t... I>(std::index_sequence<I...>) {
((Member<I>(aggregate) = DecodeValue<T, Leaf + LeafStart<M, I>(), MemberType<I, M>>(row, heap)), ...);
}(std::make_index_sequence<memberCount<M>>{});
return aggregate;
} else {
static_assert(shape == Shape::Scalar, "unsupported member shape reached the decoder");
}
}
// --- row level -----------------------------------------------------------
template<Aggregate T, std::size_t I>
View<MemberType<I, T>> DecodeMember(const std::byte* row, const Heap& heap) {
return DecodeView<T, LeafStart<T, I>(), MemberType<I, T>>(row, heap);
}
template<Aggregate T, std::size_t I>
MemberType<I, T> DecodeMemberValue(const std::byte* row, const Heap& heap) {
return DecodeValue<T, LeafStart<T, I>(), MemberType<I, T>>(row, heap);
}
template<Aggregate T>
T DecodeRow(const std::byte* row, const Heap& heap) {
T value{};
[&]<std::size_t... I>(std::index_sequence<I...>) {
((Member<I>(value) = DecodeMemberValue<T, I>(row, heap)), ...);
}(std::make_index_sequence<memberCount<T>>{});
return value;
}
// Writes a live row (flags = 0) for `value` into `row`, which must be
// Layout<T>::rowSize bytes of writable memory.
template<Aggregate T>
void EncodeRow(const T& value, std::byte* row, HeapWriter& heap) {
std::memset(row, 0, Layout<T>::rowSize);
[&]<std::size_t... I>(std::index_sequence<I...>) {
(EncodeInto<T, LeafStart<T, I>(), MemberType<I, T>>(row, heap, Member<I>(value)), ...);
}(std::make_index_sequence<memberCount<T>>{});
}
// Whether an Emplace argument of type A is acceptable for a member of type
// M: what the formatter's Encode accepts for variable-length members, an
// implicit conversion for everything else.
template<class M, class A>
consteval bool EmplaceAccepts() {
constexpr Shape shape = ShapeOf<M>();
if constexpr (shape == Shape::Var) return requires(const A& a, HeapWriter& w) { Formatter<M>::Encode(a, w); };
else if constexpr (shape == Shape::FixedBytes && FixedStringTraits<M>::value) return std::is_constructible_v<M, const A&>;
else return std::is_convertible_v<const A&, M>;
}
template<Aggregate T, std::size_t I, class A>
consteval bool CheckEmplaceArgument() {
static_assert(EmplaceAccepts<MemberType<I, T>, A>(), Message({ "Crafter.DB: the Emplace argument for member '", MemberName<I, T>(), "' of ", TypeName<T>(), " has type ", TypeName<A>(), ", which does not convert to the member; arguments follow declaration order." }));
return true;
}
// One argument per member, each encoded straight into its columns.
template<Aggregate T, class... Args>
void EmplaceRow(std::byte* row, HeapWriter& heap, const Args&... args) {
static_assert(sizeof...(Args) == memberCount<T>, "Crafter.DB: Emplace takes exactly one argument per member of the table struct, in declaration order");
[&]<std::size_t... I>(std::index_sequence<I...>) {
(CheckEmplaceArgument<T, I, Args>(), ...);
}(std::make_index_sequence<memberCount<T>>{});
std::memset(row, 0, Layout<T>::rowSize);
[&]<std::size_t... I>(std::index_sequence<I...>) {
(EncodeInto<T, LeafStart<T, I>(), MemberType<I, T>>(row, heap, args), ...);
}(std::make_index_sequence<memberCount<T>>{});
}
// Encodes member I only (Set); the rest of the row is left alone.
template<Aggregate T, std::size_t I, class V>
void EncodeMember(std::byte* row, HeapWriter& heap, const V& value) {
EncodeInto<T, LeafStart<T, I>(), MemberType<I, T>>(row, heap, value);
}
// Out-of-line heap bytes referenced by member I's columns.
template<Aggregate T, std::size_t I>
std::uint64_t MemberHeapBytes(const std::byte* row) noexcept {
std::uint64_t bytes = 0;
constexpr std::size_t first = LeafStart<T, I>();
constexpr std::size_t count = LeafCount<MemberType<I, T>>();
for (std::size_t leaf = first; leaf < first + count; leaf++) {
const ColumnInfo& column = Layout<T>::columns[Layout<T>::Column(leaf)];
if (column.kind != Kind::String && column.kind != Kind::Blob) continue;
Var var;
std::memcpy(&var, row + column.offset, sizeof(var));
if (!VarInline(var.length, column.kind == Kind::Blob && column.paramNum != 0 ? column.paramNum : 1)) bytes += var.length;
}
return bytes;
}
// --- ordered column ------------------------------------------------------
template<class T>
using OrderedKey = MemberTypeOf<T::orderedBy>;
template<class T> requires HasOrderedBy<T>
OrderedKey<T> KeyOfSlot(const TableFile& file, std::uint64_t slot) {
return DecodeMemberValue<T, Layout<T>::orderedMember>(file.Slot(slot), file.HeapView());
}
template<class T> requires HasOrderedBy<T>
std::uint64_t LowerBoundSlot(const TableFile& file, const OrderedKey<T>& key, std::uint64_t first, std::uint64_t last) {
while (first < last) {
const std::uint64_t mid = first + (last - first) / 2;
if (KeyOfSlot<T>(file, mid) < key) first = mid + 1;
else last = mid;
}
return first;
}
template<class T> requires HasOrderedBy<T>
std::uint64_t UpperBoundSlot(const TableFile& file, const OrderedKey<T>& key, std::uint64_t first, std::uint64_t last) {
while (first < last) {
const std::uint64_t mid = first + (last - first) / 2;
if (key < KeyOfSlot<T>(file, mid)) last = mid;
else first = mid + 1;
}
return first;
}
// Throws OrderError when `key` would sit below the key of slot `before`
// (or above the key of slot `after`, when given).
template<class T> requires HasOrderedBy<T>
void CheckOrder(const TableFile& file, const OrderedKey<T>& key, std::optional<std::uint64_t> before, std::optional<std::uint64_t> after) {
if (before && key < KeyOfSlot<T>(file, *before)) {
throw OrderError(std::format("{}: value would break the order of '{}' (it sorts before the previous row)", file.Path().string(), MemberName<Layout<T>::orderedMember, T>()));
}
if (after && KeyOfSlot<T>(file, *after) < key) {
throw OrderError(std::format("{}: value would break the order of '{}' (it sorts after the next row)", file.Path().string(), MemberName<Layout<T>::orderedMember, T>()));
}
}
// Appends encoded rows. Checks the ordered column against the last slot
// and within the batch before anything is written.
template<Aggregate T, std::ranges::input_range R> requires std::convertible_to<std::ranges::range_reference_t<R>, const T&>
std::uint64_t AppendRows(TableFile& file, R&& values) {
file.RequireWritable();
const std::uint64_t first = file.SlotCount();
if constexpr (std::ranges::sized_range<R> && std::ranges::random_access_range<R>) {
const std::uint64_t count = std::ranges::size(values);
if (count == 0) return first;
if constexpr (HasOrderedBy<T>) {
std::optional<std::uint64_t> before;
if (first != 0) before = first - 1;
auto it = std::ranges::begin(values);
CheckOrder<T>(file, static_cast<const T&>(*it).*T::orderedBy, before, std::nullopt);
for (std::uint64_t i = 1; i < count; i++) {
if (static_cast<const T&>(it[i]).*T::orderedBy < static_cast<const T&>(it[i - 1]).*T::orderedBy) {
throw OrderError(std::format("{}: batch is not sorted by '{}'", file.Path().string(), MemberName<Layout<T>::orderedMember, T>()));
}
}
}
std::byte* out = file.BeginAppend(count);
HeapWriter& heap = file.Writer();
try {
for (const T& value : values) {
EncodeRow<T>(value, out, heap);
out += Layout<T>::rowSize;
}
} catch (...) {
file.AbortAppend();
throw;
}
file.CommitAppend(count);
return first;
} else {
// A range that cannot be sized is buffered first, so the order
// check runs over the whole batch before any row is written and a
// failing batch leaves the table untouched.
std::vector<T> buffered;
for (const T& value : values) buffered.push_back(value);
return AppendRows<T>(file, std::span<const T>(buffered));
}
}
}

65
interfaces/Crafter.DB-Error.cppm Normal file
View file

@ -0,0 +1,65 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
export module Crafter.DB:Error;
import std;
export namespace Crafter::DB {
// Root of every exception the library throws. Catch this to handle
// "the table is unusable" uniformly; catch the subclasses to react.
class Error : public std::runtime_error {
public:
using std::runtime_error::runtime_error;
};
// A system call failed. `code` carries errno, `path` the file involved.
class IOError : public Error {
public:
IOError(std::string_view what, std::error_code code, std::filesystem::path path)
: Error(std::format("{}: {} ({})", path.string(), what, code.message())), code(code), path(std::move(path)) {}
std::error_code code;
std::filesystem::path path;
};
// The file is not a Crafter.DB table or its header/commit records fail
// validation (bad magic, both commit CRCs wrong, extents outside the file).
class CorruptError : public Error {
public:
using Error::Error;
};
// The compiled struct and the file disagree: same version but a different
// fingerprint (struct changed without bumping `version`), a file newer
// than the code, or no `Previous` chain reaching the file's version.
class SchemaError : public Error {
public:
using Error::Error;
};
// A migration is needed but cannot run: the table was opened read-only or
// with `migrate = false`.
class MigrationError : public Error {
public:
using Error::Error;
};
// Another process (or another Table object in this process) holds the
// file lock in a conflicting mode.
class LockedError : public Error {
public:
using Error::Error;
};
// A write would break the order of the `orderedBy` column.
class OrderError : public Error {
public:
using Error::Error;
};
// A write was attempted on a table opened with `readOnly = true`.
class ReadOnlyError : public Error {
public:
using Error::Error;
};
}

142
interfaces/Crafter.DB-File.cppm Normal file
View file

@ -0,0 +1,142 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Everything that talks to the operating system: file descriptors, advisory
// locks, preallocation, the reserved virtual ranges that keep row pointers
// stable while a table grows, durability calls and the atomic create/replace
// dance. Nothing here knows what a table is; the Storage partition does.
export module Crafter.DB:File;
import std;
import :Error;
export namespace Crafter::DB {
enum class OpenMode : std::uint8_t {
OpenOrCreate, // open the table, create it when missing (default)
OpenExisting, // IOError when missing
CreateNew, // IOError when present
};
enum class Durability : std::uint8_t {
OnFlush, // durable after Flush() and on destruction (default)
EveryAppend, // Flush() after every Append/Emplace/Update/Set/Erase
};
struct OpenOptions {
OpenMode mode = OpenMode::OpenOrCreate;
bool readOnly = false; // PROT_READ mapping, shared lock, no migration, no writes
bool createDirectories = true; // create missing parent directories with directoryPermissions
std::filesystem::perms filePermissions = std::filesystem::perms::owner_read | std::filesystem::perms::owner_write; // 0600
std::filesystem::perms directoryPermissions = std::filesystem::perms::owner_all; // 0700
Durability durability = Durability::OnFlush;
bool migrate = true; // run the Previous chain on an older file; false throws MigrationError instead
bool backupBeforeMigrate = true; // keep the pre-migration file as <file>.v<old>.bak
std::uint64_t reserveBytes = std::uint64_t(32) << 30; // virtual address space reserved per region
bool excludeFromCoreDumps = true; // MADV_DONTDUMP on the mappings
bool prefetch = false; // MADV_WILLNEED over the committed data at open (cold-cache batch jobs)
};
// An open file descriptor. Every operation throws IOError on failure.
class File {
public:
File() = default;
File(const File&) = delete;
File& operator=(const File&) = delete;
File(File&& other) noexcept;
File& operator=(File&& other) noexcept;
~File();
static File Open(const std::filesystem::path& path, bool writable);
static File CreateExclusive(const std::filesystem::path& path, std::filesystem::perms permissions);
bool IsOpen() const noexcept { return descriptor != -1; }
std::int32_t Descriptor() const noexcept { return descriptor; }
const std::filesystem::path& Path() const noexcept { return path; }
void SetPath(std::filesystem::path newPath) { path = std::move(newPath); }
// flock(LOCK_NB). Advisory, per open file description, released on close.
void LockExclusive();
void LockShared();
std::uint64_t Size() const;
void Truncate(std::uint64_t length);
// fallocate: real blocks, so a full disk is an error here and never a
// SIGBUS on first touch of a mapped page.
void Allocate(std::uint64_t offset, std::uint64_t length);
void ReadAt(std::uint64_t offset, std::span<std::byte> out) const;
void WriteAt(std::uint64_t offset, std::span<const std::byte> in);
void SyncData();
void Sync();
void Close() noexcept;
private:
std::int32_t descriptor = -1;
std::filesystem::path path;
};
// A reserved virtual address range into which file extents are mapped
// back to back, so the region reads as one contiguous array that grows
// without ever moving.
class MappedRegion {
public:
MappedRegion() = default;
MappedRegion(const MappedRegion&) = delete;
MappedRegion& operator=(const MappedRegion&) = delete;
MappedRegion(MappedRegion&& other) noexcept;
MappedRegion& operator=(MappedRegion&& other) noexcept;
~MappedRegion();
void Reserve(std::uint64_t bytes);
// Maps [fileOffset, fileOffset + length) at Base() + Mapped().
void Map(const File& file, std::uint64_t fileOffset, std::uint64_t length, bool writable);
void ExcludeFromCoreDumps();
void WillNeed(std::uint64_t length);
void Release() noexcept;
std::byte* Base() const noexcept { return base; }
std::uint64_t Mapped() const noexcept { return mapped; }
std::uint64_t Reserved() const noexcept { return reserved; }
private:
std::byte* base = nullptr;
std::uint64_t reserved = 0;
std::uint64_t mapped = 0;
bool dontDump = false;
};
// A plain shared mapping of a few pages (the header).
class MappedPages {
public:
MappedPages() = default;
MappedPages(const MappedPages&) = delete;
MappedPages& operator=(const MappedPages&) = delete;
MappedPages(MappedPages&& other) noexcept;
MappedPages& operator=(MappedPages&& other) noexcept;
~MappedPages();
void Map(const File& file, std::uint64_t fileOffset, std::uint64_t length, bool writable);
void Sync();
void Release() noexcept;
std::byte* Data() const noexcept { return data; }
std::uint64_t Length() const noexcept { return length; }
private:
std::byte* data = nullptr;
std::uint64_t length = 0;
};
namespace Os {
// mkdir -p with an explicit mode for every created component, so a
// private cache directory is never world-readable, not even briefly.
void CreateDirectories(const std::filesystem::path& directory, std::filesystem::perms permissions);
void Rename(const std::filesystem::path& from, const std::filesystem::path& to);
// Atomic rename that fails instead of replacing; false when `to` exists.
bool RenameNoReplace(const std::filesystem::path& from, const std::filesystem::path& to);
void HardLink(const std::filesystem::path& from, const std::filesystem::path& to);
bool Remove(const std::filesystem::path& path) noexcept;
void SyncDirectory(const std::filesystem::path& directory);
void RandomBytes(std::span<std::byte> out);
std::uint64_t UnixNanos() noexcept;
std::uint64_t PageSize() noexcept;
std::int64_t ProcessId() noexcept;
}
}

316
interfaces/Crafter.DB-Format.cppm Normal file
View file

@ -0,0 +1,316 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// The on-disk format: byte-exact header, commit record, extent table, schema
// blob and the 16-byte variable-length handle. Every struct here is a POD
// that is read and written with memcpy; the static_asserts pin each offset so
// a reordered member cannot silently change the file format. Little-endian
// only (asserted), which is every target Crafter builds for.
export module Crafter.DB:Format;
import std;
export namespace Crafter::DB {
inline constexpr std::uint32_t pageSize = 4096;
inline constexpr std::uint16_t formatVersion = 1;
inline constexpr std::size_t maxExtents = 32;
inline constexpr std::size_t regionSlots = 4;
inline constexpr std::uint16_t regionCount = 2; // 0 rows, 1 heap; 2-3 reserved
inline constexpr std::uint32_t minExtentBytes = 16384;
inline constexpr std::uint16_t noColumn = 0xFFFF;
inline constexpr std::uint32_t inlineVarBytes = 12; // strings up to this length live in the handle
inline constexpr std::array<char, 8> fileMagic = { 'C', 'R', 'A', 'F', 'T', 'R', 'D', 'B' };
inline constexpr std::array<char, 4> schemaMagic = { 'S', 'C', 'H', 'M' };
inline constexpr std::uint32_t formatFlagLittleEndian = 1;
inline constexpr std::uint64_t commitFlagCleanClose = 1;
inline constexpr std::uint8_t rowFlagDeleted = 1;
static_assert(std::endian::native == std::endian::little, "Crafter.DB stores little-endian and does not byte-swap");
// Column kinds as stored in the descriptor table. Append only: values are
// part of the file format.
enum class Kind : std::uint8_t {
Flags = 0,
Bool = 1,
Int8 = 2,
Int16 = 3,
Int32 = 4,
Int64 = 5,
UInt8 = 6,
UInt16 = 7,
UInt32 = 8,
UInt64 = 9,
Float32 = 10,
Float64 = 11,
TimePoint = 12,
Duration = 13,
String = 14,
Blob = 15,
FixedBytes = 16,
};
inline constexpr std::uint8_t columnFlagOptional = 1; // payload of a std::optional; presenceOffset is valid
inline constexpr std::uint8_t columnFlagEnum = 2; // integer column backing an enum
inline constexpr std::uint8_t columnFlagPresence = 4; // the 1-byte presence column of a std::optional
constexpr std::string_view KindName(Kind kind) noexcept {
switch (kind) {
case Kind::Flags: return "flags";
case Kind::Bool: return "bool";
case Kind::Int8: return "i8";
case Kind::Int16: return "i16";
case Kind::Int32: return "i32";
case Kind::Int64: return "i64";
case Kind::UInt8: return "u8";
case Kind::UInt16: return "u16";
case Kind::UInt32: return "u32";
case Kind::UInt64: return "u64";
case Kind::Float32: return "f32";
case Kind::Float64: return "f64";
case Kind::TimePoint: return "timepoint";
case Kind::Duration: return "duration";
case Kind::String: return "string";
case Kind::Blob: return "blob";
case Kind::FixedBytes: return "bytes";
}
return "?";
}
struct Extent {
std::uint64_t fileOffset;
std::uint64_t length;
};
static_assert(sizeof(Extent) == 16);
// One of the two alternating commit slots. The writer fills the slot that
// is not current, so a torn write can only damage the slot whose CRC then
// fails, and the other slot still describes the previous commit.
struct CommitRecord {
std::uint64_t commitSeq;
std::uint64_t rowCount;
std::uint64_t liveCount;
std::uint64_t heapUsed;
std::uint64_t heapGarbage;
std::array<std::uint32_t, regionSlots> extentCount;
std::uint64_t fileLength;
std::uint64_t modifiedUnixNanos;
std::uint64_t flags;
std::array<std::uint8_t, 428> reserved;
std::uint32_t crc32c;
};
static_assert(sizeof(CommitRecord) == 512);
static_assert(__builtin_offsetof(CommitRecord, extentCount) == 40);
static_assert(__builtin_offsetof(CommitRecord, fileLength) == 56);
static_assert(__builtin_offsetof(CommitRecord, flags) == 72);
static_assert(__builtin_offsetof(CommitRecord, crc32c) == 508);
struct FileHeader {
std::array<char, 8> magic;
std::uint16_t formatVersion;
std::uint16_t headerPages;
std::uint32_t formatFlags;
std::uint32_t pageSize;
std::uint32_t rowSize;
std::uint16_t rowAlign;
std::uint16_t columnCount;
std::uint16_t orderedColumn;
std::uint16_t regionCount;
std::uint32_t schemaVersion;
std::uint32_t reserved0;
std::uint64_t schemaFingerprint;
std::uint32_t schemaBlobLength;
std::uint32_t minExtentBytes;
std::uint64_t createdUnixNanos;
std::array<std::uint8_t, 16> fileUuid;
std::uint32_t migratedFromVersion;
std::array<std::uint8_t, 44> reserved1;
std::array<std::array<Extent, maxExtents>, regionSlots> extents;
std::array<std::uint8_t, 896> reserved2;
std::array<CommitRecord, 2> commits;
};
static_assert(sizeof(FileHeader) == pageSize);
static_assert(__builtin_offsetof(FileHeader, schemaVersion) == 32);
static_assert(__builtin_offsetof(FileHeader, schemaFingerprint) == 40);
static_assert(__builtin_offsetof(FileHeader, createdUnixNanos) == 56);
static_assert(__builtin_offsetof(FileHeader, fileUuid) == 64);
static_assert(__builtin_offsetof(FileHeader, migratedFromVersion) == 80);
static_assert(__builtin_offsetof(FileHeader, extents) == 128);
static_assert(__builtin_offsetof(FileHeader, reserved2) == 2176);
static_assert(__builtin_offsetof(FileHeader, commits) == 3072);
struct SchemaBlobHeader {
std::array<char, 4> magic;
std::uint16_t schemaFormat;
std::uint16_t columnCount;
std::uint32_t rowSize;
std::uint32_t namePoolLength;
};
static_assert(sizeof(SchemaBlobHeader) == 16);
struct ColumnDescriptor {
std::uint16_t offset;
std::uint16_t size;
Kind kind;
std::uint8_t flags;
std::uint16_t presenceOffset;
std::uint16_t nameOffset;
std::uint16_t nameLength;
std::uint32_t paramNum; // chrono period numerator; element alignment for blobs; N for fixed bytes
std::uint32_t paramDen; // chrono period denominator; element size for blobs
std::uint32_t variant; // element kind for blobs; clock code for time points; 0 otherwise
};
static_assert(sizeof(ColumnDescriptor) == 24);
inline constexpr std::uint8_t maxKind = static_cast<std::uint8_t>(Kind::FixedBytes);
// The 16-byte handle of a variable-length column (Umbra / "German string").
// Bytes 4..16 are the first 12 bytes of the value, so a value of up to 12
// bytes is entirely inline and contiguous at (handle + 4). Longer values
// keep their 4-byte prefix inline and store an offset into the heap region
// where the whole value, prefix included, lives contiguously.
struct Var {
std::uint32_t length;
std::uint32_t prefix;
std::uint64_t tailOrOffset;
};
static_assert(sizeof(Var) == 16 && alignof(Var) == 8);
constexpr std::size_t KindSize(Kind kind) noexcept {
switch (kind) {
case Kind::Flags: case Kind::Bool: case Kind::Int8: case Kind::UInt8: return 1;
case Kind::Int16: case Kind::UInt16: return 2;
case Kind::Int32: case Kind::UInt32: case Kind::Float32: return 4;
case Kind::Int64: case Kind::UInt64: case Kind::Float64: case Kind::TimePoint: case Kind::Duration: return 8;
case Kind::String: case Kind::Blob: return sizeof(Var);
case Kind::FixedBytes: return 0; // size comes from the descriptor
}
return 0;
}
// FNV-1a 64. Chosen over std::hash because the result is part of the file
// format and must not change with the standard library.
inline constexpr std::uint64_t fnvOffset = 14695981039346656037ull;
inline constexpr std::uint64_t fnvPrime = 1099511628211ull;
constexpr std::uint64_t Fnv1a(std::uint64_t hash, std::string_view bytes) noexcept {
for (const char c : bytes) {
hash ^= static_cast<std::uint8_t>(c);
hash *= fnvPrime;
}
return hash;
}
constexpr std::uint64_t Fnv1a(std::uint64_t hash, std::uint64_t value) noexcept {
for (std::size_t i = 0; i < 8; i++) {
hash ^= (value >> (8 * i)) & 0xFF;
hash *= fnvPrime;
}
return hash;
}
// CRC-32C (Castagnoli), table driven. 508 bytes per commit is too little
// for the hardware instruction to matter.
namespace Detail {
consteval std::array<std::uint32_t, 256> MakeCrcTable() {
std::array<std::uint32_t, 256> table{};
for (std::uint32_t i = 0; i < 256; i++) {
std::uint32_t c = i;
for (std::int32_t k = 0; k < 8; k++) c = (c & 1) ? (0x82F63B78u ^ (c >> 1)) : (c >> 1);
table[i] = c;
}
return table;
}
inline constexpr std::array<std::uint32_t, 256> crcTable = MakeCrcTable();
}
constexpr std::uint32_t Crc32c(std::span<const std::byte> bytes) noexcept {
std::uint32_t crc = 0xFFFFFFFFu;
for (const std::byte b : bytes) crc = Detail::crcTable[(crc ^ static_cast<std::uint8_t>(b)) & 0xFF] ^ (crc >> 8);
return crc ^ 0xFFFFFFFFu;
}
inline std::uint32_t CommitCrc(const CommitRecord& record) noexcept {
return Crc32c(std::as_bytes(std::span(&record, 1)).first(__builtin_offsetof(CommitRecord, crc32c)));
}
inline bool CommitValid(const CommitRecord& record) noexcept {
return record.commitSeq != 0 && CommitCrc(record) == record.crc32c;
}
// A parsed schema blob: what the file says its columns are. Column 0 is
// always the flags byte at offset 0.
struct SchemaInfo {
std::vector<ColumnDescriptor> columns;
std::string namePool;
std::uint32_t rowSize = 0;
std::uint16_t rowAlign = 1;
std::string_view Name(std::size_t column) const noexcept {
const ColumnDescriptor& d = columns[column];
return std::string_view(namePool).substr(d.nameOffset, d.nameLength);
}
bool HasHeap() const noexcept {
return std::ranges::any_of(columns, [](const ColumnDescriptor& d) { return d.kind == Kind::String || d.kind == Kind::Blob; });
}
std::vector<std::byte> Serialize() const {
std::vector<std::byte> out(sizeof(SchemaBlobHeader) + columns.size() * sizeof(ColumnDescriptor) + namePool.size());
const SchemaBlobHeader header{ schemaMagic, 1, static_cast<std::uint16_t>(columns.size()), rowSize, static_cast<std::uint32_t>(namePool.size()) };
std::memcpy(out.data(), &header, sizeof(header));
std::memcpy(out.data() + sizeof(header), columns.data(), columns.size() * sizeof(ColumnDescriptor));
std::memcpy(out.data() + sizeof(header) + columns.size() * sizeof(ColumnDescriptor), namePool.data(), namePool.size());
return out;
}
// Returns false when the bytes are not a well-formed blob. Every field
// of every descriptor is range-checked here, because the engine and
// DynamicRow index rows with them without further checks.
static bool Parse(std::span<const std::byte> bytes, std::uint16_t rowAlign, SchemaInfo& out) {
if (bytes.size() < sizeof(SchemaBlobHeader)) return false;
SchemaBlobHeader header;
std::memcpy(&header, bytes.data(), sizeof(header));
if (header.magic != schemaMagic || header.schemaFormat != 1 || header.columnCount == 0 || header.rowSize == 0) return false;
const std::size_t descriptorBytes = std::size_t(header.columnCount) * sizeof(ColumnDescriptor);
if (bytes.size() < sizeof(header) + descriptorBytes + header.namePoolLength) return false;
out.columns.resize(header.columnCount);
std::memcpy(out.columns.data(), bytes.data() + sizeof(header), descriptorBytes);
out.namePool.assign(reinterpret_cast<const char*>(bytes.data() + sizeof(header) + descriptorBytes), header.namePoolLength);
out.rowSize = header.rowSize;
out.rowAlign = rowAlign;
for (std::size_t i = 0; i < out.columns.size(); i++) {
const ColumnDescriptor& d = out.columns[i];
if (static_cast<std::uint8_t>(d.kind) > maxKind) return false;
if (std::size_t(d.nameOffset) + d.nameLength > out.namePool.size()) return false;
if (d.size == 0 || std::size_t(d.offset) + d.size > out.rowSize) return false;
if (d.kind != Kind::FixedBytes && KindSize(d.kind) != d.size) return false;
if (d.presenceOffset != noColumn && d.presenceOffset >= out.rowSize) return false;
if ((d.flags & columnFlagOptional) != 0 && d.presenceOffset == noColumn) return false;
if (i == 0 && (d.kind != Kind::Flags || d.offset != 0)) return false;
if (i != 0 && d.kind == Kind::Flags) return false;
}
return true;
}
// Canonical hash of the layout; the same function runs at compile time
// on Layout<T> so the two sides agree byte for byte. The ordered
// column is part of it: changing `orderedBy` changes what the file
// promises about its rows.
std::uint64_t Fingerprint(std::uint16_t orderedColumn) const noexcept {
std::uint64_t hash = Fnv1a(fnvOffset, "crafter-db-v1");
hash = Fnv1a(hash, rowSize);
hash = Fnv1a(hash, rowAlign);
hash = Fnv1a(hash, orderedColumn);
for (std::size_t i = 0; i < columns.size(); i++) {
const ColumnDescriptor& d = columns[i];
hash = Fnv1a(hash, Name(i));
hash = Fnv1a(hash, static_cast<std::uint64_t>(d.kind));
hash = Fnv1a(hash, d.offset);
hash = Fnv1a(hash, d.size);
hash = Fnv1a(hash, d.flags);
hash = Fnv1a(hash, d.paramNum);
hash = Fnv1a(hash, d.paramDen);
hash = Fnv1a(hash, d.variant);
}
return hash;
}
};
}

372
interfaces/Crafter.DB-Formatter.cppm Normal file
View file

@ -0,0 +1,372 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// The customisation point for member types, in the spirit of std::formatter:
// specialise Crafter::DB::Formatter<T> to say how a T is stored. Fixed-width
// formatters map T to a storable `Stored` type; variable-length formatters
// map it to the 16-byte Var handle and write the bytes into the heap. This
// partition also holds the reader-side handle views (StringRef, BlobRef), the
// pre-encoded search Key, and the built-in formatters for std::string,
// std::vector<Trivial> and std::chrono types.
export module Crafter.DB:Formatter;
import std;
import :Format;
import :Error;
import :Reflect;
export namespace Crafter::DB {
template<class T>
concept Scalar = (std::is_arithmetic_v<T> && !std::same_as<T, long double>) || std::is_enum_v<T> || std::same_as<T, std::byte>;
template<class T>
concept ByteLike = std::same_as<T, char> || std::same_as<T, unsigned char> || std::same_as<T, signed char> || std::same_as<T, std::byte> || std::same_as<T, char8_t>;
// The column kind of a scalar. `char` is Int8 whatever the platform's
// signedness, so a file's fingerprint does not depend on the ABI.
template<Scalar T>
consteval Kind ScalarKind() {
if constexpr (std::same_as<T, bool>) return Kind::Bool;
else if constexpr (std::is_enum_v<T>) return ScalarKind<std::underlying_type_t<T>>();
else if constexpr (std::same_as<T, std::byte>) return Kind::UInt8;
else if constexpr (std::same_as<T, char>) return Kind::Int8;
else if constexpr (std::same_as<T, wchar_t>) return sizeof(wchar_t) == 2 ? Kind::UInt16 : Kind::Int32;
else if constexpr (std::is_floating_point_v<T>) return sizeof(T) == 4 ? Kind::Float32 : Kind::Float64;
else if constexpr (std::is_signed_v<T>) return sizeof(T) == 1 ? Kind::Int8 : sizeof(T) == 2 ? Kind::Int16 : sizeof(T) == 4 ? Kind::Int32 : Kind::Int64;
else return sizeof(T) == 1 ? Kind::UInt8 : sizeof(T) == 2 ? Kind::UInt16 : sizeof(T) == 4 ? Kind::UInt32 : Kind::UInt64;
}
// A 32-bit code that identifies a type in the schema: the scalar kind
// for scalars, a hash of the compiler's spelling otherwise.
template<class T>
consteval std::uint32_t TypeCode() {
if constexpr (Scalar<T>) return static_cast<std::uint32_t>(ScalarKind<T>());
else return static_cast<std::uint32_t>(Fnv1a(fnvOffset, TypeName<T>()) & 0xFFFFFFFF) | 0x80000000u;
}
// Reader-side view of a table's heap region. `used` is the published
// length, so a handle pointing past it (a corrupt file) is caught before
// the bytes are touched.
struct Heap {
const std::byte* base = nullptr;
const std::atomic<std::uint64_t>* used = nullptr;
std::span<const std::byte> Read(std::uint64_t offset, std::uint64_t length) const {
const std::uint64_t limit = used->load(std::memory_order_acquire);
if (offset > limit || length > limit - offset) throw CorruptError(std::format("heap reference [{}, +{}) is outside the committed heap of {} bytes", offset, length, limit));
return { base + offset, static_cast<std::size_t>(length) };
}
};
// Writer-side cursor into the heap region. Formatters append through it;
// the table decides when the bytes become visible. Grow() is only reached
// when an extent fills up, so the virtual call never sits on the fast path.
class HeapWriter {
public:
virtual ~HeapWriter() = default;
std::uint64_t Write(std::span<const std::byte> bytes, std::uint32_t align = 1) {
std::uint64_t at = cursor;
if (align > 1) at = (at + align - 1) & ~std::uint64_t(align - 1);
if (at + bytes.size() > capacity) Grow(at + bytes.size());
if (!bytes.empty()) std::memcpy(base + at, bytes.data(), bytes.size());
cursor = at + bytes.size();
return at;
}
std::uint64_t Cursor() const noexcept { return cursor; }
void Rewind(std::uint64_t to) noexcept { cursor = to; }
protected:
virtual void Grow(std::uint64_t neededCapacity) = 0;
std::byte* base = nullptr;
std::uint64_t cursor = 0;
std::uint64_t capacity = 0;
};
// Whether a value of this length and element alignment is stored inside
// the handle. Alignments above 4 cannot be inline because the inline bytes
// start at handle offset 4.
constexpr bool VarInline(std::uint64_t length, std::uint32_t align) noexcept {
return length <= inlineVarBytes && align <= 4;
}
inline Var EncodeVar(std::span<const std::byte> bytes, HeapWriter& heap, std::uint32_t align = 1) {
if (bytes.size() >= std::uint64_t(1) << 32) throw Error(std::format("a value of {} bytes does not fit a Crafter.DB column (limit 4 GiB)", bytes.size()));
Var var{};
var.length = static_cast<std::uint32_t>(bytes.size());
if (VarInline(bytes.size(), align)) {
if (!bytes.empty()) std::memcpy(reinterpret_cast<std::byte*>(&var) + 4, bytes.data(), bytes.size());
} else {
// An empty blob can land here when its element alignment forbids
// inline storage; it has no prefix bytes to copy.
if (!bytes.empty()) std::memcpy(&var.prefix, bytes.data(), std::min<std::size_t>(bytes.size(), 4));
var.tailOrOffset = heap.Write(bytes, align);
}
return var;
}
// The bytes a handle refers to, inline or in the heap.
template<std::uint32_t Align = 1>
std::span<const std::byte> VarBytes(const std::byte* handle, const Heap& heap) {
Var var;
std::memcpy(&var, handle, sizeof(var));
if (VarInline(var.length, Align)) return { handle + 4, var.length };
return heap.Read(var.tailOrOffset, var.length);
}
// A search key encoded once, so a scan compares one 8-byte word per row.
// `head` is the first 8 bytes of a matching handle (length | prefix << 32)
// and `tail` the second 8 bytes when the value is inline.
class Key {
public:
explicit Key(std::string_view value) : text(value) {
head = static_cast<std::uint64_t>(static_cast<std::uint32_t>(text.size()));
std::uint32_t prefix = 0;
if (!text.empty()) std::memcpy(&prefix, text.data(), std::min<std::size_t>(text.size(), 4));
head |= std::uint64_t(prefix) << 32;
tail = 0;
if (text.size() > 4 && text.size() <= inlineVarBytes) std::memcpy(&tail, text.data() + 4, text.size() - 4);
}
std::uint64_t head;
std::uint64_t tail;
std::string text;
};
// Handle view of a string column. Points into the mapped row (and, for
// long values, the mapped heap); valid for as long as the table and that
// row are. Converts to std::string_view on demand; its comparison operators
// avoid that conversion and reject almost every non-match on the first
// 8-byte compare.
class StringRef {
public:
StringRef() = default;
StringRef(const std::byte* handle, const Heap* heap) noexcept : handle(handle), heap(heap) {}
std::uint32_t Length() const noexcept {
std::uint32_t length;
std::memcpy(&length, handle, sizeof(length));
return length;
}
bool Empty() const noexcept { return Length() == 0; }
std::string_view View() const {
const std::span<const std::byte> bytes = VarBytes<1>(handle, *heap);
return { reinterpret_cast<const char*>(bytes.data()), bytes.size() };
}
operator std::string_view() const { return View(); }
std::uint64_t Head() const noexcept {
std::uint64_t head;
std::memcpy(&head, handle, sizeof(head));
return head;
}
std::uint64_t Tail() const noexcept {
std::uint64_t tail;
std::memcpy(&tail, handle + 8, sizeof(tail));
return tail;
}
friend bool operator==(StringRef ref, const Key& key) {
if (ref.Head() != key.head) return false;
if (key.text.size() <= inlineVarBytes) return ref.Tail() == key.tail;
const std::span<const std::byte> bytes = ref.heap->Read(ref.Tail(), key.text.size());
return std::memcmp(bytes.data() + 4, key.text.data() + 4, key.text.size() - 4) == 0;
}
friend bool operator==(StringRef ref, std::string_view value) {
std::uint64_t head = static_cast<std::uint64_t>(static_cast<std::uint32_t>(value.size()));
std::uint32_t prefix = 0;
if (!value.empty()) std::memcpy(&prefix, value.data(), std::min<std::size_t>(value.size(), 4));
head |= std::uint64_t(prefix) << 32;
if (ref.Head() != head) return false;
if (value.size() <= inlineVarBytes) {
std::uint64_t tail = 0;
if (value.size() > 4) std::memcpy(&tail, value.data() + 4, value.size() - 4);
return ref.Tail() == tail;
}
const std::span<const std::byte> bytes = ref.heap->Read(ref.Tail(), value.size());
return std::memcmp(bytes.data() + 4, value.data() + 4, value.size() - 4) == 0;
}
friend bool operator==(StringRef a, StringRef b) {
if (a.Head() != b.Head()) return false;
if (a.Length() <= inlineVarBytes) return a.Tail() == b.Tail();
return a.View() == b.View();
}
friend std::strong_ordering operator<=>(StringRef ref, std::string_view value) {
const std::int32_t c = ref.View().compare(value);
return c < 0 ? std::strong_ordering::less : c > 0 ? std::strong_ordering::greater : std::strong_ordering::equal;
}
friend std::strong_ordering operator<=>(StringRef a, StringRef b) { return a <=> b.View(); }
private:
const std::byte* handle = nullptr;
const Heap* heap = nullptr;
};
// Handle view of a blob column holding elements of E.
template<class E>
class BlobRef {
public:
BlobRef() = default;
BlobRef(const std::byte* handle, const Heap* heap) noexcept : handle(handle), heap(heap) {}
std::size_t Size() const noexcept {
std::uint32_t length;
std::memcpy(&length, handle, sizeof(length));
return length / sizeof(E);
}
std::span<const E> View() const {
const std::span<const std::byte> bytes = VarBytes<alignof(E)>(handle, *heap);
return { reinterpret_cast<const E*>(bytes.data()), bytes.size() / sizeof(E) };
}
operator std::span<const E>() const { return View(); }
friend bool operator==(BlobRef ref, std::span<const E> value) {
const std::span<const E> mine = ref.View();
return mine.size() == value.size() && std::memcmp(mine.data(), value.data(), value.size_bytes()) == 0;
}
private:
const std::byte* handle = nullptr;
const Heap* heap = nullptr;
};
// Specialise this for your own member types. Left undefined on purpose:
// "no Formatter" is then a clean concept failure with a readable message.
template<class T>
struct Formatter;
// A fixed-width formatter maps T to a `Stored` type that is itself storable
// (scalar, enum, std::array, aggregate of storables). Optional members:
// `kind` (overrides the column kind), `paramNum`/`paramDen` (chrono period).
template<class T>
concept FixedFormatter = requires { typename Formatter<T>::Stored; }
&& !std::same_as<typename Formatter<T>::Stored, Var>
&& requires(const T& value, const typename Formatter<T>::Stored& stored) {
{ Formatter<T>::Encode(value) } -> std::same_as<typename Formatter<T>::Stored>;
{ Formatter<T>::Decode(stored) } -> std::convertible_to<T>;
};
// A variable-length formatter maps T to a Var handle. `View` is what a Row
// hands out (zero-copy); Materialize(View) -> T is optional and defaults to
// T(view). Optional: `kind` (String or Blob), `align` for blob elements.
template<class T>
concept VarFormatter = requires { typename Formatter<T>::Stored; typename Formatter<T>::View; }
&& std::same_as<typename Formatter<T>::Stored, Var>
&& requires(const T& value, HeapWriter& writer, const std::byte* handle, const Heap& heap) {
{ Formatter<T>::Encode(value, writer) } -> std::same_as<Var>;
{ Formatter<T>::Decode(handle, heap) } -> std::same_as<typename Formatter<T>::View>;
};
template<class T>
concept HasFormatterKind = requires { { Formatter<T>::kind } -> std::convertible_to<Kind>; };
template<class T>
concept HasFormatterPeriod = requires { { Formatter<T>::paramNum } -> std::convertible_to<std::uint32_t>; { Formatter<T>::paramDen } -> std::convertible_to<std::uint32_t>; };
template<class T>
concept HasFormatterVariant = requires { { Formatter<T>::variant } -> std::convertible_to<std::uint32_t>; };
template<class T>
concept HasMaterialize = VarFormatter<T> && requires(typename Formatter<T>::View view) { { Formatter<T>::Materialize(view) } -> std::same_as<T>; };
template<VarFormatter T>
T MaterializeVar(typename Formatter<T>::View view) {
if constexpr (HasMaterialize<T>) return Formatter<T>::Materialize(view);
else return T(view);
}
template<>
struct Formatter<std::string> {
using Stored = Var;
using View = StringRef;
static constexpr Kind kind = Kind::String;
static Var Encode(std::string_view value, HeapWriter& heap) { return EncodeVar(std::as_bytes(std::span(value)), heap, 1); }
static StringRef Decode(const std::byte* handle, const Heap& heap) noexcept { return { handle, &heap }; }
static std::string Materialize(StringRef view) { return std::string(view.View()); }
};
template<class E> requires (std::is_trivially_copyable_v<E> && !std::is_pointer_v<E> && !std::is_member_pointer_v<E>)
struct Formatter<std::vector<E>> {
using Stored = Var;
using View = BlobRef<E>;
static constexpr Kind kind = Kind::Blob;
static constexpr std::uint32_t paramNum = alignof(E);
static constexpr std::uint32_t paramDen = sizeof(E);
static constexpr std::uint32_t variant = TypeCode<E>(); // the element type is part of the schema
static Var Encode(std::span<const E> value, HeapWriter& heap) { return EncodeVar(std::as_bytes(value), heap, alignof(E)); }
static BlobRef<E> Decode(const std::byte* handle, const Heap& heap) noexcept { return { handle, &heap }; }
static std::vector<E> Materialize(BlobRef<E> view) {
const std::span<const E> elements = view.View();
return std::vector<E>(elements.begin(), elements.end());
}
};
template<class Rep, class Period>
struct Formatter<std::chrono::duration<Rep, Period>> {
static_assert(Period::num > 0 && Period::num <= 0xFFFFFFFF && Period::den > 0 && Period::den <= 0xFFFFFFFF, "chrono period does not fit the column descriptor");
using Stored = Rep;
static constexpr Kind kind = Kind::Duration;
static constexpr std::uint32_t paramNum = static_cast<std::uint32_t>(Period::num);
static constexpr std::uint32_t paramDen = static_cast<std::uint32_t>(Period::den);
static constexpr Rep Encode(const std::chrono::duration<Rep, Period>& value) noexcept { return value.count(); }
static constexpr std::chrono::duration<Rep, Period> Decode(const Rep& stored) noexcept { return std::chrono::duration<Rep, Period>(stored); }
};
template<class Clock, class Duration>
struct Formatter<std::chrono::time_point<Clock, Duration>> {
using Period = typename Duration::period;
static_assert(Period::num > 0 && Period::num <= 0xFFFFFFFF && Period::den > 0 && Period::den <= 0xFFFFFFFF, "chrono period does not fit the column descriptor");
using Stored = typename Duration::rep;
static constexpr Kind kind = Kind::TimePoint;
static constexpr std::uint32_t paramNum = static_cast<std::uint32_t>(Period::num);
static constexpr std::uint32_t paramDen = static_cast<std::uint32_t>(Period::den);
static constexpr std::uint32_t variant = TypeCode<Clock>(); // which clock the epoch belongs to
static constexpr Stored Encode(const std::chrono::time_point<Clock, Duration>& value) noexcept { return value.time_since_epoch().count(); }
static constexpr std::chrono::time_point<Clock, Duration> Decode(const Stored& stored) noexcept { return std::chrono::time_point<Clock, Duration>(Duration(stored)); }
};
// An inline, fixed-capacity string column: N bytes in the row, NUL padded,
// no heap. For codes with a known maximum length (BIC, currency, ISIN).
template<std::size_t N>
struct FixedString {
std::array<char, N> chars{};
constexpr FixedString() = default;
constexpr FixedString(std::string_view value) {
if (value.size() > N) throw std::length_error(std::format("'{}' does not fit a FixedString<{}>", value, N));
std::ranges::copy(value, chars.begin());
}
constexpr std::string_view View() const noexcept {
const auto end = std::ranges::find(chars, '\0');
return { chars.data(), static_cast<std::size_t>(end - chars.begin()) };
}
constexpr operator std::string_view() const noexcept { return View(); }
friend constexpr bool operator==(const FixedString&, const FixedString&) = default;
friend constexpr auto operator<=>(const FixedString&, const FixedString&) = default;
friend constexpr bool operator==(const FixedString& a, std::string_view b) noexcept { return a.View() == b; }
};
static_assert(VarFormatter<std::string>);
static_assert(VarFormatter<std::vector<std::byte>>);
static_assert(VarFormatter<std::vector<std::uint32_t>>);
static_assert(FixedFormatter<std::chrono::system_clock::time_point>);
static_assert(FixedFormatter<std::chrono::milliseconds>);
static_assert(HasMaterialize<std::string>);
}

212
interfaces/Crafter.DB-Migration.cppm Normal file
View file

@ -0,0 +1,212 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Schema migrations in C++. A table struct names its predecessor with
// `using Previous = OldStruct;` and either supplies
// `static T Migrate(OldStruct&&)` or, for purely additive changes, nothing:
// members are then copied by name and new ones value-initialised. Opening a
// file whose version is older than the struct's walks the chain to the type
// with that version and streams every live row through the conversions into
// a new file that replaces the old one atomically.
export module Crafter.DB:Migration;
import std;
import :Error;
import :Reflect;
import :Format;
import :Formatter;
import :Schema;
import :File;
import :Storage;
import :Codec;
export namespace Crafter::DB {
template<class From, std::size_t I, class To>
consteval std::size_t FindMemberByName() {
constexpr std::string_view name = MemberName<I, From>();
std::size_t found = noMember;
[&]<std::size_t... J>(std::index_sequence<J...>) {
((void)[&] { if (MemberName<J, To>() == name) found = J; }(), ...);
}(std::make_index_sequence<memberCount<To>>{});
return found;
}
// Additive migration: every member of From must exist in To with the same
// name and type; members only in To are value-initialised.
template<Aggregate To, Aggregate From>
To CopyByName(From&& from) {
To to{};
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
constexpr std::size_t j = FindMemberByName<From, I, To>();
static_assert(j != noMember, Message({ "Crafter.DB: member '", MemberName<I, From>(), "' of ", TypeName<From>(), " has no member of the same name in ", TypeName<To>(), "; a removed or renamed column needs an explicit `static ", TypeName<To>(), " Migrate(", TypeName<From>(), "&&)`." }));
if constexpr (j != noMember) {
static_assert(std::is_same_v<MemberType<I, From>, MemberType<j, To>>, Message({ "Crafter.DB: member '", MemberName<I, From>(), "' changed type between ", TypeName<From>(), " and ", TypeName<To>(), "; a type change needs an explicit `static ", TypeName<To>(), " Migrate(", TypeName<From>(), "&&)`." }));
Member<j>(to) = std::move(Member<I>(from));
}
}(), ...);
}(std::make_index_sequence<memberCount<From>>{});
return to;
}
// One step up the chain: Previous -> Next.
template<class Next>
Next ConvertStep(typename Next::Previous&& from) {
if constexpr (HasMigrate<Next>) return Next::Migrate(std::move(from));
else return CopyByName<Next>(std::move(from));
}
// The type directly above `From` in To's chain.
template<class To, class From>
consteval auto NextAbove() {
if constexpr (std::is_same_v<typename To::Previous, From>) return std::type_identity<To>{};
else return NextAbove<typename To::Previous, From>();
}
template<class To, class From>
To ConvertChain(From&& from) {
if constexpr (std::is_same_v<From, To>) return std::move(from);
else {
using Next = typename decltype(NextAbove<To, From>())::type;
Next next = ConvertStep<Next>(std::move(from));
return ConvertChain<To, Next>(std::move(next));
}
}
std::string DescribeColumns(const SchemaInfo& schema) {
std::string out = "[";
for (std::size_t i = 1; i < schema.columns.size(); i++) {
if (i != 1) out += ", ";
out += std::format("{}:{}", schema.Name(i), KindName(schema.columns[i].kind));
}
return out + "]";
}
template<Aggregate T>
void RequireFingerprint(const std::filesystem::path& path, const FileHeader& header, const SchemaInfo& schema) {
if (header.schemaFingerprint == Layout<T>::fingerprint && header.rowSize == Layout<T>::rowSize && header.orderedColumn == Layout<T>::orderedColumn) return;
const std::string onDisk = DescribeColumns(schema);
const std::string compiled = DescribeColumns(MakeSchemaInfo<T>());
if (onDisk == compiled && header.orderedColumn != Layout<T>::orderedColumn) {
const auto name = [&](std::uint16_t column) { return column == noColumn ? std::string("none") : std::string(schema.Name(column)); };
throw SchemaError(std::format("{}: {} version {} on disk is ordered by {} but the compiled struct is ordered by {}: `orderedBy` changed without bumping `version`", path.string(), TypeName<T>(), header.schemaVersion, name(header.orderedColumn), name(Layout<T>::orderedColumn)));
}
throw SchemaError(std::format("{}: {} version {} on disk has columns {} but the compiled struct has {}: the struct changed without bumping `version`", path.string(), TypeName<T>(), header.schemaVersion, onDisk, compiled));
}
// Writes the converted live rows of `old` into a fresh file and returns
// its path. Streaming keeps memory flat; `sorted` collects everything and
// stable-sorts by the new orderedBy first, for the case where the old data
// is not in the order the new struct declares.
template<TableType T, Aggregate Old>
std::filesystem::path WriteConverted(const TableFile& old, const std::filesystem::path& path, const OpenOptions& options, bool sorted) {
const TableStats stats = old.Stats();
CreateParams params;
params.schema = MakeSchemaInfo<T>();
params.schemaVersion = Layout<T>::version;
params.fingerprint = Layout<T>::fingerprint;
params.orderedColumn = Layout<T>::orderedColumn;
params.initialRowBytes = stats.liveCount * Layout<T>::rowSize;
params.initialHeapBytes = stats.heapUsed - std::min(stats.heapUsed, stats.heapGarbage);
params.migratedFromVersion = old.Header().schemaVersion;
OpenOptions targetOptions = options;
targetOptions.readOnly = false;
targetOptions.createDirectories = false;
const std::filesystem::path temp = TableFile::TempPath(path);
TableFile::Create(temp, params, targetOptions);
try {
TableFile target(temp, targetOptions);
const std::uint64_t slots = old.SlotCount();
if (sorted) {
std::vector<T> rows;
rows.reserve(stats.liveCount);
for (std::uint64_t i = 0; i < slots; i++) {
const std::byte* row = old.Slot(i);
if (!TableFile::Deleted(row)) rows.push_back(ConvertChain<T, Old>(DecodeRow<Old>(row, old.HeapView())));
}
if constexpr (HasOrderedBy<T>) std::ranges::stable_sort(rows, {}, [](const T& row) { return row.*T::orderedBy; });
AppendRows<T>(target, std::span<const T>(rows));
} else {
for (std::uint64_t i = 0; i < slots; i++) {
const std::byte* row = old.Slot(i);
if (TableFile::Deleted(row)) continue;
T converted = ConvertChain<T, Old>(DecodeRow<Old>(row, old.HeapView()));
AppendRows<T>(target, std::span<const T>(&converted, 1));
}
}
target.Flush(true);
} catch (...) {
Os::Remove(temp);
throw;
}
return temp;
}
template<TableType T, Aggregate Old>
void RewriteFrom(const std::filesystem::path& path, const OpenOptions& options) {
OpenOptions sourceOptions = options;
sourceOptions.readOnly = false;
sourceOptions.mode = OpenMode::OpenExisting;
TableFile old(path, sourceOptions);
const std::uint32_t oldVersion = old.Header().schemaVersion;
std::filesystem::path temp;
try {
temp = WriteConverted<T, Old>(old, path, options, false);
} catch (const OrderError&) {
if constexpr (!HasOrderedBy<T>) throw;
temp = WriteConverted<T, Old>(old, path, options, true);
}
try {
if (options.backupBeforeMigrate) {
const std::filesystem::path backup = TableFile::BackupPath(path, oldVersion);
Os::Remove(backup);
Os::HardLink(path, backup);
}
old.Close();
TableFile::Replace(path, temp);
} catch (...) {
Os::Remove(temp);
throw;
}
}
template<TableType T, Aggregate Old>
bool WalkChain(const std::filesystem::path& path, const Inspection& info, const OpenOptions& options) {
if (Layout<Old>::version == info.header.schemaVersion) {
RequireFingerprint<Old>(path, info.header, info.schema);
RewriteFrom<T, Old>(path, options);
return true;
}
if constexpr (HasPrevious<Old>) return WalkChain<T, typename Old::Previous>(path, info, options);
else return false;
}
template<class T>
consteval std::uint32_t OldestVersion() {
if constexpr (HasPrevious<T>) return OldestVersion<typename T::Previous>();
else return Layout<T>::version;
}
// Brings a file whose version differs from T's up to date, or throws.
template<TableType T>
void MigrateTable(const std::filesystem::path& path, const Inspection& info, const OpenOptions& options) {
const std::uint32_t fileVersion = info.header.schemaVersion;
if (fileVersion == Layout<T>::version) return;
if (fileVersion > Layout<T>::version) throw SchemaError(std::format("{}: file is at version {} but this build of {} is version {}; the file was written by a newer build", path.string(), fileVersion, TypeName<T>(), Layout<T>::version));
if (options.readOnly) throw MigrationError(std::format("{}: file is at version {} and {} is version {}, but the table was opened read-only", path.string(), fileVersion, TypeName<T>(), Layout<T>::version));
if (!options.migrate) throw MigrationError(std::format("{}: file is at version {} and {} is version {}, and migration was disabled", path.string(), fileVersion, TypeName<T>(), Layout<T>::version));
if (!WalkChain<T, T>(path, info, options)) {
throw SchemaError(std::format("{}: no migration path from on-disk version {} to {}: the Previous chain of {} stops at version {}", path.string(), fileVersion, Layout<T>::version, TypeName<T>(), OldestVersion<T>()));
}
}
// Explicit form: migrate the file at `path` to T without keeping it open.
template<TableType T>
void Migrate(const std::filesystem::path& path, const OpenOptions& options = {}) {
MigrateTable<T>(path, TableFile::Inspect(path), options);
}
std::uint32_t PeekVersion(const std::filesystem::path& path) {
return TableFile::Inspect(path).header.schemaVersion;
}
}

205
interfaces/Crafter.DB-Query.cppm Normal file
View file

@ -0,0 +1,205 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Queries are C++. A lambda over Row<T> is the general form; column tags
// give a readable expression form (iban == x && amount > 100) whose types
// the engine can look into: comparisons on the orderedBy column found in a
// top-level && chain become binary-search bounds instead of a scan, and
// string comparisons pre-encode their Key once instead of once per row.
export module Crafter.DB:Query;
import std;
import :Reflect;
import :Formatter;
import :Schema;
import :Storage;
import :Codec;
import :Row;
export namespace Crafter::DB {
enum class CompareOp : std::uint8_t { Equal, NotEqual, Less, LessEqual, Greater, GreaterEqual };
template<auto Ptr, CompareOp Op, class Operand>
struct Compare {
using Table = MemberClass<Ptr>;
static constexpr auto pointer = Ptr;
static constexpr CompareOp op = Op;
Operand operand;
bool operator()(Row<Table> row) const {
const auto value = row.template Get<Ptr>();
if constexpr (Op == CompareOp::Equal) return value == operand;
else if constexpr (Op == CompareOp::NotEqual) return !(value == operand);
else if constexpr (Op == CompareOp::Less) return value < operand;
else if constexpr (Op == CompareOp::LessEqual) return value <= operand;
else if constexpr (Op == CompareOp::Greater) return value > operand;
else return value >= operand;
}
};
template<class A, class B>
struct And {
A a;
B b;
template<class R> bool operator()(R row) const { return a(row) && b(row); }
};
template<class A, class B>
struct Or {
A a;
B b;
template<class R> bool operator()(R row) const { return a(row) || b(row); }
};
template<class A>
struct Not {
A a;
template<class R> bool operator()(R row) const { return !a(row); }
};
template<class P> struct IsExpression : std::false_type {};
template<auto Ptr, CompareOp Op, class V> struct IsExpression<Compare<Ptr, Op, V>> : std::true_type {};
template<class A, class B> struct IsExpression<And<A, B>> : std::true_type {};
template<class A, class B> struct IsExpression<Or<A, B>> : std::true_type {};
template<class A> struct IsExpression<Not<A>> : std::true_type {};
template<class P>
concept Expression = IsExpression<std::remove_cvref_t<P>>::value;
template<class M>
concept StringColumn = std::is_same_v<M, std::string>;
template<auto Ptr, CompareOp Op, class V>
auto MakeCompare(V&& value) {
using M = MemberTypeOf<Ptr>;
if constexpr (StringColumn<M>) {
if constexpr (Op == CompareOp::Equal || Op == CompareOp::NotEqual) return Compare<Ptr, Op, Key>{ Key(std::string_view(value)) };
else return Compare<Ptr, Op, std::string>{ std::string(std::string_view(value)) };
} else {
return Compare<Ptr, Op, M>{ static_cast<M>(std::forward<V>(value)) };
}
}
template<auto Ptr, class V> auto operator==(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::Equal>(std::forward<V>(value)); }
template<auto Ptr, class V> auto operator!=(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::NotEqual>(std::forward<V>(value)); }
template<auto Ptr, class V> auto operator<(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::Less>(std::forward<V>(value)); }
template<auto Ptr, class V> auto operator<=(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::LessEqual>(std::forward<V>(value)); }
template<auto Ptr, class V> auto operator>(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::Greater>(std::forward<V>(value)); }
template<auto Ptr, class V> auto operator>=(Column<Ptr>, V&& value) { return MakeCompare<Ptr, CompareOp::GreaterEqual>(std::forward<V>(value)); }
template<Expression A, Expression B> auto operator&&(A a, B b) { return And<A, B>{ std::move(a), std::move(b) }; }
template<Expression A, Expression B> auto operator||(A a, B b) { return Or<A, B>{ std::move(a), std::move(b) }; }
template<Expression A> auto operator!(A a) { return Not<A>{ std::move(a) }; }
// Bounds on the ordered column harvested from a predicate's top-level
// conjunction. Anything under || or ! contributes nothing (the whole
// predicate still runs as the filter, so this is only ever a narrowing).
template<class K>
struct Bounds {
std::optional<K> lower;
bool lowerExclusive = false;
std::optional<K> upper;
bool upperExclusive = false;
void Lower(const K& key, bool exclusive) {
if (!lower || *lower < key || (*lower == key && exclusive)) {
lower = key;
lowerExclusive = exclusive;
}
}
void Upper(const K& key, bool exclusive) {
if (!upper || key < *upper || (*upper == key && exclusive)) {
upper = key;
upperExclusive = exclusive;
}
}
};
template<class P> struct IsAndType : std::false_type {};
template<class A, class B> struct IsAndType<And<A, B>> : std::true_type {};
template<class P> struct IsCompareType : std::false_type {};
template<auto Ptr, CompareOp Op, class V> struct IsCompareType<Compare<Ptr, Op, V>> : std::true_type {};
template<class T, class P> requires HasOrderedBy<T>
void CollectBounds(const P& predicate, Bounds<OrderedKey<T>>& bounds) {
if constexpr (IsAndType<P>::value) {
CollectBounds<T>(predicate.a, bounds);
CollectBounds<T>(predicate.b, bounds);
} else if constexpr (IsCompareType<P>::value) {
if constexpr (SameMemberPointer<P::pointer, T::orderedBy>()) {
const OrderedKey<T> key = static_cast<OrderedKey<T>>(predicate.operand);
switch (P::op) {
case CompareOp::Equal: bounds.Lower(key, false); bounds.Upper(key, false); break;
case CompareOp::Less: bounds.Upper(key, true); break;
case CompareOp::LessEqual: bounds.Upper(key, false); break;
case CompareOp::Greater: bounds.Lower(key, true); break;
case CompareOp::GreaterEqual: bounds.Lower(key, false); break;
case CompareOp::NotEqual: break;
}
}
}
}
// A lazy range of Row<T> with the conveniences a result needs. Chainable:
// table.Range<...>(a, b).Where(pred).Select<&T::amount>().
template<TableType T, std::ranges::view R> requires std::is_same_v<std::ranges::range_value_t<R>, Row<T>>
class Query : public std::ranges::view_interface<Query<T, R>> {
public:
Query() = default;
explicit Query(R range) : range(std::move(range)) {}
// A Query is iterable as a const object whenever its underlying view
// is (LiveRange is; std::ranges::filter_view, by the standard's
// design, is not).
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
auto begin(this Self&& self) { return std::ranges::begin(self.range); }
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
auto end(this Self&& self) { return std::ranges::end(self.range); }
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
std::vector<T> Materialize(this Self&& self) {
std::vector<T> out;
for (Row<T> row : self.range) out.push_back(row.Materialize());
return out;
}
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
std::uint64_t Count(this Self&& self) {
std::uint64_t n = 0;
for ([[maybe_unused]] Row<T> row : self.range) n++;
return n;
}
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
std::optional<Row<T>> First(this Self&& self) {
for (Row<T> row : self.range) return row;
return std::nullopt;
}
template<class Self> requires std::ranges::range<std::conditional_t<std::is_const_v<std::remove_reference_t<Self>>, const R, R>>
bool Any(this Self&& self) { return self.First().has_value(); }
template<class Pred>
auto Where(Pred pred) && {
return Query<T, std::ranges::filter_view<R, Pred>>(std::ranges::filter_view<R, Pred>(std::move(range), std::move(pred)));
}
template<class Pred>
auto Where(Pred pred) const& {
return Query<T, std::ranges::filter_view<R, Pred>>(std::ranges::filter_view<R, Pred>(range, std::move(pred)));
}
// Lazy view of one column's values (views for strings, values otherwise).
template<auto Ptr> requires std::is_same_v<MemberClass<Ptr>, T>
auto Select() && {
return std::views::transform(std::move(range), [](Row<T> row) { return row.template Get<Ptr>(); });
}
const R& Range() const noexcept { return range; }
private:
R range;
};
}

174
interfaces/Crafter.DB-Reflect.cppm Normal file
View file

@ -0,0 +1,174 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Compile-time introspection of aggregates without P2996 (clang 23 has no
// <meta>). Three ingredients, all verified on the toolchain this library
// targets: structured-binding packs give member count and per-member access,
// std::source_location::function_name() leaks the member name when a pointer
// to a member of an extern object is used as a template argument, and address
// comparison on a constexpr instance maps a pointer-to-member to its index.
// When a clang ships <meta>, this partition is the only thing to replace.
export module Crafter.DB:Reflect;
import std;
namespace Crafter::DB::Reflect {
template<class T>
struct CountHelper {
template<class U>
static consteval auto Count(U& u) {
auto&& [...m] = u;
return std::integral_constant<std::size_t, sizeof...(m)>{};
}
static constexpr std::size_t value = decltype(Count(std::declval<T&>()))::value;
};
// A constant-initialised instance whose member addresses are turned into
// text by function_name(). Only its address is ever used, in a constant
// expression, so nothing reads or writes it at run time; no_destroy keeps
// the compiler from registering a destructor for it, which lets the linker
// drop the object entirely. (An undefined `extern` declaration would be
// cheaper still, but clang warns about it at every instantiation.)
template<class T>
[[clang::no_destroy]] inline const T external{};
template<auto P>
consteval std::string_view RawName() {
return std::source_location::current().function_name();
}
template<class T>
consteval std::string_view RawTypeName() {
return std::source_location::current().function_name();
}
}
export namespace Crafter::DB {
// True for class types whose non-static data members can be bound by a
// structured binding: public members, no bases carrying members, no
// user-declared constructors. Tables and nested column groups must be this.
template<class T>
concept Aggregate = std::is_aggregate_v<T> && !std::is_array_v<T> && requires(T& t) { Reflect::CountHelper<T>::value; };
template<Aggregate T>
inline constexpr std::size_t memberCount = Reflect::CountHelper<T>::value;
// Reference to the I-th member of an aggregate lvalue.
template<std::size_t I, class T>
constexpr decltype(auto) Member(T&& t) noexcept {
auto&& [...m] = t;
return (m...[I]);
}
template<std::size_t I, class T>
using MemberType = std::remove_cvref_t<decltype(Member<I>(std::declval<T&>()))>;
// The declared name of the I-th member, e.g. "iban". Parsed out of
// "std::string_view Crafter::DB::Reflect::RawName() [P = &external.iban]".
template<std::size_t I, Aggregate T>
consteval std::string_view MemberName() {
constexpr std::string_view raw = Reflect::RawName<&Member<I>(Reflect::external<T>)>();
const std::size_t end = raw.rfind(']');
const std::size_t dot = raw.rfind('.', end);
return raw.substr(dot + 1, end - dot - 1);
}
// The type's qualified name as the compiler spells it ("Nostro::Transaction").
// Used in diagnostics only, so the exact spelling is not load-bearing.
template<class T>
consteval std::string_view TypeName() {
constexpr std::string_view raw = Reflect::RawTypeName<T>();
const std::size_t end = raw.rfind(']');
const std::size_t eq = raw.rfind("T = ", end);
return raw.substr(eq + 4, end - eq - 4);
}
template<class Ptr>
struct MemberPointerTraits;
template<class C, class M>
struct MemberPointerTraits<M C::*> {
using Class = C;
using Type = M;
};
template<auto Ptr>
using MemberClass = typename MemberPointerTraits<decltype(Ptr)>::Class;
template<auto Ptr>
using MemberTypeOf = typename MemberPointerTraits<decltype(Ptr)>::Type;
inline constexpr std::size_t noMember = static_cast<std::size_t>(-1);
// Equality of two pointers-to-member that may point into different member
// types (comparing those directly is ill-formed): false unless the types
// match and the pointers are equal.
template<auto A, auto B>
consteval bool SameMemberPointer() {
if constexpr (std::is_same_v<std::remove_cv_t<decltype(A)>, std::remove_cv_t<decltype(B)>>) return A == B;
else return false;
}
// Index of the member a pointer-to-member designates, or noMember. Works
// by comparing the address of each bound member with the address the
// pointer selects on the same constexpr instance.
template<Aggregate T, auto Ptr>
consteval std::size_t MemberIndex() {
T t{};
std::size_t index = noMember;
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
if constexpr (std::is_same_v<MemberType<I, T>, std::remove_cvref_t<decltype(t.*Ptr)>>) {
if (std::addressof(Member<I>(t)) == std::addressof(t.*Ptr)) index = I;
}
}(), ...);
}(std::make_index_sequence<memberCount<T>>{});
return index;
}
// Runtime counterpart of MemberIndex for a pointer that is not a template
// argument. Every call site passes a literal `&T::member`, and the probe
// object is constant-initialised (no guard to check), so after inlining the
// comparison chain folds to a constant.
template<Aggregate T, class M>
std::size_t MemberIndexOf(M T::* ptr) noexcept {
const T& instance = Reflect::external<T>;
std::size_t index = noMember;
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
if constexpr (std::is_same_v<MemberType<I, T>, M>) {
if (std::addressof(Member<I>(instance)) == std::addressof(instance.*ptr)) index = I;
}
}(), ...);
}(std::make_index_sequence<memberCount<T>>{});
return index;
}
// Calls f.template operator()<I>(member) for every member, in order.
template<Aggregate T, class F>
constexpr void ForEachMember(T& t, F&& f) {
[&]<std::size_t... I>(std::index_sequence<I...>) {
(f.template operator()<I>(Member<I>(t)), ...);
}(std::make_index_sequence<memberCount<T>>{});
}
}
// Self-tests: a compiler whose function_name() text changes shape fails here,
// at the library's build, instead of producing garbage column names.
namespace Crafter::DB::Reflect {
struct Probe {
std::int32_t alpha;
double beta;
std::string gamma;
std::chrono::system_clock::time_point delta;
};
static_assert(Aggregate<Probe>);
static_assert(memberCount<Probe> == 4);
static_assert(MemberName<0, Probe>() == "alpha");
static_assert(MemberName<2, Probe>() == "gamma");
static_assert(MemberName<3, Probe>() == "delta");
static_assert(MemberIndex<Probe, &Probe::gamma>() == 2);
static_assert(MemberIndex<Probe, &Probe::delta>() == 3);
static_assert(std::is_same_v<MemberType<1, Probe>, double>);
static_assert(std::is_same_v<MemberTypeOf<&Probe::gamma>, std::string>);
static_assert(TypeName<Probe>().ends_with("Probe"));
}

277
interfaces/Crafter.DB-Row.cppm Normal file
View file

@ -0,0 +1,277 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Row<T>: a 16-byte view of one stored row that decodes columns on demand,
// plus the ranges over rows: LiveRange skips tombstones, SlotView is random
// access over every slot (RowId space).
export module Crafter.DB:Row;
import std;
import :Reflect;
import :Format;
import :Formatter;
import :Schema;
import :Storage;
import :Codec;
export namespace Crafter::DB {
using RowId = std::uint64_t;
// Column tag: `inline constexpr Column<&Transaction::iban> iban{};`
// lets you write row[iban] and iban == x. The operators live in :Query.
template<auto Ptr>
struct Column {
using Table = MemberClass<Ptr>;
using Type = MemberTypeOf<Ptr>;
static constexpr auto pointer = Ptr;
static constexpr std::size_t index = MemberIndex<Table, Ptr>();
static constexpr std::string_view Name() noexcept { return MemberName<index, Table>(); }
};
template<TableType T>
class Row {
public:
Row() = default;
Row(const std::byte* row, const TableFile* file) noexcept : row(row), file(file) {}
// Positional access: member I, resolved at compile time.
template<std::size_t I>
View<MemberType<I, T>> Get() const {
return DecodeMember<T, I>(row, file->HeapView());
}
// Member-pointer access, resolved at compile time: row.Get<&T::iban>().
template<auto Ptr> requires std::is_same_v<MemberClass<Ptr>, T>
View<MemberTypeOf<Ptr>> Get() const {
constexpr std::size_t index = MemberIndex<T, Ptr>();
static_assert(index != noMember, "Crafter.DB: pointer does not designate a direct member of the table struct");
return Get<index>();
}
// Member-pointer access with a run-time pointer: row[&T::iban]. Every
// call site passes a literal, so the lookup folds after inlining.
template<class M>
View<M> operator[](M T::* ptr) const {
const std::size_t index = MemberIndexOf<T>(ptr);
View<M> result{};
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
if constexpr (std::is_same_v<MemberType<I, T>, M>) {
if (index == I) result = Get<I>();
}
}(), ...);
}(std::make_index_sequence<memberCount<T>>{});
return result;
}
template<auto Ptr> requires std::is_same_v<MemberClass<Ptr>, T>
View<MemberTypeOf<Ptr>> operator[](Column<Ptr>) const {
return Get<Ptr>();
}
// Copies every column out into a T (allocates for strings and blobs).
T Materialize() const { return DecodeRow<T>(row, file->HeapView()); }
explicit operator T() const { return Materialize(); }
RowId Id() const noexcept { return static_cast<RowId>((row - file->Rows()) / Layout<T>::rowSize); }
bool Erased() const noexcept { return TableFile::Deleted(row); }
const std::byte* Bytes() const noexcept { return row; }
const TableFile* File() const noexcept { return file; }
friend bool operator==(const Row&, const Row&) = default;
private:
const std::byte* row = nullptr;
const TableFile* file = nullptr;
};
// Forward iterator over live rows of a slot range. The end of the range
// is a sentinel, so the iterator's own snapshot of the last slot is the
// only bound it ever compares against: a concurrent Append between two
// reads of the slot count cannot make it walk into unpublished memory.
// `checkFlags` is false when the table has no tombstones, which removes
// the per-row test.
template<TableType T>
class LiveIterator {
public:
using value_type = Row<T>;
using difference_type = std::ptrdiff_t;
using iterator_concept = std::forward_iterator_tag;
using iterator_category = std::forward_iterator_tag;
LiveIterator() = default;
LiveIterator(const std::byte* current, const std::byte* last, const TableFile* file, bool checkFlags) noexcept : current(current), last(last), file(file), checkFlags(checkFlags) { Skip(); }
Row<T> operator*() const noexcept { return { current, file }; }
LiveIterator& operator++() noexcept {
current += Layout<T>::rowSize;
Skip();
return *this;
}
LiveIterator operator++(int) noexcept {
LiveIterator copy = *this;
++*this;
return copy;
}
friend bool operator==(const LiveIterator& a, const LiveIterator& b) noexcept { return a.current == b.current; }
friend bool operator==(const LiveIterator& it, std::default_sentinel_t) noexcept { return it.current >= it.last; }
private:
void Skip() noexcept {
if (!checkFlags) return;
while (current < last && TableFile::Deleted(current)) current += Layout<T>::rowSize;
}
const std::byte* current = nullptr;
const std::byte* last = nullptr;
const TableFile* file = nullptr;
bool checkFlags = false;
};
template<TableType T>
class LiveRange : public std::ranges::view_interface<LiveRange<T>> {
public:
LiveRange() = default;
LiveRange(const TableFile* file, std::uint64_t firstSlot, std::uint64_t endSlot) noexcept
: file(file), first(file->Slot(firstSlot)), last(file->Slot(endSlot)) {}
// The tombstone flag is read when iteration starts, not when the range
// was built, so a query object created before an Erase still skips it.
LiveIterator<T> begin() const noexcept { return { first, last, file, file->HasTombstones() }; }
std::default_sentinel_t end() const noexcept { return {}; }
std::uint64_t SlotCount() const noexcept { return static_cast<std::uint64_t>((last - first) / Layout<T>::rowSize); }
private:
const TableFile* file = nullptr;
const std::byte* first = nullptr;
const std::byte* last = nullptr;
};
// Random access over every slot, tombstones included (row.Erased() tells).
template<TableType T>
class SlotView : public std::ranges::view_interface<SlotView<T>> {
public:
class Iterator {
public:
using value_type = Row<T>;
using difference_type = std::ptrdiff_t;
using iterator_concept = std::random_access_iterator_tag;
using iterator_category = std::random_access_iterator_tag;
Iterator() = default;
Iterator(const TableFile* file, std::uint64_t index) noexcept : file(file), index(index) {}
Row<T> operator*() const noexcept { return { file->Slot(index), file }; }
Row<T> operator[](difference_type n) const noexcept { return *(*this + n); }
Iterator& operator++() noexcept { ++index; return *this; }
Iterator operator++(int) noexcept { Iterator c = *this; ++index; return c; }
Iterator& operator--() noexcept { --index; return *this; }
Iterator operator--(int) noexcept { Iterator c = *this; --index; return c; }
Iterator& operator+=(difference_type n) noexcept { index = static_cast<std::uint64_t>(static_cast<difference_type>(index) + n); return *this; }
Iterator& operator-=(difference_type n) noexcept { return *this += -n; }
friend Iterator operator+(Iterator it, difference_type n) noexcept { return it += n; }
friend Iterator operator+(difference_type n, Iterator it) noexcept { return it += n; }
friend Iterator operator-(Iterator it, difference_type n) noexcept { return it -= n; }
friend difference_type operator-(const Iterator& a, const Iterator& b) noexcept { return static_cast<difference_type>(a.index) - static_cast<difference_type>(b.index); }
friend bool operator==(const Iterator& a, const Iterator& b) noexcept { return a.index == b.index; }
friend auto operator<=>(const Iterator& a, const Iterator& b) noexcept { return a.index <=> b.index; }
private:
const TableFile* file = nullptr;
std::uint64_t index = 0;
};
SlotView() = default;
SlotView(const TableFile* file, std::uint64_t firstSlot, std::uint64_t endSlot) noexcept : file(file), first(firstSlot), last(endSlot) {}
Iterator begin() const noexcept { return { file, first }; }
Iterator end() const noexcept { return { file, last }; }
std::size_t size() const noexcept { return static_cast<std::size_t>(last - first); }
Row<T> operator[](std::uint64_t index) const noexcept { return { file->Slot(first + index), file }; }
private:
const TableFile* file = nullptr;
std::uint64_t first = 0;
std::uint64_t last = 0;
};
// --- debug formatting ---------------------------------------------------
template<class V> struct IsBlobRef : std::false_type {};
template<class E> struct IsBlobRef<BlobRef<E>> : std::true_type {};
template<class V>
void FormatValue(std::string& out, const V& value) {
if constexpr (std::is_same_v<V, StringRef>) out += std::format("\"{}\"", value.View());
else if constexpr (std::is_same_v<V, std::string>) out += std::format("\"{}\"", value);
else if constexpr (std::is_same_v<V, bool>) out += value ? "true" : "false";
else if constexpr (std::is_enum_v<V>) out += std::format("{}", std::to_underlying(value));
else if constexpr (std::is_same_v<V, std::byte>) out += std::format("{}", std::to_integer<std::uint32_t>(value));
else if constexpr (std::is_arithmetic_v<V>) out += std::format("{}", value);
else if constexpr (IsBlobRef<V>::value) out += std::format("<{} elements>", value.Size());
else if constexpr (FixedStringTraits<V>::value) out += std::format("\"{}\"", value.View());
else if constexpr (OptionalTraits<V>::value) {
if (value) FormatValue(out, *value);
else out += "null";
}
else if constexpr (StdArrayTraits<V>::value) {
out += "[";
for (std::size_t i = 0; i < value.size(); i++) {
if (i != 0) out += ", ";
FormatValue(out, value[i]);
}
out += "]";
}
else if constexpr (std::formattable<V, char>) out += std::format("{}", value);
else if constexpr (Aggregate<V>) {
out += "{";
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
if (I != 0) out += ", ";
out += MemberName<I, V>();
out += "=";
FormatValue(out, Member<I>(value));
}(), ...);
}(std::make_index_sequence<memberCount<V>>{});
out += "}";
}
else if constexpr (requires { value.size(); value.begin(); }) out += std::format("<{} elements>", value.size());
else out += "?";
}
template<TableType T>
std::string DescribeRow(Row<T> row) {
std::string out(TypeName<T>());
out += "{";
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
if (I != 0) out += ", ";
out += MemberName<I, T>();
out += "=";
FormatValue(out, row.template Get<I>());
}(), ...);
}(std::make_index_sequence<memberCount<T>>{});
out += "}";
return out;
}
}
// Neither view owns the mapping, so iterators may outlive the view object.
template<class T>
inline constexpr bool std::ranges::enable_borrowed_range<Crafter::DB::SlotView<T>> = true;
template<class T>
inline constexpr bool std::ranges::enable_borrowed_range<Crafter::DB::LiveRange<T>> = true;
template<Crafter::DB::TableType T>
struct std::formatter<Crafter::DB::Row<T>> : std::formatter<std::string> {
auto format(const Crafter::DB::Row<T>& row, std::format_context& ctx) const {
return std::formatter<std::string>::format(Crafter::DB::DescribeRow(row), ctx);
}
};
template<>
struct std::formatter<Crafter::DB::StringRef> : std::formatter<std::string_view> {
auto format(const Crafter::DB::StringRef& ref, std::format_context& ctx) const {
return std::formatter<std::string_view>::format(ref.View(), ctx);
}
};

512
interfaces/Crafter.DB-Schema.cppm Normal file
View file

@ -0,0 +1,512 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// From a C++ struct to a row layout, entirely at compile time: members are
// classified by Shape, nested aggregates are flattened into dotted column
// names, columns are placed by ascending alignment after the flags byte at
// offset 0, and the result is hashed into the schema fingerprint the file
// header carries. The same partition holds the table-level declarations
// (version, orderedBy, Previous, Migrate) and the diagnostics for misuse.
export module Crafter.DB:Schema;
import std;
import :Reflect;
import :Format;
import :Formatter;
export namespace Crafter::DB {
enum class Shape : std::uint8_t { Scalar, FixedBytes, ScalarArray, Optional, Fixed, Var, Aggregate, Pointer, Unsupported };
template<class T> struct OptionalTraits : std::false_type {};
template<class M> struct OptionalTraits<std::optional<M>> : std::true_type { using Type = M; };
template<class T> struct StdArrayTraits : std::false_type {};
template<class E, std::size_t N> struct StdArrayTraits<std::array<E, N>> : std::true_type {
using Element = E;
static constexpr std::size_t size = N;
};
template<class T> struct FixedStringTraits : std::false_type {};
template<std::size_t N> struct FixedStringTraits<FixedString<N>> : std::true_type { static constexpr std::size_t size = N; };
template<class T>
consteval Shape ShapeOf() {
if constexpr (std::is_pointer_v<T> || std::is_member_pointer_v<T> || std::is_reference_v<T> || std::is_null_pointer_v<T>) return Shape::Pointer;
else if constexpr (Scalar<T>) return Shape::Scalar;
else if constexpr (FixedStringTraits<T>::value) return Shape::FixedBytes;
else if constexpr (StdArrayTraits<T>::value) {
if constexpr (ByteLike<typename StdArrayTraits<T>::Element>) return Shape::FixedBytes;
else if constexpr (Scalar<typename StdArrayTraits<T>::Element>) return Shape::ScalarArray;
else return Shape::Unsupported;
}
else if constexpr (OptionalTraits<T>::value) return Shape::Optional;
else if constexpr (VarFormatter<T>) return Shape::Var;
else if constexpr (FixedFormatter<T>) return Shape::Fixed;
else if constexpr (Aggregate<T>) return Shape::Aggregate;
else return Shape::Unsupported;
}
// Number of row columns a member of type T occupies.
template<class T>
consteval std::size_t LeafCount() {
constexpr Shape shape = ShapeOf<T>();
if constexpr (shape == Shape::Scalar || shape == Shape::FixedBytes || shape == Shape::Var) return 1;
else if constexpr (shape == Shape::ScalarArray) return StdArrayTraits<T>::size;
else if constexpr (shape == Shape::Optional) return 1 + LeafCount<typename OptionalTraits<T>::Type>();
else if constexpr (shape == Shape::Fixed) return LeafCount<typename Formatter<T>::Stored>();
else if constexpr (shape == Shape::Aggregate) {
return []<std::size_t... I>(std::index_sequence<I...>) { return (std::size_t(0) + ... + LeafCount<MemberType<I, T>>()); }(std::make_index_sequence<memberCount<T>>{});
}
else return 1; // unsupported shapes are rejected by CheckTable; count 1 keeps indices sane meanwhile
}
// First leaf of member I of aggregate T.
template<class T, std::size_t I>
consteval std::size_t LeafStart() {
return []<std::size_t... J>(std::index_sequence<J...>) { return (std::size_t(0) + ... + LeafCount<MemberType<J, T>>()); }(std::make_index_sequence<I>{});
}
// A compile-time string with value semantics (std::string cannot cross
// from consteval into a constexpr variable). Overflow is a build error,
// never a silent truncation: a truncated column name would alias another
// column, a truncated diagnostic would lose its hint.
template<std::size_t Capacity>
struct FixedText {
std::array<char, Capacity> chars{};
std::uint32_t length = 0;
constexpr void Append(std::string_view text) {
if (length + text.size() > chars.size()) throw "Crafter.DB: a column name or diagnostic exceeds the fixed buffer; shorten the member path";
for (const char c : text) chars[length++] = c;
}
constexpr std::string_view View() const noexcept { return { chars.data(), length }; }
constexpr std::size_t size() const noexcept { return length; }
constexpr const char* data() const noexcept { return chars.data(); }
};
using FixedName = FixedText<256>; // a dotted column path
using MessageText = FixedText<1024>; // a static_assert message
inline constexpr std::uint32_t noLeaf = 0xFFFFFFFF;
struct LeafInfo {
FixedName name;
Kind kind = Kind::UInt8;
std::uint8_t flags = 0;
std::uint16_t size = 1;
std::uint16_t align = 1;
std::uint32_t paramNum = 0;
std::uint32_t paramDen = 0;
std::uint32_t variant = 0;
std::uint32_t presenceLeaf = noLeaf;
};
template<class T>
consteval void CollectLeaves(std::vector<LeafInfo>& out, const FixedName& name, std::uint32_t presenceLeaf) {
constexpr Shape shape = ShapeOf<T>();
const std::uint8_t optionalFlag = presenceLeaf == noLeaf ? 0 : columnFlagOptional;
if constexpr (shape == Shape::Scalar) {
LeafInfo leaf{ name, ScalarKind<T>(), optionalFlag, sizeof(T), alignof(T) };
if constexpr (std::is_enum_v<T>) leaf.flags |= columnFlagEnum;
leaf.presenceLeaf = presenceLeaf;
out.push_back(leaf);
} else if constexpr (shape == Shape::FixedBytes) {
constexpr std::size_t n = [] {
if constexpr (FixedStringTraits<T>::value) return FixedStringTraits<T>::size;
else return StdArrayTraits<T>::size;
}();
static_assert(n > 0 && n <= 0xFFFF, "fixed byte columns must be 1..65535 bytes");
LeafInfo leaf{ name, Kind::FixedBytes, optionalFlag, static_cast<std::uint16_t>(n), 1, static_cast<std::uint32_t>(n), 0 };
leaf.presenceLeaf = presenceLeaf;
out.push_back(leaf);
} else if constexpr (shape == Shape::ScalarArray) {
for (std::size_t i = 0; i < StdArrayTraits<T>::size; i++) {
FixedName element = name;
element.Append("[");
char digits[20]{};
std::size_t len = 0;
std::size_t v = i;
do { digits[len++] = static_cast<char>('0' + v % 10); v /= 10; } while (v != 0);
for (std::size_t k = 0; k < len; k++) element.Append(std::string_view(&digits[len - 1 - k], 1));
element.Append("]");
CollectLeaves<typename StdArrayTraits<T>::Element>(out, element, presenceLeaf);
}
} else if constexpr (shape == Shape::Optional) {
FixedName presence = name;
presence.Append("?");
LeafInfo leaf{ presence, Kind::UInt8, static_cast<std::uint8_t>(columnFlagPresence | optionalFlag), 1, 1 };
leaf.presenceLeaf = presenceLeaf;
const std::uint32_t index = static_cast<std::uint32_t>(out.size());
out.push_back(leaf);
CollectLeaves<typename OptionalTraits<T>::Type>(out, name, index);
} else if constexpr (shape == Shape::Var) {
LeafInfo leaf{ name, Kind::String, optionalFlag, sizeof(Var), alignof(Var) };
if constexpr (HasFormatterKind<T>) leaf.kind = Formatter<T>::kind;
if constexpr (HasFormatterPeriod<T>) {
leaf.paramNum = Formatter<T>::paramNum;
leaf.paramDen = Formatter<T>::paramDen;
}
if constexpr (HasFormatterVariant<T>) leaf.variant = Formatter<T>::variant;
leaf.presenceLeaf = presenceLeaf;
out.push_back(leaf);
} else if constexpr (shape == Shape::Fixed) {
const std::size_t first = out.size();
using S = typename Formatter<T>::Stored;
CollectLeaves<S>(out, name, presenceLeaf);
if constexpr (HasFormatterKind<T>) {
static_assert(LeafCount<S>() == 1, "a Formatter may only override `kind` when Stored is a single column");
// A chrono type with a floating rep keeps its Float kind so the
// bytes are read back as what they are; the period still records
// that it is a time.
if constexpr (std::is_integral_v<S>) out[first].kind = Formatter<T>::kind;
}
if constexpr (HasFormatterPeriod<T>) {
out[first].paramNum = Formatter<T>::paramNum;
out[first].paramDen = Formatter<T>::paramDen;
}
if constexpr (HasFormatterVariant<T>) out[first].variant = Formatter<T>::variant;
} else if constexpr (shape == Shape::Aggregate) {
[&]<std::size_t... I>(std::index_sequence<I...>) {
((void)[&] {
FixedName member = name;
if (member.length != 0) member.Append(".");
member.Append(MemberName<I, T>());
CollectLeaves<MemberType<I, T>>(out, member, presenceLeaf);
}(), ...);
}(std::make_index_sequence<memberCount<T>>{});
} else {
LeafInfo leaf{ name, Kind::UInt8, optionalFlag, 1, 1 };
out.push_back(leaf);
}
}
template<class T>
consteval auto MakeLeaves() {
std::vector<LeafInfo> leaves;
CollectLeaves<T>(leaves, FixedName{}, noLeaf);
std::array<LeafInfo, LeafCount<T>()> out{};
for (std::size_t i = 0; i < out.size(); i++) out[i] = leaves[i];
return out;
}
// One placed column: a leaf with its row offset, or the flags byte.
struct ColumnInfo {
FixedName name;
Kind kind = Kind::Flags;
std::uint8_t flags = 0;
std::uint16_t size = 1;
std::uint16_t align = 1;
std::uint16_t offset = 0;
std::uint16_t presenceOffset = noColumn;
std::uint32_t paramNum = 0;
std::uint32_t paramDen = 0;
std::uint32_t variant = 0;
};
template<std::size_t LeafCountValue>
struct Arrangement {
std::array<ColumnInfo, LeafCountValue + 1> columns{};
std::array<std::uint16_t, LeafCountValue> leafOffset{};
std::array<std::uint16_t, LeafCountValue> leafColumn{};
std::uint32_t rowSize = 0;
std::uint16_t rowAlign = 1;
};
template<std::size_t N>
consteval Arrangement<N> Arrange(const std::array<LeafInfo, N>& leaves) {
Arrangement<N> result;
result.columns[0].name.Append("flags");
std::array<std::size_t, N> order{};
for (std::size_t i = 0; i < N; i++) order[i] = i;
std::ranges::stable_sort(order, [&](std::size_t a, std::size_t b) { return leaves[a].align < leaves[b].align; });
std::uint32_t cursor = 1;
std::uint16_t rowAlign = 1;
for (std::size_t k = 0; k < N; k++) {
const LeafInfo& leaf = leaves[order[k]];
const std::uint32_t align = leaf.align;
cursor = (cursor + align - 1) / align * align;
if (cursor + leaf.size > 0xFFFF) throw "a row must fit in 65535 bytes";
ColumnInfo& column = result.columns[k + 1];
column.name = leaf.name;
column.kind = leaf.kind;
column.flags = leaf.flags;
column.size = leaf.size;
column.align = leaf.align;
column.offset = static_cast<std::uint16_t>(cursor);
column.paramNum = leaf.paramNum;
column.paramDen = leaf.paramDen;
column.variant = leaf.variant;
result.leafOffset[order[k]] = static_cast<std::uint16_t>(cursor);
result.leafColumn[order[k]] = static_cast<std::uint16_t>(k + 1);
cursor += leaf.size;
if (leaf.align > rowAlign) rowAlign = leaf.align;
}
for (std::size_t k = 0; k < N; k++) {
const LeafInfo& leaf = leaves[order[k]];
if (leaf.presenceLeaf != noLeaf) result.columns[k + 1].presenceOffset = result.leafOffset[leaf.presenceLeaf];
}
result.rowAlign = rowAlign;
result.rowSize = (cursor + rowAlign - 1) / rowAlign * rowAlign;
return result;
}
template<std::size_t N>
consteval std::uint64_t FingerprintOf(const Arrangement<N>& arrangement, std::uint16_t orderedColumn) {
std::uint64_t hash = Fnv1a(fnvOffset, "crafter-db-v1");
hash = Fnv1a(hash, arrangement.rowSize);
hash = Fnv1a(hash, arrangement.rowAlign);
hash = Fnv1a(hash, orderedColumn);
for (const ColumnInfo& column : arrangement.columns) {
hash = Fnv1a(hash, column.name.View());
hash = Fnv1a(hash, static_cast<std::uint64_t>(column.kind));
hash = Fnv1a(hash, column.offset);
hash = Fnv1a(hash, column.size);
hash = Fnv1a(hash, column.flags);
hash = Fnv1a(hash, column.paramNum);
hash = Fnv1a(hash, column.paramDen);
hash = Fnv1a(hash, column.variant);
}
return hash;
}
template<std::size_t N>
consteval std::size_t NamePoolLength(const Arrangement<N>& arrangement) {
std::size_t total = 0;
for (const ColumnInfo& column : arrangement.columns) total += column.name.length;
return total;
}
// --- table-level declarations ---------------------------------------------
template<class T>
concept HasVersion = requires { { T::version } -> std::convertible_to<std::uint32_t>; } && std::is_integral_v<std::remove_cv_t<decltype(T::version)>>;
template<class T>
concept HasOrderedBy = requires { T::orderedBy; } && std::is_member_object_pointer_v<std::remove_cv_t<decltype(T::orderedBy)>>;
template<class T>
concept HasPrevious = requires { typename T::Previous; } && Aggregate<typename T::Previous>;
template<class T>
concept HasMigrate = HasPrevious<T> && requires(typename T::Previous&& previous) { { T::Migrate(std::move(previous)) } -> std::same_as<T>; };
// What `orderedBy` may point at: a type whose on-disk column is a single
// integer with a total order. Floats are out (NaN breaks the search),
// strings and fixed byte arrays are out (no binary search over them).
template<class M>
concept OrderableKey = (std::is_integral_v<M> && !std::is_same_v<M, bool>) || std::is_enum_v<M>
|| (FixedFormatter<M> && HasFormatterKind<M> && std::is_integral_v<typename Formatter<M>::Stored> && std::totally_ordered<M>);
// What a table struct must be. The diagnostics live in CheckTable.
template<class T>
concept TableType = Aggregate<T> && HasVersion<T> && std::is_default_constructible_v<T>;
// Everything the engine knows about T's row at compile time.
template<Aggregate T>
struct Layout {
static constexpr std::size_t leafCount = LeafCount<T>();
static constexpr std::size_t columnCount = leafCount + 1;
static constexpr std::array<LeafInfo, leafCount> leaves = MakeLeaves<T>();
static constexpr Arrangement<leafCount> arrangement = Arrange(leaves);
static constexpr const std::array<ColumnInfo, columnCount>& columns = arrangement.columns;
static constexpr std::uint32_t rowSize = arrangement.rowSize;
static constexpr std::uint16_t rowAlign = arrangement.rowAlign;
static_assert(columnCount <= 0xFFFF, "Crafter.DB: a table may have at most 65535 columns");
static_assert(NamePoolLength(arrangement) <= 0xFFFF, "Crafter.DB: the column names of a table may total at most 65535 bytes");
// Row offset of a leaf (leaf indices follow declaration order, depth first).
static constexpr std::uint16_t Offset(std::size_t leaf) noexcept { return arrangement.leafOffset[leaf]; }
static constexpr std::uint16_t Column(std::size_t leaf) noexcept { return arrangement.leafColumn[leaf]; }
static constexpr std::uint32_t version = [] {
if constexpr (HasVersion<T>) return static_cast<std::uint32_t>(T::version);
else return std::uint32_t(0);
}();
static constexpr std::size_t orderedMember = [] {
if constexpr (HasOrderedBy<T>) return MemberIndex<T, T::orderedBy>();
else return noMember;
}();
static constexpr std::uint16_t orderedColumn = [] {
if constexpr (HasOrderedBy<T>) return Column(LeafStart<T, MemberIndex<T, T::orderedBy>()>());
else return noColumn;
}();
static constexpr std::uint64_t fingerprint = FingerprintOf(arrangement, orderedColumn);
static constexpr bool hasHeap = std::ranges::any_of(columns, [](const ColumnInfo& c) { return c.kind == Kind::String || c.kind == Kind::Blob; });
};
// The file-side description of Layout<T>, byte-compatible with what
// SchemaInfo::Parse reads back; MakeSchemaInfo<T>().Fingerprint() equals
// Layout<T>::fingerprint by construction (a test pins this).
template<Aggregate T>
SchemaInfo MakeSchemaInfo() {
SchemaInfo info;
info.rowSize = Layout<T>::rowSize;
info.rowAlign = Layout<T>::rowAlign;
for (const ColumnInfo& column : Layout<T>::columns) {
ColumnDescriptor d{};
d.offset = column.offset;
d.size = column.size;
d.kind = column.kind;
d.flags = column.flags;
d.presenceOffset = column.presenceOffset;
d.nameOffset = static_cast<std::uint16_t>(info.namePool.size());
d.nameLength = static_cast<std::uint16_t>(column.name.length);
d.paramNum = column.paramNum;
d.paramDen = column.paramDen;
d.variant = column.variant;
info.namePool.append(column.name.View());
info.columns.push_back(d);
}
return info;
}
// --- diagnostics ---------------------------------------------------------
consteval MessageText Message(std::initializer_list<std::string_view> parts) {
MessageText message;
for (const std::string_view part : parts) message.Append(part);
return message;
}
template<class T, std::size_t I>
consteval MessageText MemberMessage(std::string_view problem, std::string_view hint) {
return Message({ "Crafter.DB: member '", MemberName<I, T>(), "' of ", TypeName<T>(), " ", problem, " ", hint });
}
template<class T>
consteval bool CheckMembers();
template<class T, std::size_t I>
consteval bool CheckMember() {
using M = MemberType<I, T>;
constexpr Shape shape = ShapeOf<M>();
static_assert(shape != Shape::Pointer, MemberMessage<T, I>("is a pointer or reference and cannot be stored;", "store a RowId, an index or a blob instead."));
static_assert(shape != Shape::Unsupported, MemberMessage<T, I>("has a type with no Crafter::DB::Formatter;", "specialise Crafter::DB::Formatter for it or make it an aggregate of storable members."));
if constexpr (shape == Shape::Aggregate) return CheckMembers<M>();
else if constexpr (shape == Shape::Optional) {
using P = typename OptionalTraits<M>::Type;
static_assert(ShapeOf<P>() != Shape::Pointer && ShapeOf<P>() != Shape::Unsupported, MemberMessage<T, I>("is a std::optional of an unsupported type;", "specialise Crafter::DB::Formatter for the payload type."));
static_assert(ShapeOf<P>() != Shape::Optional, MemberMessage<T, I>("is a std::optional of a std::optional;", "nested optionals have no distinct column names; wrap the inner one in a struct."));
if constexpr (ShapeOf<P>() == Shape::Aggregate) return CheckMembers<P>();
} else if constexpr (shape == Shape::Fixed) {
using S = typename Formatter<M>::Stored;
static_assert(ShapeOf<S>() != Shape::Pointer && ShapeOf<S>() != Shape::Unsupported && ShapeOf<S>() != Shape::Var && ShapeOf<S>() != Shape::Optional, MemberMessage<T, I>("has a Formatter whose Stored type is not storable;", "Stored must be a scalar, enum, std::array or aggregate of those."));
if constexpr (ShapeOf<S>() == Shape::Aggregate) return CheckMembers<S>();
}
return true;
}
template<class T>
consteval bool CheckMembers() {
return []<std::size_t... I>(std::index_sequence<I...>) { return (true && ... && CheckMember<T, I>()); }(std::make_index_sequence<memberCount<T>>{});
}
// Instantiated by Table<T>; every rule a table struct must follow is
// reported here with the table and member name in the message.
template<class T>
consteval bool CheckTable() {
static_assert(Aggregate<T>, Message({ "Crafter.DB: ", TypeName<T>(), " must be an aggregate: public data members only, no base classes with members, no user-declared constructors. Replace constructors with designated initialisers; Table::Emplace builds rows from fields." }));
static_assert(HasVersion<T>, Message({ "Crafter.DB: ", TypeName<T>(), " must declare `static constexpr std::uint32_t version = 1;`" }));
static_assert(std::is_default_constructible_v<T>, Message({ "Crafter.DB: ", TypeName<T>(), " must be default constructible (rows are decoded into a value-initialised instance)." }));
if constexpr (Aggregate<T>) {
CheckMembers<T>();
if constexpr (HasOrderedBy<T>) {
static_assert(std::is_same_v<MemberClass<T::orderedBy>, T>, Message({ "Crafter.DB: ", TypeName<T>(), "::orderedBy must point to a member of ", TypeName<T>() }));
static_assert(OrderableKey<MemberTypeOf<T::orderedBy>>, Message({ "Crafter.DB: ", TypeName<T>(), "::orderedBy must name an integer, enum or integral chrono member; floats (NaN has no order), strings, byte arrays and aggregates cannot be ordered." }));
}
if constexpr (HasPrevious<T>) {
static_assert(HasVersion<typename T::Previous>, Message({ "Crafter.DB: ", TypeName<T>(), "::Previous must itself declare `version`." }));
static_assert(Layout<typename T::Previous>::version < Layout<T>::version, Message({ "Crafter.DB: ", TypeName<T>(), "::Previous must have a lower version than ", TypeName<T>(), "; the Previous chain is strictly decreasing." }));
}
}
return true;
}
}
namespace Crafter::DB::SchemaTests {
enum class Currency : std::uint16_t { Euro };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
std::chrono::system_clock::time_point timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Transaction::timePoint;
};
static_assert(TableType<Transaction>);
static_assert(CheckTable<Transaction>());
static_assert(Layout<Transaction>::leafCount == 7);
static_assert(Layout<Transaction>::rowSize == 88);
static_assert(Layout<Transaction>::rowAlign == 8);
static_assert(Layout<Transaction>::columns[0].kind == Kind::Flags && Layout<Transaction>::columns[0].offset == 0);
static_assert(Layout<Transaction>::Offset(4) == 2); // currency: the only 2-byte column, right after the flags byte
static_assert(Layout<Transaction>::Offset(0) == 8); // name
static_assert(Layout<Transaction>::Offset(2) == 40); // iban
static_assert(Layout<Transaction>::Offset(5) == 72); // amount
static_assert(Layout<Transaction>::Offset(6) == 80); // timePoint
static_assert(Layout<Transaction>::columns[Layout<Transaction>::Column(6)].kind == Kind::TimePoint);
static_assert(Layout<Transaction>::columns[Layout<Transaction>::Column(6)].paramDen == 1000000);
static_assert(Layout<Transaction>::columns[Layout<Transaction>::Column(4)].flags == columnFlagEnum);
static_assert(Layout<Transaction>::columns[Layout<Transaction>::Column(2)].name.View() == "iban");
static_assert(Layout<Transaction>::orderedColumn == Layout<Transaction>::Column(6));
static_assert(Layout<Transaction>::hasHeap);
struct Money {
std::int64_t minor;
Currency currency;
};
struct Nested {
std::optional<std::string> note;
Money money;
std::array<std::uint16_t, 3> tags;
FixedString<11> bic;
std::vector<std::uint32_t> ids;
static constexpr std::uint32_t version = 2;
};
static_assert(CheckTable<Nested>());
static_assert(Layout<Nested>::leafCount == 2 + 2 + 3 + 1 + 1);
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(0)].name.View() == "note?");
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(1)].name.View() == "note");
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(1)].flags == columnFlagOptional);
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(1)].presenceOffset == Layout<Nested>::Offset(0));
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(2)].name.View() == "money.minor");
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(5)].name.View() == "tags[1]");
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(7)].kind == Kind::FixedBytes && Layout<Nested>::columns[Layout<Nested>::Column(7)].size == 11);
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(8)].kind == Kind::Blob && Layout<Nested>::columns[Layout<Nested>::Column(8)].paramDen == 4);
static_assert(Layout<Nested>::columns[Layout<Nested>::Column(8)].variant == static_cast<std::uint32_t>(Kind::UInt32));
struct FloatBlob {
std::vector<float> ids;
static constexpr std::uint32_t version = 1;
};
struct IntBlob {
std::vector<std::int32_t> ids;
static constexpr std::uint32_t version = 1;
};
static_assert(Layout<FloatBlob>::fingerprint != Layout<IntBlob>::fingerprint, "blob element type must be part of the fingerprint");
struct Unordered {
std::int64_t at;
std::string text;
static constexpr std::uint32_t version = 1;
};
struct Ordered {
std::int64_t at;
std::string text;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Ordered::at;
};
static_assert(Layout<Unordered>::fingerprint != Layout<Ordered>::fingerprint, "orderedBy must be part of the fingerprint");
}

201
interfaces/Crafter.DB-Storage.cppm Normal file
View file

@ -0,0 +1,201 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// The type-erased storage engine. A TableFile knows rows only as `rowSize`
// bytes and columns only through the descriptors the file carries, which is
// what lets it create, open, validate, recover, grow, compact and inspect any
// table without the C++ struct. Table<T> is a thin typed layer on top that
// computes column offsets at compile time and encodes/decodes through here.
export module Crafter.DB:Storage;
import std;
import :Error;
import :Format;
import :Formatter;
import :File;
export namespace Crafter::DB {
struct TableStats {
std::uint64_t slotCount = 0; // rows including tombstones
std::uint64_t liveCount = 0; // rows not deleted
std::uint64_t heapUsed = 0; // bytes of variable-length data written
std::uint64_t heapGarbage = 0; // of which orphaned by Erase/Update; Compact() reclaims
std::uint64_t fileLength = 0;
std::uint64_t commitSeq = 0;
};
struct CreateParams {
SchemaInfo schema;
std::uint32_t schemaVersion = 0;
std::uint64_t fingerprint = 0;
std::uint16_t orderedColumn = noColumn;
std::uint64_t initialRowBytes = 0; // size the first extents for a known payload (Compact, Migrate)
std::uint64_t initialHeapBytes = 0;
std::uint32_t migratedFromVersion = 0;
};
// Everything Inspect() can tell about a table without its struct.
struct Inspection {
FileHeader header{};
CommitRecord commit{};
SchemaInfo schema;
};
// One scalar column value read through a descriptor, for tools and Verify.
using ScalarValue = std::variant<std::int64_t, std::uint64_t, double>;
// A type-erased view of one row: column access by name or index through
// the file's own descriptors. This is what a dump tool or a migration from
// a struct that no longer exists uses.
class DynamicRow {
public:
DynamicRow() = default;
DynamicRow(const std::byte* row, const SchemaInfo* schema, const Heap* heap) noexcept : row(row), schema(schema), heap(heap) {}
bool Deleted() const noexcept;
std::size_t ColumnCount() const noexcept { return schema->columns.size(); }
const ColumnDescriptor& Column(std::size_t index) const noexcept { return schema->columns[index]; }
std::string_view Name(std::size_t index) const noexcept { return schema->Name(index); }
std::optional<std::size_t> Find(std::string_view name) const noexcept;
// False only for the payload of a disengaged std::optional.
bool Present(std::size_t index) const noexcept;
// Integers, bools, enums and chrono reps as int64/uint64; floats as double.
std::optional<ScalarValue> Scalar(std::size_t index) const noexcept;
// String, Blob and FixedBytes columns as raw bytes (bounds-checked).
std::optional<std::span<const std::byte>> Bytes(std::size_t index) const;
std::optional<std::string_view> Text(std::size_t index) const;
// "name=value, ..." for every column, for dumps and diagnostics.
std::string Describe() const;
private:
const std::byte* row = nullptr;
const SchemaInfo* schema = nullptr;
const Heap* heap = nullptr;
};
struct VerifyResult {
std::vector<std::string> problems;
bool Ok() const noexcept { return problems.empty(); }
};
class TableFile {
public:
// Writes a complete empty table to a temp file beside `path`, syncs it
// and renames it into place: a visible table file is always valid.
static void Create(const std::filesystem::path& path, const CreateParams& params, const OpenOptions& options);
static std::filesystem::path TempPath(const std::filesystem::path& path);
static void Replace(const std::filesystem::path& path, const std::filesystem::path& temp);
static std::filesystem::path BackupPath(const std::filesystem::path& path, std::uint32_t version);
static Inspection Inspect(const std::filesystem::path& path);
static void RemoveStaleTemps(const std::filesystem::path& path) noexcept;
TableFile(const std::filesystem::path& path, const OpenOptions& options);
TableFile(const TableFile&) = delete;
TableFile& operator=(const TableFile&) = delete;
~TableFile();
const FileHeader& Header() const noexcept { return header; }
const SchemaInfo& Schema() const noexcept { return schema; }
std::uint32_t RowSize() const noexcept { return header.rowSize; }
bool Writable() const noexcept { return writable; }
const std::filesystem::path& Path() const noexcept { return file.Path(); }
void SetPath(std::filesystem::path path) { file.SetPath(std::move(path)); }
TableStats Stats() const noexcept;
// Published counts. Readers take SlotCount() once per scan; rows below
// it are complete and never move.
std::uint64_t SlotCount() const noexcept { return publishedRows.load(std::memory_order_acquire); }
std::uint64_t LiveCount() const noexcept { return publishedLive.load(std::memory_order_relaxed); }
std::uint64_t HeapUsed() const noexcept { return publishedHeap.load(std::memory_order_acquire); }
// Set by the first Erase and never cleared while the file is open, so
// readers decide between the dense scan and the tombstone-checking
// scan from one monotonic flag rather than a racy pair of counters.
bool HasTombstones() const noexcept { return tombstones.load(std::memory_order_acquire); }
// False in a process that inherited this object through fork().
bool OwnedByThisProcess() const noexcept;
const std::byte* Rows() const noexcept { return rows.Base(); }
const std::byte* Slot(std::uint64_t index) const noexcept { return rows.Base() + index * header.rowSize; }
const Heap& HeapView() const noexcept { return heap; }
static bool Deleted(const std::byte* row) noexcept;
bool Deleted(std::uint64_t index) const noexcept { return Deleted(Slot(index)); }
// Append protocol: BeginAppend(n) returns the first of n writable slots
// (capacity guaranteed, contents undefined: callers encode into a
// zeroed buffer and copy). Variable-length bytes go through Writer().
// CommitAppend publishes; AbortAppend forgets the heap bytes.
std::byte* BeginAppend(std::uint64_t count);
HeapWriter& Writer() noexcept { return writer; }
void CommitAppend(std::uint64_t count) noexcept;
void AbortAppend() noexcept;
std::byte* MutableSlot(std::uint64_t index);
// Sets the tombstone bit. Returns false when the row was already deleted.
bool MarkDeleted(std::uint64_t index);
void AddHeapGarbage(std::uint64_t bytes) noexcept { heapGarbage += bytes; }
// Out-of-line bytes a row references (for garbage accounting).
std::uint64_t HeapBytesOf(const std::byte* row) const noexcept;
// Durability: fdatasync the data, then write the inactive commit slot.
void Flush(bool cleanClose = false);
void Close();
void RequireWritable() const;
// Streams the live rows into `target` (same schema, empty, writable),
// repacking their heap values in row order and dropping tombstones.
void CompactInto(TableFile& target) const;
DynamicRow Dynamic(std::uint64_t index) const noexcept { return { Slot(index), &schema, &heap }; }
// Full consistency scan: every handle inside the committed heap, the
// ordered column non-decreasing, live count equal to the recount,
// presence bytes 0 or 1. Returns the problems instead of throwing so a
// tool can print them all.
VerifyResult Verify() const;
private:
class Arena final : public HeapWriter {
public:
void Attach(TableFile* table, std::byte* arenaBase, std::uint64_t arenaCursor, std::uint64_t arenaCapacity) noexcept {
owner = table;
base = arenaBase;
cursor = arenaCursor;
capacity = arenaCapacity;
}
void SetCapacity(std::uint64_t arenaCapacity) noexcept { capacity = arenaCapacity; }
protected:
void Grow(std::uint64_t neededCapacity) override;
private:
TableFile* owner = nullptr;
};
void AddExtent(std::uint32_t region, std::uint64_t minimumBytes);
void EnsureHeapCapacity(std::uint64_t bytes);
void WriteCommit(bool cleanClose);
void RecountLive() noexcept;
File file;
OpenOptions options;
bool writable = false;
bool closed = false;
FileHeader header{};
SchemaInfo schema;
CommitRecord current{};
std::size_t activeSlot = 0;
MappedPages headerPages;
MappedRegion rows;
MappedRegion heapRegion;
Heap heap;
Arena writer;
std::atomic<std::uint64_t> publishedRows{ 0 };
std::atomic<std::uint64_t> publishedLive{ 0 };
std::atomic<std::uint64_t> publishedHeap{ 0 };
std::atomic<bool> tombstones{ false };
std::uint64_t heapGarbage = 0;
std::uint64_t fileLength = 0;
std::int64_t ownerProcess = 0;
std::array<std::uint32_t, regionSlots> extentCount{};
std::vector<std::size_t> varColumns;
};
}

440
interfaces/Crafter.DB-Table.cppm Normal file
View file

@ -0,0 +1,440 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Table<T>: the user-facing handle. Opening creates, validates and migrates;
// writes go through the typed encoders into the mapped file; reads are
// zero-copy Row<T> views; queries are lambdas or column expressions.
export module Crafter.DB:Table;
import std;
import :Error;
import :Reflect;
import :Format;
import :Formatter;
import :Schema;
import :File;
import :Storage;
import :Codec;
import :Row;
import :Query;
import :Migration;
export namespace Crafter::DB {
template<class T>
class Table {
static_assert(CheckTable<T>());
public:
using Value = T;
explicit Table(const std::filesystem::path& path, OpenOptions options = {}) : path(path), options(options) {
Open();
}
Table(Table&&) noexcept = default;
Table& operator=(Table&&) noexcept = default;
Table(const Table&) = delete;
Table& operator=(const Table&) = delete;
// --- information ------------------------------------------------------
const std::filesystem::path& Path() const noexcept { return path; }
TableStats Stats() const noexcept { return file->Stats(); }
// Live rows.
std::uint64_t Size() const noexcept { return file->LiveCount(); }
// Slots, tombstones included: the range of valid RowIds.
std::uint64_t SlotCount() const noexcept { return file->SlotCount(); }
bool Empty() const noexcept { return Size() == 0; }
bool ReadOnly() const noexcept { return !file->Writable(); }
TableFile& File() noexcept { return *file; }
const TableFile& File() const noexcept { return *file; }
// --- writing ----------------------------------------------------------
RowId Append(const T& row) {
const RowId id = AppendRows<T>(*file, std::span<const T>(&row, 1));
AutoFlush();
return id;
}
template<std::ranges::input_range R> requires std::convertible_to<std::ranges::range_reference_t<R>, const T&>
RowId Append(R&& rows) {
const RowId id = AppendRows<T>(*file, std::forward<R>(rows));
AutoFlush();
return id;
}
// One argument per member, in declaration order, encoded straight into
// the columns (a std::string_view is enough for a std::string member).
template<class... Args>
RowId Emplace(const Args&... args) {
static_assert(sizeof...(Args) == memberCount<T>, "Crafter.DB: Emplace takes exactly one argument per member of the table struct, in declaration order");
file->RequireWritable();
const RowId id = file->SlotCount();
if constexpr (HasOrderedBy<T>) {
std::optional<std::uint64_t> before;
if (id != 0) before = id - 1;
const OrderedKey<T> key = static_cast<OrderedKey<T>>(std::get<Layout<T>::orderedMember>(std::tie(args...)));
CheckOrder<T>(*file, key, before, std::nullopt);
}
std::byte* out = file->BeginAppend(1);
try {
EmplaceRow<T>(out, file->Writer(), args...);
} catch (...) {
file->AbortAppend();
throw;
}
file->CommitAppend(1);
AutoFlush();
return id;
}
// Replaces a live row in place. The new heap bytes are committed to
// disk before the row starts pointing at them, so a crash leaves
// either the old row or the new one, never a dangling handle. The row
// bytes themselves are not atomic against a power loss mid-write.
void Update(RowId id, const T& row) {
file->RequireWritable();
RequireLive(id);
if constexpr (HasOrderedBy<T>) CheckNeighbours(id, row.*T::orderedBy);
alignas(8) std::array<std::byte, Layout<T>::rowSize> buffer;
HeapWriter& heap = file->Writer();
const std::uint64_t heapBefore = heap.Cursor();
try {
EncodeRow<T>(row, buffer.data(), heap);
} catch (...) {
file->AbortAppend();
throw;
}
if (heap.Cursor() != heapBefore) {
file->CommitAppend(0);
file->Flush(false);
}
std::byte* slot = file->MutableSlot(id);
file->AddHeapGarbage(file->HeapBytesOf(slot));
std::memcpy(slot, buffer.data(), Layout<T>::rowSize);
AutoFlush();
}
// Sets one member of a live row, with the same durability order as
// Update. Only that member's columns are written, so a concurrent
// Erase of the row is not undone.
template<auto Ptr, class V> requires std::is_same_v<MemberClass<Ptr>, T>
void Set(RowId id, const V& value) {
constexpr std::size_t index = MemberIndex<T, Ptr>();
static_assert(index != noMember, "Crafter.DB: Set needs a pointer to a direct member of the table struct");
constexpr std::size_t first = LeafStart<T, index>();
constexpr std::size_t count = LeafCount<MemberType<index, T>>();
file->RequireWritable();
RequireLive(id);
if constexpr (HasOrderedBy<T>) {
if constexpr (SameMemberPointer<Ptr, T::orderedBy>()) CheckNeighbours(id, static_cast<OrderedKey<T>>(value));
}
std::byte* slot = file->MutableSlot(id);
alignas(8) std::array<std::byte, Layout<T>::rowSize> buffer;
std::memcpy(buffer.data(), slot, Layout<T>::rowSize);
HeapWriter& heap = file->Writer();
const std::uint64_t heapBefore = heap.Cursor();
try {
EncodeMember<T, index>(buffer.data(), heap, value);
} catch (...) {
file->AbortAppend();
throw;
}
if (heap.Cursor() != heapBefore) {
file->CommitAppend(0);
file->Flush(false);
}
file->AddHeapGarbage(MemberHeapBytes<T, index>(slot));
CopyLeaves<T, first, count>(slot, buffer.data());
AutoFlush();
}
template<auto Ptr, class V> requires std::is_same_v<MemberClass<Ptr>, T>
void Set(RowId id, Column<Ptr>, const V& value) {
Set<Ptr>(id, value);
}
// Tombstones the row. Returns false when it was already erased.
bool Erase(RowId id) {
file->RequireWritable();
RequireSlot(id);
const bool erased = file->MarkDeleted(id);
if (erased) AutoFlush();
return erased;
}
template<class Pred>
std::uint64_t EraseIf(Pred pred) {
file->RequireWritable();
std::uint64_t erased = 0;
const std::uint64_t slots = file->SlotCount();
for (std::uint64_t i = 0; i < slots; i++) {
const Row<T> row(file->Slot(i), file.get());
if (row.Erased() || !pred(row)) continue;
if (file->MarkDeleted(i)) erased++;
}
if (erased != 0) AutoFlush();
return erased;
}
void Flush() { file->Flush(false); }
// Flushes and closes now, reporting failure as an exception. The
// destructor does the same but cannot report, so call this when a
// failed final flush must not go unnoticed. The table is unusable after.
void Close() { file->Close(); }
// Consistency scan of the whole file (see TableFile::Verify).
VerifyResult Verify() const { return file->Verify(); }
// Rewrites the file without tombstones and heap garbage. Invalidates
// every RowId, Row and view taken before the call.
void Compact() {
file->RequireWritable();
const TableStats stats = file->Stats();
Rebuild([&](TableFile& target) { file->CompactInto(target); }, stats.liveCount * Layout<T>::rowSize, stats.heapUsed - std::min(stats.heapUsed, stats.heapGarbage));
}
// Removes every row. Invalidates RowIds, Rows and views.
void Clear() {
file->RequireWritable();
Rebuild([](TableFile&) {}, 0, 0);
}
// --- reading ----------------------------------------------------------
Row<T> operator[](RowId id) const noexcept { return { file->Slot(id), file.get() }; }
Row<T> At(RowId id) const {
RequireLive(id);
return (*this)[id];
}
bool Contains(RowId id) const noexcept { return id < file->SlotCount() && !file->Deleted(id); }
Row<T> Front() const {
for (Row<T> row : Live()) return row;
throw std::out_of_range("Table::Front on an empty table");
}
Row<T> Back() const {
std::uint64_t index = file->SlotCount();
while (index != 0) {
index--;
if (!file->Deleted(index)) return (*this)[index];
}
throw std::out_of_range("Table::Back on an empty table");
}
LiveRange<T> Live() const noexcept { return { file.get(), 0, file->SlotCount() }; }
LiveIterator<T> begin() const noexcept { return Live().begin(); }
std::default_sentinel_t end() const noexcept { return {}; }
SlotView<T> Slots() const noexcept { return { file.get(), 0, file->SlotCount() }; }
std::vector<T> Materialize() const {
std::vector<T> out;
out.reserve(Size());
for (Row<T> row : Live()) out.push_back(row.Materialize());
return out;
}
// Live rows of the slot range [first, last).
Query<T, LiveRange<T>> Rows(RowId first, RowId last) const noexcept {
const std::uint64_t end = std::min<std::uint64_t>(last, file->SlotCount());
return Query<T, LiveRange<T>>(LiveRange<T>(file.get(), std::min(first, end), end));
}
// Lazy filtered view. For a column expression on an orderedBy table the
// bounds on the ordered column are turned into a binary search first.
template<class Pred>
auto Where(Pred pred) const {
std::uint64_t first = 0;
std::uint64_t last = file->SlotCount();
if constexpr (HasOrderedBy<T> && Expression<Pred>) {
Bounds<OrderedKey<T>> bounds;
CollectBounds<T>(pred, bounds);
if (bounds.lower) first = bounds.lowerExclusive ? UpperBoundSlot<T>(*file, *bounds.lower, 0, last) : LowerBoundSlot<T>(*file, *bounds.lower, 0, last);
if (bounds.upper) last = bounds.upperExclusive ? LowerBoundSlot<T>(*file, *bounds.upper, first, last) : UpperBoundSlot<T>(*file, *bounds.upper, first, last);
if (last < first) last = first;
}
return Query<T, std::ranges::filter_view<LiveRange<T>, Pred>>(std::ranges::filter_view<LiveRange<T>, Pred>(LiveRange<T>(file.get(), first, last), std::move(pred)));
}
template<auto Ptr> requires std::is_same_v<MemberClass<Ptr>, T>
auto Select() const {
return std::views::transform(Live(), [](Row<T> row) { return row.template Get<Ptr>(); });
}
// --- ordered column ---------------------------------------------------
template<auto Ptr>
RowId LowerBound(const MemberTypeOf<Ptr>& key) const {
RequireOrdered<Ptr>();
return LowerBoundSlot<T>(*file, key, 0, file->SlotCount());
}
template<auto Ptr>
RowId UpperBound(const MemberTypeOf<Ptr>& key) const {
RequireOrdered<Ptr>();
return UpperBoundSlot<T>(*file, key, 0, file->SlotCount());
}
// Live rows with from <= key <= to.
template<auto Ptr>
Query<T, LiveRange<T>> Range(const MemberTypeOf<Ptr>& from, const MemberTypeOf<Ptr>& to) const {
RequireOrdered<Ptr>();
const std::uint64_t slots = file->SlotCount();
const std::uint64_t first = LowerBoundSlot<T>(*file, from, 0, slots);
const std::uint64_t last = UpperBoundSlot<T>(*file, to, first, slots);
return Rows(first, last);
}
// key >= from
template<auto Ptr>
Query<T, LiveRange<T>> From(const MemberTypeOf<Ptr>& from) const {
RequireOrdered<Ptr>();
return Rows(LowerBoundSlot<T>(*file, from, 0, file->SlotCount()), file->SlotCount());
}
// key > after
template<auto Ptr>
Query<T, LiveRange<T>> After(const MemberTypeOf<Ptr>& after) const {
RequireOrdered<Ptr>();
return Rows(UpperBoundSlot<T>(*file, after, 0, file->SlotCount()), file->SlotCount());
}
// key <= to
template<auto Ptr>
Query<T, LiveRange<T>> Until(const MemberTypeOf<Ptr>& to) const {
RequireOrdered<Ptr>();
return Rows(0, UpperBoundSlot<T>(*file, to, 0, file->SlotCount()));
}
// key < before
template<auto Ptr>
Query<T, LiveRange<T>> Before(const MemberTypeOf<Ptr>& before) const {
RequireOrdered<Ptr>();
return Rows(0, LowerBoundSlot<T>(*file, before, 0, file->SlotCount()));
}
private:
template<auto Ptr>
static consteval void RequireOrdered() {
static_assert(HasOrderedBy<T>, "Crafter.DB: Range/From/After/Until/Before/LowerBound/UpperBound need `static constexpr auto orderedBy = &T::member;` on the table struct");
if constexpr (HasOrderedBy<T>) {
static_assert(SameMemberPointer<Ptr, T::orderedBy>(), "Crafter.DB: binary search is only available on the member named by `orderedBy`; use Where() for other columns");
}
}
void CreateFile() {
CreateParams params;
params.schema = MakeSchemaInfo<T>();
params.schemaVersion = Layout<T>::version;
params.fingerprint = Layout<T>::fingerprint;
params.orderedColumn = Layout<T>::orderedColumn;
TableFile::Create(path, params, options); // IOError(file_exists) when the file is already there
}
// One open() does the work: the file's own header says whether it is
// current (done) or older (migrate, then reopen). A missing file is
// created with a rename that never replaces, so two processes racing
// to create the same table end up sharing one file.
void Open() {
if (options.mode == OpenMode::CreateNew) {
CreateFile();
file = std::make_unique<TableFile>(path, options);
return;
}
std::unique_ptr<TableFile> opened;
for (std::int32_t attempt = 0; !opened; attempt++) {
try {
opened = std::make_unique<TableFile>(path, options);
} catch (const IOError& error) {
if (error.code != std::errc::no_such_file_or_directory || options.mode == OpenMode::OpenExisting || options.readOnly || attempt != 0) throw;
try {
CreateFile();
file = std::make_unique<TableFile>(path, options);
return;
} catch (const IOError& created) {
if (created.code != std::errc::file_exists) throw;
// Lost the race: another process created it; open theirs.
}
}
}
if (opened->Header().schemaVersion != Layout<T>::version) {
Inspection info;
info.header = opened->Header();
info.schema = opened->Schema();
opened.reset(); // the migration takes the lock itself
MigrateTable<T>(path, info, options);
opened = std::make_unique<TableFile>(path, options);
if (opened->Header().schemaVersion != Layout<T>::version) {
throw SchemaError(std::format("{}: version {} on disk after migration, expected {}", path.string(), opened->Header().schemaVersion, Layout<T>::version));
}
}
RequireFingerprint<T>(path, opened->Header(), opened->Schema());
file = std::move(opened);
}
void AutoFlush() {
if (options.durability == Durability::EveryAppend) file->Flush(false);
}
void RequireSlot(RowId id) const {
if (id >= file->SlotCount()) throw std::out_of_range(std::format("RowId {} is outside the table ({} slots)", id, file->SlotCount()));
}
void RequireLive(RowId id) const {
RequireSlot(id);
if (file->Deleted(id)) throw std::out_of_range(std::format("RowId {} was erased", id));
}
template<class Key>
void CheckNeighbours(RowId id, const Key& key) {
std::optional<std::uint64_t> before;
std::optional<std::uint64_t> after;
if (id != 0) before = id - 1;
if (id + 1 < file->SlotCount()) after = id + 1;
CheckOrder<T>(*file, key, before, after);
}
template<class Fill>
void Rebuild(Fill&& fill, std::uint64_t initialRowBytes, std::uint64_t initialHeapBytes) {
CreateParams params;
params.schema = file->Schema();
params.schemaVersion = Layout<T>::version;
params.fingerprint = Layout<T>::fingerprint;
params.orderedColumn = Layout<T>::orderedColumn;
params.initialRowBytes = initialRowBytes;
params.initialHeapBytes = initialHeapBytes;
params.migratedFromVersion = file->Header().migratedFromVersion;
OpenOptions targetOptions = options;
targetOptions.createDirectories = false;
const std::filesystem::path temp = TableFile::TempPath(path);
TableFile::Create(temp, params, targetOptions);
std::unique_ptr<TableFile> target;
try {
target = std::make_unique<TableFile>(temp, targetOptions);
fill(*target);
target->Flush(false);
TableFile::Replace(path, temp);
} catch (...) {
target.reset();
Os::Remove(temp);
throw;
}
target->SetPath(path);
// The old inode is unlinked; a failing final flush of it is
// irrelevant and must not leave this Table without a file.
try {
file->Close();
} catch (...) {
}
file = std::move(target);
}
std::unique_ptr<TableFile> file;
std::filesystem::path path;
OpenOptions options;
};
}

17
interfaces/Crafter.DB.cppm Normal file
View file

@ -0,0 +1,17 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
export module Crafter.DB;
export import :Error;
export import :Reflect;
export import :Format;
export import :Formatter;
export import :Schema;
export import :File;
export import :Storage;
export import :Codec;
export import :Row;
export import :Query;
export import :Migration;
export import :Table;

53
project.cpp Normal file
View file

@ -0,0 +1,53 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
Configuration cfg;
cfg.path = "./";
cfg.name = "Crafter.DB";
cfg.outputName = "Crafter.DB";
cfg.type = ConfigurationType::LibraryStatic;
ApplyStandardArgs(cfg, args);
std::array<fs::path, 13> ifaces = {
"interfaces/Crafter.DB",
"interfaces/Crafter.DB-Error",
"interfaces/Crafter.DB-Reflect",
"interfaces/Crafter.DB-Format",
"interfaces/Crafter.DB-Formatter",
"interfaces/Crafter.DB-Schema",
"interfaces/Crafter.DB-File",
"interfaces/Crafter.DB-Storage",
"interfaces/Crafter.DB-Codec",
"interfaces/Crafter.DB-Row",
"interfaces/Crafter.DB-Query",
"interfaces/Crafter.DB-Migration",
"interfaces/Crafter.DB-Table",
};
std::array<fs::path, 2> impls = { "implementations/Crafter.DB-File", "implementations/Crafter.DB-Storage" };
cfg.GetInterfacesAndImplementations(ifaces, impls);
// Tests link the local static library. Linux only: the engine uses
// flock, fallocate and fixed-address mmap.
if (cfg.target.ends_with("-linux-gnu")) {
cfg.AddTest("ShouldRoundTrip").Dependencies({ &cfg });
cfg.AddTest("ShouldKeepLayoutGolden").Dependencies({ &cfg });
cfg.AddTest("ShouldKeepViewsStableAcrossGrowth").Dependencies({ &cfg });
cfg.AddTest("ShouldQuery").Dependencies({ &cfg });
cfg.AddTest("ShouldCompareStringRefs").Dependencies({ &cfg });
cfg.AddTest("ShouldEraseAndCompact").Dependencies({ &cfg });
cfg.AddTest("ShouldMigrate").Dependencies({ &cfg });
cfg.AddTest("ShouldRejectSchemaChange").Dependencies({ &cfg });
cfg.AddTest("ShouldRecoverCommits").Dependencies({ &cfg });
cfg.AddTest("ShouldFormatCustomTypes").Dependencies({ &cfg });
cfg.AddTest("ShouldLockAcrossProcesses").Dependencies({ &cfg });
cfg.AddTest("ShouldVerify").Dependencies({ &cfg });
cfg.AddTest("ShouldReadWhileAppending").Dependencies({ &cfg });
cfg.AddTest("Benchmark").Dependencies({ &cfg });
}
return cfg;
}

245
tests/Benchmark/main.cpp Normal file
View file

@ -0,0 +1,245 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Crafter.DB against a cleaned-up copy of Nostro's hand-rolled format
// (iostream, uint16-prefixed strings inline, linear scans) on the five
// operations Nostro performs. Prints a table; never fails. Runs on 100k
// rows so a full suite stays fast; raise `rowCount` for a serious run.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
using Clock = std::chrono::steady_clock;
namespace {
enum class Currency : std::uint16_t { Euro };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Transaction::timePoint;
};
inline constexpr DB::Column<&Transaction::iban> iban{};
inline constexpr DB::Column<&Transaction::timePoint> timePoint{};
constexpr std::uint64_t rowCount = 100'000;
std::vector<Transaction> MakeRows() {
std::mt19937_64 rng(42);
const std::array<std::string_view, 6> names = { "Albert Heijn", "Jumbo Supermarkten", "NS Reizigers", "Vattenfall", "Bol.com", "Gemeente Utrecht" };
const std::array<std::string_view, 4> bics = { "ABNANL2A", "RABONL2U", "INGBNL2A", "TRIONL2U" };
std::vector<std::string> ibans;
for (std::uint32_t i = 0; i < 1000; i++) ibans.push_back(std::format("NL{:02}ABNA{:010}", i % 97, 417164300u + i));
std::vector<Transaction> rows;
rows.reserve(rowCount);
for (std::uint64_t i = 0; i < rowCount; i++) {
rows.push_back({ std::string(names[rng() % names.size()]), std::string(bics[rng() % bics.size()]), ibans[rng() % ibans.size()], std::format("payment {} for order {}", i, rng() % 100000), Currency::Euro, static_cast<std::int64_t>(rng() % 200000) - 100000, TimePoint(std::chrono::microseconds(1'700'000'000'000'000 + i * 60'000'000)) });
}
return rows;
}
// --- the Nostro format, cleaned up -----------------------------------------
namespace Nostro {
void WriteString(std::ofstream& file, std::string_view str) {
const std::uint16_t size = static_cast<std::uint16_t>(str.size());
file.write(reinterpret_cast<const char*>(&size), sizeof(size));
file.write(str.data(), size);
}
void ReadString(std::ifstream& file, std::string& str) {
std::uint16_t size;
file.read(reinterpret_cast<char*>(&size), sizeof(size));
str.resize(size);
file.read(str.data(), size);
}
void Save(const fs::path& path, std::span<const Transaction> rows) {
std::ofstream file(path, std::ios::binary | std::ios::app);
for (const Transaction& tr : rows) {
WriteString(file, tr.name);
WriteString(file, tr.bic);
WriteString(file, tr.iban);
WriteString(file, tr.description);
file.write(reinterpret_cast<const char*>(&tr.amount), sizeof(tr.amount));
file.write(reinterpret_cast<const char*>(&tr.currency), sizeof(Currency));
const std::int64_t time = std::chrono::duration_cast<std::chrono::microseconds>(tr.timePoint.time_since_epoch()).count();
file.write(reinterpret_cast<const char*>(&time), sizeof(time));
}
}
bool Load(std::ifstream& file, Transaction& tr) {
if (file.peek() == std::ifstream::traits_type::eof()) return false;
ReadString(file, tr.name);
ReadString(file, tr.bic);
ReadString(file, tr.iban);
ReadString(file, tr.description);
file.read(reinterpret_cast<char*>(&tr.amount), sizeof(tr.amount));
file.read(reinterpret_cast<char*>(&tr.currency), sizeof(tr.currency));
std::int64_t time;
file.read(reinterpret_cast<char*>(&time), sizeof(time));
tr.timePoint = TimePoint(std::chrono::microseconds(time));
return true;
}
TimePoint LastTime(const fs::path& path) {
std::ifstream file(path, std::ios::binary);
file.seekg(-static_cast<std::streamoff>(sizeof(std::int64_t)), std::ios::end);
std::int64_t time;
file.read(reinterpret_cast<char*>(&time), sizeof(time));
return TimePoint(std::chrono::microseconds(time));
}
std::vector<Transaction> LoadAll(const fs::path& path) {
std::ifstream file(path, std::ios::binary);
std::vector<Transaction> out;
Transaction tr;
while (Load(file, tr)) out.push_back(tr);
return out;
}
std::vector<Transaction> LoadRange(const fs::path& path, TimePoint from, TimePoint to) {
std::ifstream file(path, std::ios::binary);
std::vector<Transaction> out;
Transaction tr;
while (Load(file, tr)) {
if (tr.timePoint >= from && tr.timePoint <= to) out.push_back(tr);
if (tr.timePoint > to) break;
}
return out;
}
std::vector<Transaction> LoadByIban(const fs::path& path, std::string_view wanted) {
std::ifstream file(path, std::ios::binary);
std::vector<Transaction> out;
Transaction tr;
while (Load(file, tr)) {
if (tr.iban == wanted) out.push_back(tr);
}
return out;
}
}
template<class F>
double Millis(F&& f) {
const Clock::time_point start = Clock::now();
f();
return std::chrono::duration<double, std::milli>(Clock::now() - start).count();
}
struct Line {
std::string name;
double nostro;
double crafter;
std::string note;
};
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-benchmark";
fs::remove_all(dir);
fs::create_directories(dir);
const fs::path nostroPath = dir / "nostro.bin";
const fs::path crafterPath = dir / "crafter.cdb";
const std::vector<Transaction> rows = MakeRows();
const std::string_view wantedIban = rows[rowCount / 2].iban;
const TimePoint from = rows[rowCount * 45 / 100].timePoint;
const TimePoint to = rows[rowCount * 55 / 100].timePoint;
std::vector<Line> lines;
std::size_t sink = 0;
// append
{
const double nostro = Millis([&] { Nostro::Save(nostroPath, rows); });
const double crafter = Millis([&] {
DB::Table<Transaction> table(crafterPath);
table.Append(rows);
table.Flush();
});
lines.push_back({ std::format("append {} rows", rowCount), nostro, crafter, "incl. open, Crafter.DB incl. fdatasync" });
}
// last timestamp (1000 opens each)
{
const double nostro = Millis([&] { for (std::int32_t i = 0; i < 1000; i++) sink += Nostro::LastTime(nostroPath).time_since_epoch().count() & 1; });
const double crafter = Millis([&] {
for (std::int32_t i = 0; i < 1000; i++) {
const DB::Table<Transaction> table(crafterPath, { .readOnly = true });
sink += table.Back()[&Transaction::timePoint].time_since_epoch().count() & 1;
}
});
lines.push_back({ "last timestamp (1000x open+read)", nostro, crafter, "open dominates both" });
}
DB::Table<Transaction> table(crafterPath, { .readOnly = true });
// load all
{
const double nostro = Millis([&] { sink += Nostro::LoadAll(nostroPath).size(); });
const double crafter = Millis([&] { sink += table.Materialize().size(); });
double zeroCopy = Millis([&] {
std::int64_t total = 0;
for (DB::Row<Transaction> row : table) total += row[&Transaction::amount] + static_cast<std::int64_t>(row[&Transaction::iban].Length());
sink += static_cast<std::size_t>(total & 1);
});
lines.push_back({ "load all (materialised)", nostro, crafter, std::format("zero-copy scan of two columns: {:.2f} ms", zeroCopy) });
}
// time range (10% of rows)
{
std::size_t nostroCount = 0;
std::size_t crafterCount = 0;
const double nostro = Millis([&] { nostroCount = Nostro::LoadRange(nostroPath, from, to).size(); });
const double crafter = Millis([&] { crafterCount = table.Range<&Transaction::timePoint>(from, to).Materialize().size(); });
lines.push_back({ "time range (10% of rows, materialised)", nostro, crafter, std::format("{} / {} rows", nostroCount, crafterCount) });
}
// iban ==
{
std::size_t nostroCount = 0;
std::size_t keyCount = 0;
std::size_t exprCount = 0;
std::size_t viewCount = 0;
const double nostro = Millis([&] { nostroCount = Nostro::LoadByIban(nostroPath, wantedIban).size(); });
const double withKey = Millis([&] {
const DB::Key key(wantedIban);
keyCount = table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == key; }).Count();
});
const double withExpression = Millis([&] { exprCount = table.Where(iban == wantedIban).Count(); });
const double withView = Millis([&] { viewCount = table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == wantedIban; }).Count(); });
lines.push_back({ "iban == x (lambda + Key)", nostro, withKey, std::format("{} / {} rows, {:.1f} ns/row", nostroCount, keyCount, withKey * 1e6 / rowCount) });
lines.push_back({ "iban == x (expression)", nostro, withExpression, std::format("{} rows, {:.1f} ns/row", exprCount, withExpression * 1e6 / rowCount) });
lines.push_back({ "iban == x (lambda + string_view)", nostro, withView, std::format("{} rows, {:.1f} ns/row", viewCount, withView * 1e6 / rowCount) });
}
// range + iban
{
std::size_t nostroCount = 0;
std::size_t crafterCount = 0;
const double nostro = Millis([&] {
for (const Transaction& tr : Nostro::LoadRange(nostroPath, from, to)) nostroCount += tr.iban == wantedIban ? 1 : 0;
});
const double crafter = Millis([&] { crafterCount = table.Where(timePoint >= from && timePoint <= to && iban == wantedIban).Count(); });
lines.push_back({ "time range + iban == x", nostro, crafter, std::format("{} / {} rows", nostroCount, crafterCount) });
}
std::println("{:<42} {:>12} {:>12} {:>8} {}", "operation", "nostro ms", "crafter ms", "speedup", "note");
for (const Line& line : lines) {
std::println("{:<42} {:>12.3f} {:>12.3f} {:>7.1f}x {}", line.name, line.nostro, line.crafter, line.nostro / std::max(line.crafter, 0.0001), line.note);
}
std::println("files: nostro {} bytes, crafter {} bytes", fs::file_size(nostroPath), fs::file_size(crafterPath));
std::println("(sink {})", sink & 1);
fs::remove_all(dir);
return 0;
}

97
tests/ShouldCompareStringRefs/main.cpp Normal file
View file

@ -0,0 +1,97 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// StringRef and BlobRef semantics: equality against string_view, Key,
// std::string, literals and other refs across the inline/heap boundary,
// prefix-equal non-matches, ordering, implicit conversion, and blob element
// alignment for 4- and 8-byte elements.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
struct Strings {
std::string a;
std::string b;
std::vector<std::uint32_t> words;
std::vector<double> doubles;
std::vector<std::byte> bytes;
static constexpr std::uint32_t version = 1;
};
std::string_view V(std::string_view s) { return s; }
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-stringref";
fs::remove_all(dir);
DB::Table<Strings> table(dir / "s.cdb");
const std::vector<std::string> values = { "", "abc", "abcd", "abcde", "twelve_chars", "thirteen_char", "a much longer value that lives in the heap region" };
for (const std::string& v : values) table.Append({ v, v, { 1, 2, 3 }, { 1.5, 2.5 }, { std::byte{ 1 } } });
table.Append({ "abcd", "abcX", { 7, 8, 9, 10, 11 }, {}, {} }); // same length, same 3-byte prefix, differs in the 4th byte
table.Append({ "thirteen_char", "thirteen_chaX", {}, { 3.5 }, {} });
for (std::size_t i = 0; i < values.size(); i++) {
const DB::StringRef ref = table[i][&Strings::a];
const std::string& v = values[i];
Check(ref == V(v), std::format("== string_view for '{}'", v));
Check(ref == v, std::format("== std::string for '{}'", v));
Check(ref == DB::Key(v), std::format("== Key for '{}'", v));
Check(ref == table[i][&Strings::b], std::format("== StringRef for '{}'", v));
Check(!(ref != V(v)), std::format("!= false for '{}'", v));
Check(std::string_view(ref) == v, std::format("implicit string_view for '{}'", v));
Check(ref.Length() == v.size(), std::format("Length for '{}'", v));
Check(ref.Empty() == v.empty(), std::format("Empty for '{}'", v));
for (std::size_t j = 0; j < values.size(); j++) {
if (j == i) continue;
Check(!(ref == V(values[j])), std::format("'{}' != '{}'", v, values[j]));
Check(!(ref == DB::Key(values[j])), std::format("'{}' != Key '{}'", v, values[j]));
Check(!(ref == table[j][&Strings::a]), std::format("'{}' != ref '{}'", v, values[j]));
}
}
Check(!(table[7][&Strings::a] == table[7][&Strings::b]), "prefix-equal inline strings differ in the tail");
Check(!(table[8][&Strings::a] == table[8][&Strings::b]), "prefix-equal heap strings differ in the last byte");
Check(table[8][&Strings::a] == V("thirteen_char") && !(table[8][&Strings::a] == V("thirteen_chaX")), "heap compare checks all bytes");
Check((table[1][&Strings::a] <=> V("abd")) == std::strong_ordering::less, "ordering less");
Check((table[1][&Strings::a] <=> V("abb")) == std::strong_ordering::greater, "ordering greater");
Check((table[1][&Strings::a] <=> V("abc")) == std::strong_ordering::equal, "ordering equal");
Check(table[1][&Strings::a] < V("abcd"), "prefix shorter sorts first");
Check(table[6][&Strings::a] < table[1][&Strings::a], "ref <=> ref (space sorts before b)");
Check(table[1][&Strings::a] == "abc", "== literal");
const DB::BlobRef<std::uint32_t> words = table[0][&Strings::words];
const std::span<const std::uint32_t> wordSpan = words;
Check(wordSpan.size() == 3 && wordSpan[0] == 1 && wordSpan[2] == 3, "u32 blob round trip (inline)");
Check(reinterpret_cast<std::uintptr_t>(wordSpan.data()) % alignof(std::uint32_t) == 0, "inline u32 blob is 4-aligned");
const std::span<const std::uint32_t> longWords = table[7][&Strings::words];
Check(longWords.size() == 5 && longWords[4] == 11, "u32 blob round trip (heap)");
Check(reinterpret_cast<std::uintptr_t>(longWords.data()) % alignof(std::uint32_t) == 0, "heap u32 blob is 4-aligned");
const std::span<const double> doubles = table[0][&Strings::doubles];
Check(doubles.size() == 2 && doubles[1] == 2.5, "double blob round trip");
Check(reinterpret_cast<std::uintptr_t>(doubles.data()) % alignof(double) == 0, "double blob is 8-aligned (never inline)");
const std::span<const double> one = table[8][&Strings::doubles];
Check(one.size() == 1 && one[0] == 3.5, "single double blob");
const std::span<const std::byte> bytes = table[0][&Strings::bytes];
Check(bytes.size() == 1 && bytes[0] == std::byte{ 1 }, "byte blob round trip");
Check(table[7][&Strings::bytes].Size() == 0, "empty blob");
const Strings copy = table[7].Materialize();
Check(copy.words == std::vector<std::uint32_t>{ 7, 8, 9, 10, 11 } && copy.doubles.empty(), "Materialize blobs");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

160
tests/ShouldEraseAndCompact/main.cpp Normal file
View file

@ -0,0 +1,160 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Tombstones and compaction: erased rows disappear from iteration while
// RowIds stay put, Back() skips trailing tombstones, Compact renumbers
// densely and reclaims heap garbage, Clear empties, a stale temp file is
// removed on open, and views taken before Compact are not used after it.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
struct Note {
std::string text;
std::int32_t rank;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Note::rank;
friend bool operator==(const Note&, const Note&) = default;
};
struct Memo {
std::optional<std::string> text;
std::int32_t id;
static constexpr std::uint32_t version = 1;
};
std::vector<std::int32_t> Ranks(const DB::Table<Note>& table) {
std::vector<std::int32_t> ranks;
for (DB::Row<Note> row : table) ranks.push_back(row[&Note::rank]);
return ranks;
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-compact";
fs::remove_all(dir);
const fs::path path = dir / "notes.cdb";
{
DB::Table<Note> table(path);
for (std::int32_t i = 0; i < 10; i++) table.Append({ std::format("note number {} with a long body", i), i });
const std::uint64_t heapBefore = table.Stats().heapUsed;
Check(table.Erase(2), "Erase returns true");
Check(!table.Erase(2), "second Erase returns false");
table.Erase(5);
table.Erase(9);
CheckEqual<std::uint64_t>(table.Size(), 7, "Size after erase");
CheckEqual<std::uint64_t>(table.SlotCount(), 10, "SlotCount unchanged");
CheckEqual(Ranks(table), std::vector<std::int32_t>{ 0, 1, 3, 4, 6, 7, 8 }, "iteration skips tombstones");
Check(table[7][&Note::rank] == 7, "RowIds stable after erase");
Check(table.Back()[&Note::rank] == 8, "Back skips a trailing tombstone");
Check(table.Front()[&Note::rank] == 0, "Front");
Check(!table.Contains(2) && table.Contains(3) && !table.Contains(10), "Contains");
Check(table[2].Erased(), "Erased flag visible through Slots");
bool threw = false;
try {
table.At(2);
} catch (const std::out_of_range&) {
threw = true;
}
Check(threw, "At on an erased row throws");
Check(table.Stats().heapGarbage == 3 * std::string("note number 0 with a long body").size(), "erase accounts heap garbage");
CheckEqual<std::uint64_t>(table.EraseIf([](DB::Row<Note> r) { return r[&Note::rank] % 4 == 3; }), 2, "EraseIf count (3 and 7)");
CheckEqual(Ranks(table), std::vector<std::int32_t>{ 0, 1, 4, 6, 8 }, "EraseIf result");
table.Update(4, { "four", 4 });
Check(table[4][&Note::text] == "four", "Update in place");
const std::uint64_t sizeBefore = fs::file_size(path);
table.Flush();
table.Compact();
CheckEqual<std::uint64_t>(table.Size(), 5, "Size after Compact");
CheckEqual<std::uint64_t>(table.SlotCount(), 5, "SlotCount after Compact");
CheckEqual(Ranks(table), std::vector<std::int32_t>{ 0, 1, 4, 6, 8 }, "order preserved by Compact");
Check(table[2][&Note::text] == "four", "rows renumbered densely");
Check(table[4][&Note::text] == "note number 8 with a long body", "heap values repacked correctly");
CheckEqual<std::uint64_t>(table.Stats().heapGarbage, 0, "no garbage after Compact");
Check(table.Stats().heapUsed < heapBefore, "heap shrank");
Check(fs::file_size(path) <= sizeBefore, "file did not grow");
table.Append({ "after compact", 9 });
CheckEqual<std::uint64_t>(table.Size(), 6, "append after Compact");
CheckEqual<std::uint64_t>(table.Back().Id(), 5, "new row gets the next dense id");
Check(table.Back()[&Note::text] == "after compact", "append after Compact readable");
}
{
// A leftover temp file (crash during Compact) is cleaned on the next open.
const fs::path stale = fs::path(path.string() + ".tmp-deadbeef0000");
{ std::ofstream(stale) << "junk"; }
DB::Table<Note> table(path);
Check(!fs::exists(stale), "stale temp file removed on open");
CheckEqual<std::uint64_t>(table.Size(), 6, "data intact after reopen");
table.Clear();
CheckEqual<std::uint64_t>(table.Size(), 0, "Clear empties the table");
CheckEqual<std::uint64_t>(table.SlotCount(), 0, "Clear resets slots");
table.Append({ "fresh", 1 });
CheckEqual<std::uint64_t>(table.Back().Id(), 0, "ids restart after Clear");
}
{
const DB::Table<Note> table(path, { .readOnly = true });
CheckEqual<std::uint64_t>(table.Size(), 1, "Clear + append persisted");
Check(table[0][&Note::text] == "fresh", "row persisted");
Check(fs::file_size(path) <= 3 * 65536, std::format("Clear did not pre-size the new file to the old data ({} bytes)", fs::file_size(path)));
const auto plain = table.Rows(0, 1);
std::uint64_t seen = 0;
for (DB::Row<Note> row : plain) seen += row[&Note::rank];
CheckEqual<std::uint64_t>(seen, 1, "const Query over a LiveRange iterates");
CheckEqual<std::uint64_t>(plain.Count(), 1, "Count on a const Query");
CheckEqual<std::uint64_t>(table.Rows(0, 1).Count(), 1, "Count on a prvalue Query");
static_assert(std::ranges::range<const decltype(plain)>, "a Query over a plain range is const-iterable");
static_assert(!std::ranges::range<const decltype(table.Where([](DB::Row<Note>) { return true; }))>, "a filter-backed Query is not const-iterable (filter_view has no const begin)");
Check(table.Where([](DB::Row<Note> r) { return r[&Note::rank] == 1; }).Any(), "Any on a prvalue filter Query");
}
{
// A disengaged optional must not keep its old heap bytes alive.
const fs::path memos = dir / "memos.cdb";
DB::Table<Memo> table(memos);
table.Append({ std::string("a note long enough to live in the heap"), 1 });
const std::uint64_t used = table.Stats().heapUsed;
Check(used > 0, "optional string went to the heap");
table.Set<&Memo::text>(0, std::optional<std::string>());
Check(!table[0][&Memo::text].has_value(), "optional cleared");
CheckEqual<std::uint64_t>(table.Stats().heapGarbage, used, "clearing the optional orphans its heap bytes");
table.Compact();
CheckEqual<std::uint64_t>(table.Stats().heapUsed, 0, "Compact dropped the dead payload");
DB::Var payload;
std::memcpy(&payload, table[0].Bytes() + DB::Layout<Memo>::Offset(1), sizeof(payload));
Check(payload.length == 0 && payload.tailOrOffset == 0, "payload handle is zero after clearing");
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

187
tests/ShouldFormatCustomTypes/main.cpp Normal file
View file

@ -0,0 +1,187 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Every built-in member shape and both user formatter shapes round-trip:
// fixed (a class stored as basis points), variable-length inheriting from the
// std::string formatter, optionals of scalars and strings, nested aggregates
// with strings, scalar arrays, byte arrays, FixedString, chrono durations,
// bool, double, std::byte and blobs.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
class Percent {
public:
Percent() = default;
explicit Percent(double value) : basisPoints(static_cast<std::int32_t>(std::lround(value * 100))) {}
double Value() const noexcept { return basisPoints / 100.0; }
std::int32_t BasisPoints() const noexcept { return basisPoints; }
friend bool operator==(const Percent&, const Percent&) = default;
private:
std::int32_t basisPoints = 0;
};
class Tags {
public:
Tags() = default;
explicit Tags(std::vector<std::string> values) : values(std::move(values)) {}
const std::vector<std::string>& Values() const noexcept { return values; }
std::string Joined() const {
std::string out;
for (const std::string& v : values) {
if (!out.empty()) out += ',';
out += v;
}
return out;
}
static Tags Split(std::string_view text) {
std::vector<std::string> values;
for (const auto part : std::views::split(text, ',')) {
if (!part.empty()) values.emplace_back(part.begin(), part.end());
}
return Tags(std::move(values));
}
friend bool operator==(const Tags&, const Tags&) = default;
private:
std::vector<std::string> values;
};
}
template<>
struct Crafter::DB::Formatter<Percent> {
using Stored = std::int32_t;
static Stored Encode(const Percent& value) noexcept { return value.BasisPoints(); }
static Percent Decode(const Stored& stored) noexcept { return Percent(stored / 100.0); }
};
template<>
struct Crafter::DB::Formatter<Tags> : Crafter::DB::Formatter<std::string> {
static Var Encode(const Tags& value, HeapWriter& heap) { return Formatter<std::string>::Encode(value.Joined(), heap); }
static Tags Materialize(StringRef view) { return Tags::Split(view.View()); }
};
namespace {
enum class Currency : std::uint16_t { Euro, Dollar };
struct Money {
std::int64_t minor;
Currency currency;
friend bool operator==(const Money&, const Money&) = default;
};
struct Party {
std::string name;
DB::FixedString<11> bic;
std::optional<std::string> note;
friend bool operator==(const Party&, const Party&) = default;
};
struct Everything {
bool flag;
std::byte raw;
double ratio;
std::chrono::milliseconds delay;
std::chrono::duration<double> seconds;
Percent fee;
Tags tags;
std::optional<std::int32_t> maybe;
std::optional<std::string> maybeText;
Money money;
Party party;
std::array<std::uint16_t, 3> triple;
std::array<char, 4> code;
DB::FixedString<34> iban;
std::vector<std::uint32_t> ids;
static constexpr std::uint32_t version = 7;
friend bool operator==(const Everything&, const Everything&) = default;
};
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-formatters";
fs::remove_all(dir);
const fs::path path = dir / "e.cdb";
const Everything a{ true, std::byte{ 7 }, 0.25, std::chrono::milliseconds(1500), std::chrono::duration<double>(2.5), Percent(12.5), Tags({ "food", "monthly" }), 42, std::string("present"), { -1234, Currency::Dollar }, { "Shop", DB::FixedString<11>("ABNANL2AXXX"), std::string("vip customer with a long note") }, { 1, 2, 3 }, { 'E', 'U', 'R', '\0' }, DB::FixedString<34>("NL91ABNA0417164300"), { 10, 20, 30, 40, 50 } };
const Everything b{ false, std::byte{ 0 }, -1.5, std::chrono::milliseconds(0), std::chrono::duration<double>(0), Percent(0), Tags(std::vector<std::string>{}), std::nullopt, std::nullopt, { 0, Currency::Euro }, { "", DB::FixedString<11>(""), std::nullopt }, { 0, 0, 0 }, { 'U', 'S', 'D', '\0' }, DB::FixedString<34>(""), {} };
{
DB::Table<Everything> table(path);
table.Append(a);
table.Append(b);
const DB::Row<Everything> ra = table[0];
Check(ra[&Everything::flag] == true, "bool");
Check(ra[&Everything::raw] == std::byte{ 7 }, "std::byte");
Check(ra[&Everything::ratio] == 0.25, "double");
Check(ra[&Everything::delay] == std::chrono::milliseconds(1500), "duration");
Check(ra[&Everything::seconds] == std::chrono::duration<double>(2.5), "floating duration");
static_assert(DB::Layout<Everything>::columns[DB::Layout<Everything>::Column(DB::LeafStart<Everything, 4>())].kind == DB::Kind::Float64, "a floating chrono rep keeps its Float kind");
static_assert(DB::Layout<Everything>::columns[DB::Layout<Everything>::Column(DB::LeafStart<Everything, 4>())].paramDen == 1, "but records its period");
Check(ra[&Everything::fee] == Percent(12.5), "fixed user formatter decodes by value");
Check(ra[&Everything::tags] == "food,monthly", "var user formatter view is the raw encoding");
Check(ra.Get<&Everything::maybe>() == std::optional<std::int32_t>(42), "optional scalar engaged");
Check(ra[&Everything::maybeText].has_value() && *ra[&Everything::maybeText] == "present", "optional string engaged (view)");
Check(ra[&Everything::money] == Money{ -1234, Currency::Dollar }, "nested aggregate by value");
const Party party = ra[&Everything::party];
Check(party == a.party, "nested aggregate with strings, FixedString and optional");
Check(ra[&Everything::triple] == std::array<std::uint16_t, 3>{ 1, 2, 3 }, "scalar array");
Check(ra[&Everything::code] == std::array<char, 4>{ 'E', 'U', 'R', '\0' }, "char array");
Check(ra[&Everything::iban].View() == "NL91ABNA0417164300", "FixedString view");
Check(std::span<const std::uint32_t>(ra[&Everything::ids]).size() == 5, "blob");
const DB::Row<Everything> rb = table[1];
Check(!rb[&Everything::maybe].has_value(), "optional scalar empty");
Check(!rb[&Everything::maybeText].has_value(), "optional string empty");
Check(rb[&Everything::party].note == std::nullopt, "nested optional empty");
Check(rb[&Everything::iban].View().empty(), "empty FixedString");
Check(table.Materialize() == std::vector<Everything>{ a, b }, "Materialize round trip of everything");
table.Set<&Everything::maybe>(1, std::optional<std::int32_t>(-9));
Check(rb.Get<&Everything::maybe>() == std::optional<std::int32_t>(-9), "Set an optional");
table.Set<&Everything::fee>(1, Percent(1.0));
Check(rb[&Everything::fee] == Percent(1.0), "Set through a user formatter");
table.Set<&Everything::party>(1, Party{ "Changed", DB::FixedString<11>("RABONL2U"), std::nullopt });
Check(rb[&Everything::party].name == "Changed", "Set a nested aggregate");
const std::string shown = std::format("{}", ra);
Check(shown.find("fee=") != std::string::npos && shown.find("money={minor=-1234, currency=1}") != std::string::npos && shown.find("maybeText=\"present\"") != std::string::npos, std::format("formatter: {}", shown));
}
{
const DB::Table<Everything> table(path, { .readOnly = true });
Check(table[0].Materialize() == a, "row a survives reopen");
const Everything bb = table[1].Materialize();
Check(bb.maybe == -9 && bb.fee == Percent(1.0) && bb.party.name == "Changed", "updated row b survives reopen");
const DB::Inspection info = DB::TableFile::Inspect(path);
bool found = false;
for (std::size_t i = 0; i < info.schema.columns.size(); i++) {
if (info.schema.Name(i) == "party.bic") found = info.schema.columns[i].kind == DB::Kind::FixedBytes && info.schema.columns[i].size == 11;
}
Check(found, "nested column 'party.bic' described as 11 fixed bytes");
}
static_assert(DB::Layout<Everything>::columns[1].offset == 1 || DB::Layout<Everything>::columns[1].size == 1, "1-byte columns follow the flags byte");
static_assert(DB::Layout<Everything>::rowAlign == 8);
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

171
tests/ShouldKeepLayoutGolden/main.cpp Normal file
View file

@ -0,0 +1,171 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Pins the on-disk layout of the reference Transaction row: 88 bytes, the
// documented column offsets, the fingerprint recomputed from an independent
// spelling of the canonical hash, and the exact bytes of the 16-byte string
// handle for an empty, an inline and a heap value. A change here is a file
// format change.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
enum class Currency : std::uint16_t { Euro };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Transaction::timePoint;
};
using Layout = DB::Layout<Transaction>;
// The documented row: flags at 0, ascending alignment classes after it.
struct ExpectedColumn {
std::string_view name;
DB::Kind kind;
std::uint16_t offset;
std::uint16_t size;
std::uint8_t flags;
std::uint32_t paramNum;
std::uint32_t paramDen;
std::uint32_t variant;
};
constexpr std::uint32_t systemClock = DB::TypeCode<std::chrono::system_clock>();
constexpr std::array<ExpectedColumn, 8> expected = { {
{ "flags", DB::Kind::Flags, 0, 1, 0, 0, 0, 0 },
{ "currency", DB::Kind::UInt16, 2, 2, DB::columnFlagEnum, 0, 0, 0 },
{ "name", DB::Kind::String, 8, 16, 0, 0, 0, 0 },
{ "bic", DB::Kind::String, 24, 16, 0, 0, 0, 0 },
{ "iban", DB::Kind::String, 40, 16, 0, 0, 0, 0 },
{ "description", DB::Kind::String, 56, 16, 0, 0, 0, 0 },
{ "amount", DB::Kind::Int64, 72, 8, 0, 0, 0, 0 },
{ "timePoint", DB::Kind::TimePoint, 80, 8, 0, 1, 1000000, systemClock },
} };
constexpr std::uint16_t orderedColumn = 7; // timePoint
// Independent spelling of the fingerprint, written from the format
// description rather than from Layout's code.
std::uint64_t SpecFingerprint() {
std::uint64_t hash = DB::Fnv1a(DB::fnvOffset, "crafter-db-v1");
hash = DB::Fnv1a(hash, std::uint64_t(88));
hash = DB::Fnv1a(hash, std::uint64_t(8));
hash = DB::Fnv1a(hash, std::uint64_t(orderedColumn));
for (const ExpectedColumn& c : expected) {
hash = DB::Fnv1a(hash, c.name);
hash = DB::Fnv1a(hash, static_cast<std::uint64_t>(c.kind));
hash = DB::Fnv1a(hash, c.offset);
hash = DB::Fnv1a(hash, c.size);
hash = DB::Fnv1a(hash, c.flags);
hash = DB::Fnv1a(hash, c.paramNum);
hash = DB::Fnv1a(hash, c.paramDen);
hash = DB::Fnv1a(hash, c.variant);
}
return hash;
}
}
int main() {
static_assert(Layout::rowSize == 88);
static_assert(Layout::rowAlign == 8);
static_assert(Layout::columnCount == 8);
for (std::size_t i = 0; i < expected.size(); i++) {
const DB::ColumnInfo& column = Layout::columns[i];
CheckEqual<std::string_view>(column.name.View(), expected[i].name, std::format("column {} name", i));
Check(column.kind == expected[i].kind, std::format("column {} kind", i));
CheckEqual<std::uint16_t>(column.offset, expected[i].offset, std::format("column {} offset", i));
CheckEqual<std::uint16_t>(column.size, expected[i].size, std::format("column {} size", i));
CheckEqual<std::uint8_t>(column.flags, expected[i].flags, std::format("column {} flags", i));
CheckEqual<std::uint32_t>(column.paramNum, expected[i].paramNum, std::format("column {} paramNum", i));
CheckEqual<std::uint32_t>(column.paramDen, expected[i].paramDen, std::format("column {} paramDen", i));
CheckEqual<std::uint32_t>(column.variant, expected[i].variant, std::format("column {} variant", i));
}
CheckEqual<std::uint16_t>(Layout::orderedColumn, orderedColumn, "ordered column index");
CheckEqual<std::uint64_t>(Layout::fingerprint, SpecFingerprint(), "fingerprint matches the specification");
CheckEqual<std::uint64_t>(DB::MakeSchemaInfo<Transaction>().Fingerprint(Layout::orderedColumn), Layout::fingerprint, "SchemaInfo fingerprint matches Layout");
// Descriptor blob round trip.
const DB::SchemaInfo info = DB::MakeSchemaInfo<Transaction>();
const std::vector<std::byte> blob = info.Serialize();
DB::SchemaInfo parsed;
Check(DB::SchemaInfo::Parse(blob, info.rowAlign, parsed), "schema blob parses");
CheckEqual<std::size_t>(parsed.columns.size(), 8, "parsed column count");
CheckEqual<std::string_view>(parsed.Name(4), "iban", "parsed column name");
CheckEqual<std::uint64_t>(parsed.Fingerprint(Layout::orderedColumn), Layout::fingerprint, "parsed fingerprint");
// Raw handle bytes.
const fs::path dir = fs::temp_directory_path() / "crafter-db-layout";
fs::remove_all(dir);
{
DB::Table<Transaction> table(dir / "t.cdb");
table.Append({ "", "ABNANL2A", "NL91ABNA0417164300", "", Currency::Euro, 1, TimePoint(std::chrono::microseconds(1)) });
const std::byte* row = table[0].Bytes();
std::uint8_t flags;
std::memcpy(&flags, row, 1);
CheckEqual<std::uint8_t>(flags, 0, "live row has flags 0");
DB::Var empty;
std::memcpy(&empty, row + 8, 16);
Check(empty.length == 0 && empty.prefix == 0 && empty.tailOrOffset == 0, "empty string is an all-zero handle");
DB::Var bic;
std::memcpy(&bic, row + 24, 16);
CheckEqual<std::uint32_t>(bic.length, 8, "inline handle length");
std::array<char, 12> inlineBytes{};
std::memcpy(inlineBytes.data(), row + 24 + 4, 12);
CheckEqual<std::string>(std::string(inlineBytes.data(), 8), "ABNANL2A", "inline handle bytes 4..12 hold the value");
Check(inlineBytes[8] == 0 && inlineBytes[11] == 0, "inline handle is zero padded");
DB::Var iban;
std::memcpy(&iban, row + 40, 16);
CheckEqual<std::uint32_t>(iban.length, 18, "heap handle length");
std::array<char, 4> prefix{};
std::memcpy(prefix.data(), &iban.prefix, 4);
CheckEqual<std::string>(std::string(prefix.data(), 4), "NL91", "heap handle prefix");
const std::span<const std::byte> heapBytes = table.File().HeapView().Read(iban.tailOrOffset, iban.length);
CheckEqual<std::string>(std::string(reinterpret_cast<const char*>(heapBytes.data()), heapBytes.size()), "NL91ABNA0417164300", "heap holds the full value, prefix included");
CheckEqual<std::uint64_t>(iban.tailOrOffset, 0, "first heap value sits at heap offset 0");
CheckEqual<std::uint64_t>(table.Stats().heapUsed, 18, "heap used equals the one out-of-line value");
std::int64_t micros;
std::memcpy(&micros, row + 80, 8);
CheckEqual<std::int64_t>(micros, 1, "timePoint stored as its microsecond rep");
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

102
tests/ShouldKeepViewsStableAcrossGrowth/main.cpp Normal file
View file

@ -0,0 +1,102 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Grows a table through many extents while holding Row views and
// string_views taken before the growth. Guards the fixed-address mapping
// promise: appends never move existing rows or heap bytes.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
struct Event {
std::string label;
std::string payload;
std::uint64_t sequence;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Event::sequence;
};
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-growth";
fs::remove_all(dir);
const fs::path path = dir / "events.cdb";
constexpr std::uint64_t count = 300'000;
{
DB::Table<Event> table(path);
table.Emplace(std::string_view("first"), std::string_view("payload of the very first row, long enough for the heap"), std::uint64_t(0));
const DB::Row<Event> first = table[0];
const std::string_view firstPayload = first[&Event::payload];
const std::byte* firstBytes = first.Bytes();
const char* firstHeapPointer = firstPayload.data();
std::vector<Event> batch;
batch.reserve(1000);
for (std::uint64_t i = 1; i < count; i++) {
batch.push_back({ std::format("row-{}", i), std::format("payload-{}-{}", i, std::string(i % 40, 'p')), i });
if (batch.size() == 1000) {
table.Append(batch);
batch.clear();
}
}
if (!batch.empty()) table.Append(batch);
CheckEqual<std::uint64_t>(table.Size(), count, "all rows appended");
Check(first.Bytes() == firstBytes, "row pointer unchanged after growth");
Check(firstPayload.data() == firstHeapPointer, "heap pointer unchanged after growth");
Check(firstPayload == "payload of the very first row, long enough for the heap", "early string_view still reads correctly");
Check(first[&Event::label] == "first", "early row view still decodes");
Check(table[count - 1][&Event::label] == std::format("row-{}", count - 1), "last row readable");
Check(table[123456][&Event::payload] == std::format("payload-{}-{}", 123456, std::string(123456 % 40, 'p')), "middle row readable");
const DB::TableStats stats = table.Stats();
Check(stats.fileLength > count * DB::Layout<Event>::rowSize, "file holds the rows");
table.Flush();
const DB::Inspection info = DB::TableFile::Inspect(path);
Check(info.commit.extentCount[0] >= 5, std::format("rows region grew through several extents ({})", info.commit.extentCount[0]));
Check(info.commit.extentCount[1] >= 5, std::format("heap region grew through several extents ({})", info.commit.extentCount[1]));
CheckEqual<std::uint64_t>(info.commit.rowCount, count, "committed row count");
Check(info.commit.fileLength == fs::file_size(path), "committed length equals file size after flush");
}
{
const DB::Table<Event> table(path, { .readOnly = true });
CheckEqual<std::uint64_t>(table.Size(), count, "rows survive reopen");
std::uint64_t expected = 0;
bool ordered = true;
for (DB::Row<Event> row : table) {
if (row[&Event::sequence] != expected) ordered = false;
expected++;
}
Check(ordered && expected == count, "sequence intact after reopen");
CheckEqual<std::uint64_t>(table.LowerBound<&Event::sequence>(150'000), 150'000, "binary search over a multi-extent table");
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

98
tests/ShouldLockAcrossProcesses/main.cpp Normal file
View file

@ -0,0 +1,98 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// One writer per file: a second writer in another process or in this one
// gets LockedError, readers are refused while a writer holds the file,
// readers coexist with each other, and a read-only table refuses writes.
#include <sys/wait.h>
#include <unistd.h>
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
struct Item {
std::string name;
static constexpr std::uint32_t version = 1;
};
// Returns the child's exit code: 0 = opened fine, 3 = LockedError, 4 = other.
int ChildOpens(const fs::path& path, bool readOnly) {
const pid_t pid = fork();
if (pid == 0) {
try {
DB::Table<Item> table(path, { .mode = DB::OpenMode::OpenExisting, .readOnly = readOnly });
_exit(0);
} catch (const DB::LockedError&) {
_exit(3);
} catch (...) {
_exit(4);
}
}
int status = 0;
waitpid(pid, &status, 0);
return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-lock";
fs::remove_all(dir);
const fs::path path = dir / "items.cdb";
{
DB::Table<Item> writer(path);
writer.Append({ "one" });
Check(ChildOpens(path, false) == 3, "second writer in another process gets LockedError");
Check(ChildOpens(path, true) == 3, "reader in another process is refused while a writer holds the file");
bool threw = false;
try {
DB::Table<Item> second(path);
} catch (const DB::LockedError&) {
threw = true;
}
Check(threw, "second Table on the same file in this process gets LockedError");
}
{
DB::Table<Item> reader(path, { .readOnly = true });
Check(ChildOpens(path, true) == 0, "readers coexist across processes");
Check(ChildOpens(path, false) == 3, "writer is refused while a reader holds the file");
bool threw = false;
try {
reader.Append({ "two" });
} catch (const DB::ReadOnlyError&) {
threw = true;
}
Check(threw, "Append on a read-only table throws ReadOnlyError");
threw = false;
try {
reader.Erase(0);
} catch (const DB::ReadOnlyError&) {
threw = true;
}
Check(threw, "Erase on a read-only table throws ReadOnlyError");
Check(reader.Size() == 1 && reader[0][&Item::name] == "one", "read-only table reads");
}
Check(ChildOpens(path, false) == 0, "writer opens once everyone is gone");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

236
tests/ShouldMigrate/main.cpp Normal file
View file

@ -0,0 +1,236 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Migrations: a V1 file opened by the V3 struct is converted in one pass
// through an additive by-name step (V1 -> V2) and an explicit Migrate
// (V2 -> V3); a V2 file skips the first step; the backup is kept; the
// refusal paths (read-only, migrate = false) throw MigrationError; the
// explicit Migrate<T>() entry point works without keeping the table open.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
enum class Currency : std::uint16_t { Euro, Dollar };
struct TransactionV1 {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &TransactionV1::timePoint;
};
// Additive: `reference` appended, nothing else changed -> no Migrate.
struct TransactionV2 {
std::string name;
std::string bic;
std::string iban;
std::string description;
std::string reference;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 2;
static constexpr auto orderedBy = &TransactionV2::timePoint;
using Previous = TransactionV1;
};
struct Money {
std::int64_t minor;
Currency currency;
friend bool operator==(const Money&, const Money&) = default;
};
// description -> memo, amount + currency -> money: needs Migrate.
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string memo;
std::string reference;
Money money;
TimePoint timePoint;
static constexpr std::uint32_t version = 3;
static constexpr auto orderedBy = &Transaction::timePoint;
using Previous = TransactionV2;
static Transaction Migrate(TransactionV2&& old) {
return {
.name = std::move(old.name),
.bic = std::move(old.bic),
.iban = std::move(old.iban),
.memo = std::move(old.description),
.reference = std::move(old.reference),
.money = { old.amount, old.currency },
.timePoint = old.timePoint,
};
}
};
struct EventV1 {
std::string name;
std::int64_t when;
static constexpr std::uint32_t version = 1;
};
struct EventV2 {
std::string name;
std::int64_t when;
static constexpr std::uint32_t version = 2;
static constexpr auto orderedBy = &EventV2::when;
using Previous = EventV1;
};
TimePoint At(std::int64_t seconds) {
return TimePoint(std::chrono::duration_cast<TimePoint::duration>(std::chrono::seconds(seconds)));
}
void WriteV1(const fs::path& path) {
DB::Table<TransactionV1> table(path);
table.Append({ "Albert Heijn", "ABNANL2A", "NL91ABNA0417164300", "groceries", Currency::Euro, -2345, At(1) });
table.Append({ "Employer", "RABONL2U", "NL20RABO0123456789", "salary for a long month of work", Currency::Dollar, 300000, At(2) });
table.Append({ "Gone", "", "", "to be erased", Currency::Euro, 1, At(3) });
table.Append({ "Landlord", "INGBNL2A", "NL69INGB0123456789", "rent", Currency::Euro, -120000, At(4) });
table.Erase(2);
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-migrate";
fs::remove_all(dir);
const fs::path path = dir / "tx.cdb";
WriteV1(path);
{
// Same struct, same version: no migration.
const DB::Table<TransactionV1> same(path);
CheckEqual<std::uint64_t>(same.Size(), 3, "V1 reopens as V1");
}
{
bool threw = false;
try {
DB::Table<Transaction> refused(path, { .readOnly = true });
} catch (const DB::MigrationError&) {
threw = true;
}
Check(threw, "read-only open of an older file throws MigrationError");
threw = false;
try {
DB::Table<Transaction> refused(path, { .migrate = false });
} catch (const DB::MigrationError&) {
threw = true;
}
Check(threw, "migrate = false throws MigrationError");
CheckEqual<std::uint32_t>(DB::PeekVersion(path), 1, "file untouched by refused migrations");
}
{
DB::Table<Transaction> table(path);
CheckEqual<std::uint64_t>(table.Size(), 3, "live rows carried over (tombstone dropped)");
CheckEqual<std::uint64_t>(table.SlotCount(), 3, "migration compacts");
const std::vector<Transaction> rows = table.Materialize();
CheckEqual<std::string>(rows[0].memo, "groceries", "description -> memo");
Check(rows[1].money == Money{ 300000, Currency::Dollar }, "amount + currency -> money");
CheckEqual<std::string>(rows[1].reference, "", "new V2 column value-initialised");
CheckEqual<std::string>(rows[2].iban, "NL69INGB0123456789", "strings carried");
Check(rows[2].timePoint == At(4), "time carried");
Check(table.Back()[&Transaction::name] == "Landlord", "view after migration");
const DB::Inspection info = DB::TableFile::Inspect(path);
CheckEqual<std::uint32_t>(info.header.schemaVersion, 3, "file now at version 3");
CheckEqual<std::uint32_t>(info.header.migratedFromVersion, 1, "migratedFromVersion recorded");
Check(fs::exists(DB::TableFile::BackupPath(path, 1)), "backup of the V1 file kept");
CheckEqual<std::uint32_t>(DB::PeekVersion(DB::TableFile::BackupPath(path, 1)), 1, "backup is the V1 file");
table.Append({ "After", "", "", "", "ref", { 5, Currency::Euro }, At(5) });
CheckEqual<std::uint64_t>(table.Size(), 4, "append after migration");
}
{
// A V2 file skips the by-name step.
const fs::path path2 = dir / "tx2.cdb";
{
DB::Table<TransactionV2> v2(path2);
v2.Append({ "A", "B", "C", "D", "REF-1", Currency::Euro, 7, At(1) });
}
DB::Table<Transaction> table(path2, { .backupBeforeMigrate = false });
CheckEqual<std::uint64_t>(table.Size(), 1, "V2 -> V3 row count");
Check(table[0][&Transaction::reference] == "REF-1", "V2 reference carried through Migrate");
Check(table[0][&Transaction::memo] == "D", "V2 description -> memo");
Check(!fs::exists(DB::TableFile::BackupPath(path2, 2)), "no backup when disabled");
}
{
// Explicit entry point and the V1 -> V2 by-name path on its own.
const fs::path path3 = dir / "tx3.cdb";
WriteV1(path3);
DB::Migrate<TransactionV2>(path3);
CheckEqual<std::uint32_t>(DB::PeekVersion(path3), 2, "explicit Migrate<V2>");
const DB::Table<TransactionV2> v2(path3, { .readOnly = true });
CheckEqual<std::uint64_t>(v2.Size(), 3, "by-name migration keeps rows");
Check(v2[1][&TransactionV2::description] == "salary for a long month of work" && v2[1][&TransactionV2::reference] == "", "by-name copies and value-initialises");
}
{
// Old data that is not in the order the new struct declares is sorted
// during the migration instead of failing it.
const fs::path events = dir / "events.cdb";
{
DB::Table<EventV1> v1(events);
v1.Append({ "late", 20 });
v1.Append({ "early", 10 });
v1.Append({ "middle", 15 });
v1.Append({ "also early", 10 });
}
DB::Table<EventV2> v2(events, { .backupBeforeMigrate = false });
CheckEqual<std::uint64_t>(v2.Size(), 4, "all rows migrated");
std::vector<std::int64_t> order;
for (DB::Row<EventV2> row : v2) order.push_back(row[&EventV2::when]);
CheckEqual(order, std::vector<std::int64_t>{ 10, 10, 15, 20 }, "rows sorted by the new orderedBy");
Check(v2[0][&EventV2::name] == "early" && v2[1][&EventV2::name] == "also early", "sort is stable");
Check(v2.Verify().Ok(), "migrated table verifies");
}
{
// A file newer than the code is refused.
bool threw = false;
try {
DB::Table<TransactionV1> older(path);
} catch (const DB::SchemaError&) {
threw = true;
}
Check(threw, "opening a V3 file with the V1 struct throws SchemaError");
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

171
tests/ShouldQuery/main.cpp Normal file
View file

@ -0,0 +1,171 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Queries: lambda filters with Key and string_view, column expressions,
// binary search on the ordered column (Range/From/After/Until/Before,
// Lower/UpperBound at the boundaries, equal keys, tombstoned slots), chaining
// Range().Where(), bound extraction from a combined expression, ranges
// composition, and the OrderError checks on Update/Set.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
enum class Currency : std::uint16_t { Euro };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Transaction::timePoint;
};
inline constexpr DB::Column<&Transaction::iban> iban{};
inline constexpr DB::Column<&Transaction::description> description{};
inline constexpr DB::Column<&Transaction::amount> amount{};
inline constexpr DB::Column<&Transaction::timePoint> timePoint{};
TimePoint At(std::int64_t seconds) {
return TimePoint(std::chrono::duration_cast<TimePoint::duration>(std::chrono::seconds(seconds)));
}
std::vector<std::uint64_t> Ids(auto&& query) {
std::vector<std::uint64_t> ids;
for (auto row : query) ids.push_back(row.Id());
return ids;
}
template<class Query>
std::vector<std::uint64_t> IdsOf(Query query) {
std::vector<std::uint64_t> ids;
for (auto row : query) ids.push_back(row.Id());
return ids;
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-query";
fs::remove_all(dir);
DB::Table<Transaction> table(dir / "q.cdb");
const std::string_view ibanA = "NL91ABNA0417164300";
const std::string_view ibanB = "NL91ABNA0417164301"; // same length, same 4-byte prefix
const std::string_view ibanC = "DE89370400440532013000";
// id key
table.Append({ "a", "ABNANL2A", std::string(ibanA), "rent", Currency::Euro, -100000, At(10) }); // 0
table.Append({ "b", "ABNANL2A", std::string(ibanB), "rent", Currency::Euro, -200, At(20) }); // 1
table.Append({ "c", "RABONL2U", std::string(ibanC), "coffee", Currency::Euro, -350, At(20) }); // 2 equal key
table.Append({ "d", "RABONL2U", std::string(ibanA), "salary", Currency::Euro, 300000, At(30) }); // 3
table.Append({ "e", "INGBNL2A", std::string(ibanC), "rent ", Currency::Euro, -100000, At(40) }); // 4 trailing space: same length as "rent" + 1
table.Append({ "f", "INGBNL2A", std::string(ibanA), "RENT", Currency::Euro, -5, At(50) }); // 5
// --- lambda filters ---------------------------------------------------
const DB::Key keyA(ibanA);
CheckEqual(IdsOf(table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == keyA; })), std::vector<std::uint64_t>{ 0, 3, 5 }, "lambda with Key");
CheckEqual(IdsOf(table.Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == ibanB; })), std::vector<std::uint64_t>{ 1 }, "lambda with string_view, prefix-equal neighbour rejected");
CheckEqual(IdsOf(table.Where([](DB::Row<Transaction> r) { return r[&Transaction::description] == "rent"; })), std::vector<std::uint64_t>{ 0, 1 }, "description equality is exact (case and length)");
CheckEqual(table.Where([](DB::Row<Transaction> r) { return r[&Transaction::amount] < 0; }).Count(), std::uint64_t(5), "Count");
Check(table.Where(amount > 0).First().has_value() && table.Where(amount > 0).First()->Id() == 3, "First");
Check(!table.Where(amount > 1'000'000).Any(), "Any on no match");
// --- column expressions -------------------------------------------------
CheckEqual(IdsOf(table.Where(iban == ibanA)), std::vector<std::uint64_t>{ 0, 3, 5 }, "expression ==");
CheckEqual(IdsOf(table.Where(iban != ibanA)), std::vector<std::uint64_t>{ 1, 2, 4 }, "expression !=");
CheckEqual(IdsOf(table.Where(iban == ibanA && amount > 0)), std::vector<std::uint64_t>{ 3 }, "expression &&");
CheckEqual(IdsOf(table.Where(description == "coffee" || description == "salary")), std::vector<std::uint64_t>{ 2, 3 }, "expression ||");
CheckEqual(IdsOf(table.Where(!(amount < 0))), std::vector<std::uint64_t>{ 3 }, "expression !");
CheckEqual(IdsOf(table.Where(description < "d")), std::vector<std::uint64_t>{ 2, 5 }, "string ordering compare ('RENT' and 'coffee' sort before 'd')");
// --- ordered column -----------------------------------------------------
CheckEqual<std::uint64_t>(table.LowerBound<&Transaction::timePoint>(At(20)), 1, "LowerBound on equal keys");
CheckEqual<std::uint64_t>(table.UpperBound<&Transaction::timePoint>(At(20)), 3, "UpperBound on equal keys");
CheckEqual<std::uint64_t>(table.LowerBound<&Transaction::timePoint>(At(5)), 0, "LowerBound below all");
CheckEqual<std::uint64_t>(table.LowerBound<&Transaction::timePoint>(At(55)), 6, "LowerBound above all");
CheckEqual(IdsOf(table.Range<&Transaction::timePoint>(At(20), At(40))), std::vector<std::uint64_t>{ 1, 2, 3, 4 }, "Range inclusive both ends");
CheckEqual(IdsOf(table.Range<&Transaction::timePoint>(At(21), At(29))), std::vector<std::uint64_t>{}, "Range with no rows");
CheckEqual(IdsOf(table.Range<&Transaction::timePoint>(At(40), At(20))), std::vector<std::uint64_t>{}, "Range with inverted bounds is empty");
CheckEqual(IdsOf(table.From<&Transaction::timePoint>(At(30))), std::vector<std::uint64_t>{ 3, 4, 5 }, "From");
CheckEqual(IdsOf(table.After<&Transaction::timePoint>(At(30))), std::vector<std::uint64_t>{ 4, 5 }, "After");
CheckEqual(IdsOf(table.Until<&Transaction::timePoint>(At(20))), std::vector<std::uint64_t>{ 0, 1, 2 }, "Until");
CheckEqual(IdsOf(table.Before<&Transaction::timePoint>(At(20))), std::vector<std::uint64_t>{ 0 }, "Before");
// --- range + property ---------------------------------------------------
const std::vector<std::uint64_t> chained = IdsOf(table.Range<&Transaction::timePoint>(At(20), At(50)).Where(iban == ibanA));
CheckEqual(chained, std::vector<std::uint64_t>{ 3, 5 }, "Range().Where(expression)");
CheckEqual(IdsOf(table.Range<&Transaction::timePoint>(At(20), At(50)).Where([&](DB::Row<Transaction> r) { return r[&Transaction::iban] == keyA; })), chained, "Range().Where(lambda)");
CheckEqual(IdsOf(table.Where(timePoint >= At(20) && timePoint <= At(50) && iban == ibanA)), chained, "combined expression extracts bounds");
CheckEqual(IdsOf(table.Where(timePoint > At(20) && timePoint < At(50))), std::vector<std::uint64_t>{ 3, 4 }, "exclusive bounds");
CheckEqual(IdsOf(table.Where(timePoint == At(20))), std::vector<std::uint64_t>{ 1, 2 }, "equality bound");
CheckEqual(IdsOf(table.Where(timePoint == At(20) || iban == ibanA)), std::vector<std::uint64_t>{ 0, 1, 2, 3, 5 }, "|| disables bound extraction but stays correct");
// --- ranges composition -------------------------------------------------
std::uint64_t taken = 0;
for (DB::Row<Transaction> row : table | std::views::filter([](DB::Row<Transaction> r) { return r[&Transaction::amount] < 0; }) | std::views::take(2)) taken += row.Id() + 1;
CheckEqual<std::uint64_t>(taken, 1 + 2, "views::filter | views::take over the table");
const std::int64_t total = std::ranges::fold_left(table.Select<&Transaction::amount>(), std::int64_t(0), std::plus{});
CheckEqual<std::int64_t>(total, -100000 - 200 - 350 + 300000 - 100000 - 5, "Select + fold_left");
CheckEqual<std::size_t>(table.Slots().size(), 6, "Slots size");
Check(std::ranges::lower_bound(table.Slots(), At(30), {}, [](DB::Row<Transaction> r) { return r[&Transaction::timePoint]; }) - table.Slots().begin() == 3, "std::ranges::lower_bound over Slots()");
// --- tombstones in the search -------------------------------------------
table.Erase(2);
CheckEqual(IdsOf(table.Range<&Transaction::timePoint>(At(20), At(20))), std::vector<std::uint64_t>{ 1 }, "Range skips tombstones but still finds neighbours");
CheckEqual<std::uint64_t>(table.UpperBound<&Transaction::timePoint>(At(20)), 3, "UpperBound unchanged by a tombstone (keys survive)");
CheckEqual(IdsOf(table.Where(timePoint == At(20))), std::vector<std::uint64_t>{ 1 }, "bound extraction with a tombstone");
// --- order checks on Update/Set -----------------------------------------
bool threw = false;
try {
table.Update(3, { "d", "RABONL2U", std::string(ibanA), "salary", Currency::Euro, 300000, At(15) }); // would sort before row 1/2
} catch (const DB::OrderError&) {
threw = true;
}
Check(threw, "Update that breaks the order throws");
threw = false;
try {
table.Set<&Transaction::timePoint>(3, At(45)); // would sort after row 4
} catch (const DB::OrderError&) {
threw = true;
}
Check(threw, "Set that breaks the order throws");
table.Set<&Transaction::timePoint>(3, At(35));
Check(table[3][&Transaction::timePoint] == At(35), "Set within the neighbours succeeds");
table.Set<&Transaction::description>(3, "wage");
Check(table[3][&Transaction::description] == "wage", "Set a string column");
CheckEqual<std::uint64_t>(table.Stats().heapGarbage, 22, "short strings leave no heap garbage (only the erased row's 22-byte iban is garbage)");
table.Set<&Transaction::description>(3, "a description longer than twelve");
Check(table[3][&Transaction::description] == "a description longer than twelve", "Set a long string column");
table.Set<&Transaction::description>(3, "short");
CheckEqual<std::uint64_t>(table.Stats().heapGarbage, 22 + 32, "replacing a heap value orphans its bytes");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

94
tests/ShouldReadWhileAppending/main.cpp Normal file
View file

@ -0,0 +1,94 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Readers scanning (with tombstones present, so the per-row flag test is on)
// while one writer appends and erases. Every row a scan yields must be a
// published, live-at-some-point row with increasing ids, and nothing may
// read past the published slots. Guards the sentinel-based iteration and the
// tombstone flag against the Append race.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
// 16-byte rows: a power-of-two size, so a full extent ends exactly on a
// row boundary and the slot after the last published one can be the
// first byte of the unmapped reservation.
struct Tick {
std::uint64_t sequence;
std::uint64_t value;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Tick::sequence;
};
static_assert(DB::Layout<Tick>::rowSize == 24 || DB::Layout<Tick>::rowSize == 32);
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-concurrent";
fs::remove_all(dir);
DB::Table<Tick> table(dir / "ticks.cdb");
table.Append({ 0, 0 });
table.Append({ 1, 1 });
table.Erase(0); // a tombstone from the start: scans run with the flag test
std::atomic<bool> stop{ false };
std::atomic<std::uint64_t> scans{ 0 };
std::atomic<std::uint64_t> badOrder{ 0 };
std::atomic<std::uint64_t> badId{ 0 };
std::atomic<std::uint64_t> sawErased{ 0 };
const auto reader = [&] {
while (!stop.load(std::memory_order_relaxed)) {
std::uint64_t last = 0;
bool first = true;
const std::uint64_t slots = table.SlotCount();
for (DB::Row<Tick> row : table) {
const std::uint64_t id = row.Id();
if (!first && id <= last) badOrder.fetch_add(1, std::memory_order_relaxed);
if (id >= table.SlotCount()) badId.fetch_add(1, std::memory_order_relaxed);
if (row[&Tick::sequence] != id) badId.fetch_add(1, std::memory_order_relaxed);
first = false;
last = id;
}
if (slots > 2 && table.Rows(0, 1).Any()) sawErased.fetch_add(1, std::memory_order_relaxed);
scans.fetch_add(1, std::memory_order_relaxed);
}
};
std::vector<std::jthread> readers;
for (std::int32_t i = 0; i < 4; i++) readers.emplace_back(reader);
std::vector<Tick> batch;
for (std::uint64_t sequence = 2; sequence < 400'000; sequence++) {
batch.push_back({ sequence, sequence });
if (batch.size() == 64) {
table.Append(batch);
batch.clear();
if (sequence % 1024 == 1) table.Erase(sequence - 500);
}
}
stop.store(true);
readers.clear();
Check(scans.load() > 0, "readers ran");
Check(badOrder.load() == 0, std::format("ids strictly increase within a scan ({} violations)", badOrder.load()));
Check(badId.load() == 0, std::format("every yielded row is a published row ({} violations)", badId.load()));
Check(sawErased.load() == 0, std::format("the erased first row never shows up ({} times)", sawErased.load()));
Check(table.Verify().Ok(), "table verifies after the run");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

213
tests/ShouldRecoverCommits/main.cpp Normal file
View file

@ -0,0 +1,213 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Crash safety of the commit protocol. A child process appends without
// flushing and dies: the parent must see exactly the committed rows. A
// damaged newer commit slot falls back to the older one. Uncommitted growth
// is truncated on the next writable open. Tombstones written before an
// unclean exit are recounted.
#include <sys/wait.h>
#include <unistd.h>
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
struct Entry {
std::string text;
std::uint64_t n;
static constexpr std::uint32_t version = 1;
};
// Runs `work` in a child. `work` must end with Die() while its table is
// still in scope, so no destructor (and no Flush) ever runs.
[[noreturn]] void Die() { _exit(0); }
void InChildWithoutCleanup(const std::function<void()>& work) {
const pid_t pid = fork();
if (pid == 0) {
try {
work();
} catch (const std::exception& error) {
std::println("child failed: {}", error.what());
_exit(2);
}
_exit(5); // work() returned instead of dying: the test is wrong
}
int status = 0;
waitpid(pid, &status, 0);
Check(WIFEXITED(status) && WEXITSTATUS(status) == 0, "child ran to its exit point");
}
void Patch(const fs::path& path, std::uint64_t offset, std::byte value) {
std::fstream file(path, std::ios::in | std::ios::out | std::ios::binary);
file.seekp(static_cast<std::streamoff>(offset));
file.write(reinterpret_cast<const char*>(&value), 1);
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-recover";
fs::remove_all(dir);
const fs::path path = dir / "log.cdb";
// Committed rows survive, unflushed rows vanish, the table stays valid.
InChildWithoutCleanup([&] {
DB::Table<Entry> table(path);
for (std::uint64_t i = 0; i < 100; i++) table.Append({ std::format("committed {}", i), i });
table.Flush();
for (std::uint64_t i = 100; i < 150; i++) table.Append({ std::format("lost {}", i), i });
Die(); // no Flush, no destructor
});
{
DB::Table<Entry> table(path);
CheckEqual<std::uint64_t>(table.Size(), 100, "only flushed rows are visible after a crash");
Check(table.Back()[&Entry::text] == "committed 99", "last committed row intact");
table.Append({ "after crash", 100 });
Check(table[100][&Entry::text] == "after crash", "slot after the committed rows is reusable");
table.Flush();
}
// EveryAppend durability: nothing is lost.
InChildWithoutCleanup([&] {
DB::Table<Entry> table(path, { .durability = DB::Durability::EveryAppend });
table.Append({ "durable", 101 });
Die();
});
{
DB::Table<Entry> table(path);
CheckEqual<std::uint64_t>(table.Size(), 102, "EveryAppend row survived a crash");
}
// Uncommitted growth is truncated on open.
std::uint64_t committedLength = 0;
InChildWithoutCleanup([&] {
DB::Table<Entry> table(path);
table.Flush();
committedLength = fs::file_size(path);
for (std::uint64_t i = 0; i < 2000; i++) table.Append({ std::string(100, 'g'), i }); // forces new extents
Die();
});
{
const DB::Inspection info = DB::TableFile::Inspect(path);
committedLength = info.commit.fileLength;
Check(fs::file_size(path) > committedLength, "uncommitted extents extend the file before recovery");
DB::Table<Entry> table(path);
CheckEqual<std::uint64_t>(fs::file_size(path), committedLength, "writable open truncates uncommitted growth");
CheckEqual<std::uint64_t>(table.Size(), 102, "rows unchanged");
}
// Tombstones written before an unclean exit are recounted.
InChildWithoutCleanup([&] {
DB::Table<Entry> table(path);
table.Erase(0);
table.Erase(1);
Die();
});
{
DB::Table<Entry> table(path);
CheckEqual<std::uint64_t>(table.Size(), 100, "live count recounted after unclean exit");
std::uint64_t seen = 0;
for ([[maybe_unused]] DB::Row<Entry> row : table) seen++;
CheckEqual<std::uint64_t>(seen, 100, "iteration honours recovered tombstones");
table.Compact();
table.Flush();
}
// An Update whose heap bytes were written but whose process died before
// any Flush: the row must read as either the old or the new value, never
// point at uncommitted heap bytes.
InChildWithoutCleanup([&] {
DB::Table<Entry> table(path);
table.Flush();
table.Update(5, { std::string(300, 'u'), 5 }); // long value -> heap
table.Set<&Entry::text>(6, std::string(200, 's'));
Die();
});
{
DB::Table<Entry> table(path);
const std::string five(table[5][&Entry::text]);
Check(five == std::string(300, 'u') || five == "committed 5", std::format("updated row is consistent after a crash (got {} bytes)", five.size()));
const std::string six(table[6][&Entry::text]);
Check(six == std::string(200, 's') || six == "committed 6", "Set row is consistent after a crash");
Check(table.Verify().Ok(), "no dangling heap references after a crashed Update");
table.Append({ "after update crash", 7 });
Check(table[5][&Entry::text] == five, "later appends do not change the updated row's text");
table.Flush();
}
// A forked child that destroys its inherited Table must not write a
// commit record: its counters are the parent's at fork time.
{
DB::Table<Entry> table(path);
table.Append({ "parent row", 8 });
table.Flush();
const DB::TableStats before = table.Stats();
table.Erase(2); // tombstone after the flush
const pid_t pid = fork();
if (pid == 0) {
// normal destruction path in the child
bool threw = false;
try {
table.Append({ "child", 9 });
} catch (const DB::Error&) {
threw = true;
}
if (!threw) _exit(6);
const std::string_view text = table[0][&Entry::text]; // reads work on the shared mapping
if (text.size() == 0) _exit(7);
table.~Table();
_exit(0);
}
int status = 0;
waitpid(pid, &status, 0);
Check(WIFEXITED(status) && WEXITSTATUS(status) == 0, std::format("child refused to write and destroyed its table (status {})", WIFEXITED(status) ? WEXITSTATUS(status) : -1));
const DB::Inspection info = DB::TableFile::Inspect(path);
CheckEqual<std::uint64_t>(info.commit.commitSeq, before.commitSeq, "child wrote no commit record");
CheckEqual<std::uint64_t>(table.Size(), before.liveCount - 1, "parent still sees its tombstone");
table.Flush();
const DB::Inspection after = DB::TableFile::Inspect(path);
CheckEqual<std::uint64_t>(after.commit.liveCount, before.liveCount - 1, "parent's commit carries the tombstone");
}
// A damaged newer commit slot falls back to the older one.
{
{
DB::Table<Entry> table(path);
table.Append({ "slot A", 1 });
table.Flush(); // commit N: rows r+1
table.Append({ "slot B", 2 }); // the destructor's clean close writes commit N+1: rows r+2
}
const DB::Inspection before = DB::TableFile::Inspect(path);
const std::size_t newer = before.header.commits[0].commitSeq > before.header.commits[1].commitSeq ? 0 : 1;
Patch(path, 3072 + newer * 512 + 8, std::byte{ 0xEE }); // corrupt rowCount of the newest slot -> CRC fails
DB::Table<Entry> table(path);
CheckEqual<std::uint64_t>(table.SlotCount(), before.commit.rowCount - 1, "fell back to the previous commit");
Check(table.Back()[&Entry::text] == "slot A", "the row of the lost commit is gone");
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

207
tests/ShouldRejectSchemaChange/main.cpp Normal file
View file

@ -0,0 +1,207 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// A struct that changed without a version bump must be refused with the
// column difference in the message; so must files that are not tables, have
// a damaged header, or have both commit records corrupted.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
struct Original {
std::string name;
std::int64_t amount;
static constexpr std::uint32_t version = 1;
};
struct AddedColumn { // same version, one more member
std::string name;
std::int64_t amount;
std::string reference;
static constexpr std::uint32_t version = 1;
};
struct RenamedColumn { // same version, same layout, different name
std::string name;
std::int64_t total;
static constexpr std::uint32_t version = 1;
};
struct RetypedColumn { // same version, int64 -> int32
std::string name;
std::int32_t amount;
static constexpr std::uint32_t version = 1;
};
struct NowOrdered { // same columns, but ordered
std::string name;
std::int64_t amount;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &NowOrdered::amount;
};
struct IntBlob {
std::vector<std::int32_t> values;
static constexpr std::uint32_t version = 1;
};
struct FloatBlob { // same bytes per element, different meaning
std::vector<float> values;
static constexpr std::uint32_t version = 1;
};
template<class T>
std::string ExpectSchemaError(const fs::path& path, std::string_view what) {
try {
DB::Table<T> table(path, { .readOnly = true });
} catch (const DB::SchemaError& error) {
return error.what();
} catch (const std::exception& error) {
std::println("FAIL: {} threw {} instead of SchemaError", what, error.what());
++failures;
return "";
}
std::println("FAIL: {} did not throw", what);
++failures;
return "";
}
void CheckOrFail(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<class E>
bool Throws(const fs::path& path) {
try {
DB::Table<Original> table(path, { .readOnly = true });
} catch (const E&) {
return true;
} catch (...) {
return false;
}
return false;
}
void Patch(const fs::path& path, std::uint64_t offset, std::span<const std::byte> bytes) {
std::fstream file(path, std::ios::in | std::ios::out | std::ios::binary);
file.seekp(static_cast<std::streamoff>(offset));
file.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-schema";
fs::remove_all(dir);
const fs::path path = dir / "o.cdb";
{
DB::Table<Original> table(path);
table.Append({ "x", 1 });
}
const std::string added = ExpectSchemaError<AddedColumn>(path, "added member");
Check(added.find("reference") != std::string::npos && added.find("without bumping") != std::string::npos, std::format("message names the new column: {}", added));
const std::string renamed = ExpectSchemaError<RenamedColumn>(path, "renamed member");
Check(renamed.find("total") != std::string::npos && renamed.find("amount") != std::string::npos, std::format("message shows both names: {}", renamed));
const std::string retyped = ExpectSchemaError<RetypedColumn>(path, "retyped member");
Check(retyped.find("i32") != std::string::npos && retyped.find("i64") != std::string::npos, std::format("message shows both kinds: {}", retyped));
const std::string ordered = ExpectSchemaError<NowOrdered>(path, "orderedBy added");
Check(ordered.find("orderedBy") != std::string::npos && ordered.find("amount") != std::string::npos, std::format("message names the ordered column change: {}", ordered));
{
const DB::Table<Original> fine(path, { .readOnly = true });
Check(fine.Size() == 1, "the original struct still opens");
}
{
const fs::path blob = dir / "blob.cdb";
{
DB::Table<IntBlob> ints(blob);
ints.Append({ { 1, 2, 3, 1065353216 } });
}
const std::string retypedBlob = ExpectSchemaError<FloatBlob>(blob, "blob element type changed");
Check(!retypedBlob.empty(), "vector<int32> -> vector<float> is refused");
}
{
// CreateNew on an existing file must refuse without touching it.
const DB::Inspection before = DB::TableFile::Inspect(path);
bool threw = false;
try {
DB::Table<Original> again(path, { .mode = DB::OpenMode::CreateNew });
} catch (const DB::IOError& error) {
threw = error.code == std::errc::file_exists;
}
Check(threw, "CreateNew on an existing table throws file_exists");
const DB::Inspection after = DB::TableFile::Inspect(path);
CheckOrFail(before.commit.commitSeq == after.commit.commitSeq, "CreateNew wrote nothing to the existing file");
}
// A descriptor whose presenceOffset points outside the row.
const fs::path presence = dir / "presence.cdb";
fs::copy_file(path, presence);
{
const DB::Inspection info = DB::TableFile::Inspect(presence);
// descriptor i starts at 4096 + 16 + i * 24; presenceOffset is at +6
const std::uint16_t bogus = 0xFFF0;
Patch(presence, 4096 + 16 + 1 * 24 + 6, std::as_bytes(std::span(&bogus, 1)));
Check(info.schema.columns.size() == 3, "fixture has three columns");
}
Check(Throws<DB::CorruptError>(presence), "an out-of-row presenceOffset throws CorruptError");
// Damaged magic.
const fs::path magic = dir / "magic.cdb";
fs::copy_file(path, magic);
const std::array<std::byte, 2> junk = { std::byte{ 'X' }, std::byte{ 'X' } };
Patch(magic, 0, junk);
Check(Throws<DB::CorruptError>(magic), "bad magic throws CorruptError");
// Both commit records damaged (flip a byte in each slot's sequence number).
const fs::path commits = dir / "commits.cdb";
fs::copy_file(path, commits);
const std::array<std::byte, 1> flip = { std::byte{ 0xFF } };
Patch(commits, 3072 + 1, flip);
Patch(commits, 3584 + 1, flip);
Check(Throws<DB::CorruptError>(commits), "two damaged commit records throw CorruptError");
// Not a table at all.
const fs::path random = dir / "random.cdb";
{
std::ofstream out(random, std::ios::binary);
std::string bytes(8192, '\0');
for (std::size_t i = 0; i < bytes.size(); i++) bytes[i] = static_cast<char>(i * 31 + 7);
out.write(bytes.data(), static_cast<std::streamsize>(bytes.size()));
}
Check(Throws<DB::CorruptError>(random), "random bytes throw CorruptError");
// Too short.
const fs::path tiny = dir / "tiny.cdb";
{ std::ofstream(tiny) << "short"; }
Check(Throws<DB::CorruptError>(tiny), "a tiny file throws CorruptError");
// Truncated below the committed length.
const fs::path truncated = dir / "truncated.cdb";
fs::copy_file(path, truncated);
fs::resize_file(truncated, 4096 * 2);
Check(Throws<DB::CorruptError>(truncated), "a truncated file throws CorruptError");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

182
tests/ShouldRoundTrip/main.cpp Normal file
View file

@ -0,0 +1,182 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Append rows of every string length class (empty, inline, heap, huge,
// UTF-8), read them back through views and through Materialize, reopen the
// file and read them again. Guards the encode/decode symmetry and the file
// round trip that everything else builds on.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
using TimePoint = std::chrono::system_clock::time_point;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
template<typename T>
void CheckEqual(const T& actual, const T& expected, std::string_view what) {
if (!(actual == expected)) {
std::println("FAIL: {} — expected '{}', got '{}'", what, expected, actual);
++failures;
}
}
enum class Currency : std::uint16_t { Euro, Dollar };
struct Transaction {
std::string name;
std::string bic;
std::string iban;
std::string description;
Currency currency;
std::int64_t amount;
TimePoint timePoint;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Transaction::timePoint;
friend bool operator==(const Transaction&, const Transaction&) = default;
};
TimePoint At(std::int64_t seconds) {
return TimePoint(std::chrono::duration_cast<TimePoint::duration>(std::chrono::seconds(seconds)));
}
std::vector<Transaction> Sample() {
std::vector<Transaction> rows;
rows.push_back({ "Albert Heijn", "ABNANL2A", "NL91ABNA0417164300", "groceries", Currency::Euro, -2345, At(100) });
rows.push_back({ "", "", "", "", Currency::Euro, 0, At(100) }); // all empty, equal key allowed
rows.push_back({ "Twelve chars", "ABCDEFGHIJKL", "ABCDEFGHIJKLM", "exactly twelve", Currency::Dollar, 1, At(101) }); // 12 inline, 13 heap
rows.push_back({ "Café Münster", "RABONL2U", "DE89370400440532013000", std::string(100 * 1024, 'x'), Currency::Euro, 99999999, At(102) });
rows.push_back({ "a", "b", "c", "d", Currency::Euro, -1, At(103) });
return rows;
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-roundtrip";
fs::remove_all(dir);
const fs::path path = dir / "sub" / "main.cdb";
const std::vector<Transaction> sample = Sample();
{
DB::Table<Transaction> table(path);
Check(table.Empty(), "new table is empty");
CheckEqual<std::uint64_t>(table.Size(), 0, "new table size");
Check(fs::exists(path), "table file created");
Check((fs::status(path).permissions() & fs::perms::others_all) == fs::perms::none, "file is private");
Check((fs::status(dir / "sub").permissions() & fs::perms::group_all) == fs::perms::none, "directory is private");
for (const Transaction& row : sample) table.Append(row);
CheckEqual<std::uint64_t>(table.Size(), sample.size(), "size after append");
CheckEqual<std::uint64_t>(table.SlotCount(), sample.size(), "slots after append");
const DB::Row<Transaction> last = table.Back();
Check(last[&Transaction::timePoint] == At(103), "Back() timePoint via member pointer");
Check(last.Get<&Transaction::amount>() == -1, "Back() amount via Get<>");
Check(last[&Transaction::iban] == "c", "Back() iban view compares to literal");
CheckEqual<std::uint64_t>(last.Id(), 4, "Back() id");
const DB::Row<Transaction> second = table[2];
Check(second[&Transaction::bic] == std::string_view("ABCDEFGHIJKL"), "12-char string inline round trip");
Check(second[&Transaction::iban] == std::string("ABCDEFGHIJKLM"), "13-char string heap round trip");
Check(second[&Transaction::iban] != std::string_view("ABCDEFGHIJKLX"), "same-length prefix-equal non-match");
Check(!(second[&Transaction::iban] == std::string_view("ABCDEFGHIJKL")), "shorter prefix-equal non-match");
Check(second[&Transaction::currency] == Currency::Dollar, "enum round trip");
CheckEqual<std::string>(std::string(second[&Transaction::description].View()), "exactly twelve", "StringRef::View");
const DB::Row<Transaction> empty = table[1];
Check(empty[&Transaction::name].Empty(), "empty string is empty");
Check(empty[&Transaction::name] == "", "empty string equals empty literal");
const DB::Row<Transaction> huge = table[3];
CheckEqual<std::uint32_t>(huge[&Transaction::description].Length(), 100 * 1024, "huge string length");
Check(huge[&Transaction::description] == sample[3].description, "huge string equals");
Check(huge[&Transaction::name] == "Café Münster", "utf-8 inline/heap string");
const std::vector<Transaction> all = table.Materialize();
Check(all == sample, "Materialize equals input");
CheckEqual<std::string>(std::string(DB::TypeName<Transaction>()).substr(0, 0), "", "noop");
const DB::RowId id = table.Emplace(std::string_view("Emplaced"), "BICBICBI", std::string("NL00EMPL0000000001"), std::string_view("via Emplace"), Currency::Euro, std::int64_t(42), At(104));
CheckEqual<std::uint64_t>(id, 5, "Emplace returns the new slot");
Check(table[id][&Transaction::name] == "Emplaced", "Emplace string_view argument");
Check(table[id][&Transaction::iban] == "NL00EMPL0000000001", "Emplace std::string argument");
Check(table[id][&Transaction::amount] == 42, "Emplace scalar argument");
const Transaction materialized = static_cast<Transaction>(table[id]);
CheckEqual<std::string>(materialized.description, "via Emplace", "explicit conversion to T");
const std::string described = std::format("{}", table[4]);
Check(described.find("name=\"a\"") != std::string::npos && described.find("amount=-1") != std::string::npos, std::format("row formatter output: {}", described));
std::uint64_t seen = 0;
for (DB::Row<Transaction> row : table) seen += row.Id() == seen ? 1 : 0;
CheckEqual<std::uint64_t>(seen, 6, "range-for visits every live row in order");
bool threw = false;
try {
table.Append(Transaction{ "late", "", "", "", Currency::Euro, 0, At(50) });
} catch (const DB::OrderError&) {
threw = true;
}
Check(threw, "out-of-order append throws OrderError");
CheckEqual<std::uint64_t>(table.Size(), 6, "failed append added nothing");
}
{
const DB::Table<Transaction> reopened(path, { .mode = DB::OpenMode::OpenExisting });
CheckEqual<std::uint64_t>(reopened.Size(), 6, "size after reopen");
const std::vector<Transaction> all = reopened.Materialize();
Check(all.size() == 6 && std::equal(sample.begin(), sample.end(), all.begin()), "rows survive reopen");
Check(reopened.Back()[&Transaction::name] == "Emplaced", "Emplaced row survives reopen");
const DB::Inspection info = DB::TableFile::Inspect(path);
CheckEqual<std::uint32_t>(info.header.schemaVersion, 1, "Inspect: schema version");
CheckEqual<std::size_t>(info.schema.columns.size(), 8, "Inspect: column count incl. flags");
CheckEqual<std::uint64_t>(info.commit.rowCount, 6, "Inspect: committed rows");
Check(info.header.schemaFingerprint == DB::Layout<Transaction>::fingerprint, "Inspect: fingerprint matches Layout");
Check(DB::MakeSchemaInfo<Transaction>().Fingerprint(DB::Layout<Transaction>::orderedColumn) == DB::Layout<Transaction>::fingerprint, "runtime and compile-time fingerprints agree");
CheckEqual<std::uint32_t>(DB::PeekVersion(path), 1, "PeekVersion");
}
{
const DB::Table<Transaction> frozen(path, { .readOnly = true });
CheckEqual<std::uint64_t>(frozen.Size(), 6, "read-only open sees rows");
bool threw = false;
try {
DB::Table<Transaction>(path, { .readOnly = true }); // second shared lock must coexist
} catch (const DB::Error&) {
threw = true;
}
Check(!threw, "two read-only opens coexist");
}
std::uint64_t temps = 0;
for (const fs::directory_entry& entry : fs::directory_iterator(dir / "sub")) {
if (entry.path().filename().string().find(".tmp-") != std::string::npos) temps++;
}
CheckEqual<std::uint64_t>(temps, 0, "no temp files left behind");
bool threwExisting = false;
try {
DB::Table<Transaction> missing(dir / "missing.cdb", { .mode = DB::OpenMode::OpenExisting });
} catch (const DB::IOError&) {
threwExisting = true;
}
Check(threwExisting, "OpenExisting on a missing file throws IOError");
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}

111
tests/ShouldVerify/main.cpp Normal file
View file

@ -0,0 +1,111 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Verify() reports a heap handle pointing outside the committed heap and an
// ordered column out of sequence, and is silent on a healthy file; the
// type-erased DynamicRow reads any column by name without the struct.
import Crafter.DB;
import std;
using namespace Crafter;
namespace fs = std::filesystem;
namespace {
int failures = 0;
void Check(bool condition, std::string_view what) {
if (!condition) {
std::println("FAIL: {}", what);
++failures;
}
}
struct Reading {
std::string sensor;
std::optional<std::string> comment;
double value;
std::int64_t at;
static constexpr std::uint32_t version = 1;
static constexpr auto orderedBy = &Reading::at;
};
void Patch(const fs::path& path, std::uint64_t offset, std::span<const std::byte> bytes) {
std::fstream file(path, std::ios::in | std::ios::out | std::ios::binary);
file.seekp(static_cast<std::streamoff>(offset));
file.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
}
}
int main() {
const fs::path dir = fs::temp_directory_path() / "crafter-db-verify";
fs::remove_all(dir);
const fs::path path = dir / "r.cdb";
std::uint64_t rowsOffset = 0;
{
DB::Table<Reading> table(path);
table.Append({ "thermometer-kitchen", std::nullopt, 21.5, 100 });
table.Append({ "hygrometer", std::string("calibrated last week"), 40.0, 200 });
table.Append({ "barometer-outside-wall", std::nullopt, 1013.25, 300 });
table.Erase(1);
const DB::VerifyResult healthy = table.Verify();
Check(healthy.Ok(), healthy.Ok() ? "healthy table verifies" : std::format("healthy table verifies: {}", healthy.problems[0]));
const DB::DynamicRow dynamic = table.File().Dynamic(2);
const std::optional<std::size_t> sensor = dynamic.Find("sensor");
Check(sensor && dynamic.Text(*sensor) == "barometer-outside-wall", "DynamicRow text by name");
const std::optional<std::size_t> value = dynamic.Find("value");
Check(value && dynamic.Scalar(*value) && std::get<double>(*dynamic.Scalar(*value)) == 1013.25, "DynamicRow double by name");
const std::optional<std::size_t> comment = dynamic.Find("comment");
Check(comment && !dynamic.Present(*comment), "DynamicRow sees a disengaged optional");
Check(table.File().Dynamic(1).Deleted() && table.File().Dynamic(1).Present(*comment) && table.File().Dynamic(1).Text(*comment) == "calibrated last week", "DynamicRow on a tombstoned row with an engaged optional");
const std::string described = dynamic.Describe();
Check(described.find("sensor=\"barometer-outside-wall\"") != std::string::npos && described.find("comment=null") != std::string::npos && described.find("at=300") != std::string::npos, std::format("Describe: {}", described));
Check(!dynamic.Find("nope").has_value(), "unknown column name");
table.Flush();
const DB::Inspection info = DB::TableFile::Inspect(path);
rowsOffset = info.header.extents[0][0].fileOffset;
}
// Damage the heap offset of row 2's long sensor name (heap value).
{
const fs::path damaged = dir / "handle.cdb";
fs::copy_file(path, damaged);
const DB::Inspection info = DB::TableFile::Inspect(damaged);
std::uint16_t sensorOffset = 0;
for (std::size_t i = 0; i < info.schema.columns.size(); i++) {
if (info.schema.Name(i) == "sensor") sensorOffset = info.schema.columns[i].offset;
}
const std::uint64_t bogus = 1 << 30;
Patch(damaged, rowsOffset + 2 * info.header.rowSize + sensorOffset + 8, std::as_bytes(std::span(&bogus, 1)));
const DB::Table<Reading> table(damaged, { .readOnly = true });
const DB::VerifyResult result = table.Verify();
Check(!result.Ok() && result.problems[0].find("row 2") != std::string::npos && result.problems[0].find("sensor") != std::string::npos, result.Ok() ? "damaged handle reported" : std::format("damaged handle reported: {}", result.problems[0]));
bool threw = false;
try {
[[maybe_unused]] const std::string_view text = table[2][&Reading::sensor];
} catch (const DB::CorruptError&) {
threw = true;
}
Check(threw, "reading through the damaged handle throws CorruptError instead of faulting");
Check(table[0][&Reading::sensor] == "thermometer-kitchen", "other rows still read");
}
// Break the order of the ordered column.
{
const fs::path damaged = dir / "order.cdb";
fs::copy_file(path, damaged);
const DB::Inspection info = DB::TableFile::Inspect(damaged);
const std::int64_t early = 50;
Patch(damaged, rowsOffset + 2 * info.header.rowSize + info.schema.columns[info.header.orderedColumn].offset, std::as_bytes(std::span(&early, 1)));
const DB::Table<Reading> table(damaged, { .readOnly = true });
const DB::VerifyResult result = table.Verify();
Check(!result.Ok() && result.problems[0].find("out of order") != std::string::npos, result.Ok() ? "order violation reported" : std::format("order violation reported: {}", result.problems[0]));
}
fs::remove_all(dir);
if (failures == 0) std::println("ok");
return failures == 0 ? 0 : 1;
}