Compare commits

...
Sign in to create a new pull request.
Author SHA1 Message Date
653dd0bb9b
fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is
Some checks failed
Flatpak / flatpak (push) Failing after 1m12s
- Shaders record the files glslang opened (a depfile) and their settings
  (source path, stage, entry point, defines, include directories) in the
  build directory, and rebuild when an included file or a setting changes,
  not only the shader's own source.
- An executable relinks when its link command changed, not only when an
  object is newer: a link flag in project.cpp, or a dependency's library
  that moved (an ExternalDependency built into a new cache directory). The
  command is recorded beside the objects; its flags come from ordered sets,
  so the order the dependency threads finish in can't change it.
- An external's newest artifact counts its libDirs and versioned .so names.
- Submodules are checked out at the revisions the fetched commit records,
  from the URLs its .gitmodules gives, after every clone, switch, checkout
  and pull, so a pinned commit builds against its own submodules, and one
  moved by hand is put back; with no .gitmodules, git isn't asked.
- Each external clone's fetch and build hold a file lock: parallel builds
  pulled into one clone at once ("cannot lock ref").
- mingw:
  - -stdlib=libstdc++ always;
  - the std module's source is found through libstdc++.modules.json in the
    mingw toolchain's own library directories (the generic lib/ holds the
    host's Linux one), asking the compiler once a run;
  - a toolchain that isn't there is a build error saying where it looked,
    not std::terminate;
  - CRAFTER_MINGW_DIR names the prefix, the directory holding
    x86_64-w64-mingw32/ and lib/gcc/;
  - inside the Flatpak the host's /usr mingw is used.
- RunScript's exit-status check reads as the one range it is.
- Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules,
  LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and
  ToolchainResolve extended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:20:29 +02:00
959c65dbb2
fix(script): a script that throws anything at all exits 1
RunScript caught std::exception only. Anything else a script threw went
on across the project.so boundary, and crafter-build ended in
std::terminate instead of reporting the script's failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
631c60892a
docs: scripts in the README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
2f8f25fd39
feat(packaging): run flatpak and run install for a local Flatpak
The local Flatpak build was one long flatpak-builder line in AGENTS.md,
with its state wherever the caller happened to stand, and installing
the result was not written down anywhere. It is not a rare chore: the
development box compiles with the distro's clang and the Flatpak with
its bundled LLVM, a major version apart at the moment, so anything
toolchain-sensitive has to be tried in the Flatpak too.

Both are now scripts in project.cpp, the first users of cfg.scripts.

`run flatpak` builds the manifest as it is, from the working tree, and
ends with `crafter-build --version` from inside the result. Its state
is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and
not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in
the sandbox, so it does not depend on the host variant, and a
variant-keyed directory would strand the LLVM cache on every host
compiler update. The manifest already skips build/ when it copies the
sources. flatpak-builder keys its cache on the manifest's path, so the
manifest is never rewritten into a copy the way CI stamps the version
in, and a local build reports 0.0.0-dev. The only argument accepted is
--jobs=<N>: anything that ends the run without a build would leave the
smoke test checking the previous one.

`run install` installs the last export as branch `local` from a user
remote of its own and makes it current, leaving the published branch
installed. It does not build first: flatpak-builder never caches a
`type: dir` module, so every build reruns build.sh and the suite.
remote-add --if-not-exists is followed by remote-modify --url, because
the remote may exist and point at another checkout.

Both refuse to run from the Flatpak build of crafter-build, which has
no flatpak to drive.

The smoke test goes through `flatpak build` and not `<builder> --run`,
which is what CI uses. flatpak-builder 1.4.9 run as the
org.flatpak.Builder app exits 1 when the command it ran succeeded and 0
when it failed (`true` gives 1, `exit 3` gives 0), so the script
reported a good build as a failure.

Measured on the 56-core box with LLVM and the WASI sysroot cached: 260
seconds, the crafter-build module only, 29 tests passing in the sandbox.
A second `run install` after a rebuild moved the installed commit.

`clean` now also drops that state, about 2 GB and a cold LLVM build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
8d031a1dba
test(script): names, arguments and exit codes, in process and by CLI
RunScript on hand-built configurations: the named script alone runs,
with the project, the project directory and the arguments exactly as
given; its return value comes back whole from 0 to 255 and as 1 outside
that; a script that throws exits 1. A project without scripts, a
duplicate, an untypeable name, a missing callback and an unknown name
all fail without running anything. ctx.Run returns the command's exit
code and 128+signal for a killed one, and ShellQuote carries quotes,
spaces and expansions through sh intact.

Then the same through Run() with a fixture project, because the parsing
is where this goes wrong. `run record a --help -v clean --debug
--version run` must hand all seven words to the script, leave the one
project arg that stood before `run` alone, and not delete the staged
bin/ and build/. `run clean` and `run help` look for scripts of those
names. Moving the two capture branches below the verbs in the parse
loop fails this test.

The command-line half needs the installed module sources and is skipped
without them, but only when the in-process half reported no failure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
561398013c
feat(script): scripts in project.cpp, run with crafter-build run
A project.cpp could describe how a project is built, tested and linted,
but not the chores around it: packaging, installing, publishing. Those
stayed shell lines in a README or in someone's history.

Configuration gains `scripts`, a plain list like `defines`. A Script is
a name and a C++ callback, as a lint rule is; there is no script
language and no config file. The callback gets a ScriptContext (the
project, the project file's directory, the arguments) and returns the
exit status. ctx.Run runs a shell command with the terminal attached
and returns its status. It is only stored when project.cpp loads, which
happens for every verb and for every project that depends on this one.

`crafter-build run <name> [args...]` runs one, a bare `run` lists them.
Only the scripts of the project crafter-build was pointed at are
runnable, never a dependency's.

Everything after the script's name is handed to the script untouched.
The two branches that do this sit ahead of every other branch of the
parse loop, --version and --help included. Below the verbs,
`run package clean` would delete build/, and any other word would land
in the project's args and so in the variant hash, moving the build to
another directory. Options before the name keep their meaning, so
project args go there.

The driver is its own partition, :Script. It refuses a project with no
scripts, a name that is empty or starts with '-' (it could never be
typed), a name declared twice (running either would be a guess) and an
unknown name, each with exit 1 and without running anything. A script
that throws exits 1. A return value outside 0..255 becomes 1: an exit
status is eight bits, and a raw wait status such as 256 would otherwise
wrap to success.

Two helpers in :Platform:

- RunCommandAttached runs a command with inherited stdio. Every helper
  so far captured through popen, which is right for a compile step and
  useless for a command that runs for half an hour or asks a question.
  It reads the wait status through DecodeWaitStatus, which
  RunCommandChecked and RunCommandWithTimeout now share.
- ShellQuote is exported. It existed twice, as ShellQuote in :External
  and ShellQuoteSh in :Test, with the same behaviour; both are gone. A
  script that puts a path into a command needs it too.

Not included: a string shorthand for one-command scripts (a one-line
lambda covers it) and a ctx.Build() helper (Build() is already
exported).

A new partition and new exported symbols: run ./build.sh.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
6dcdad4515
test(cli): a word after format is a glob, not the clean verb
Drives Run() with `format clean` on a staged project whose project file
cannot compile. Both trees must survive, and the run must fail: getting
as far as loading the project is what shows it went on to format rather
than cleaning and exiting 0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
7530954529
fix(cli): format clean no longer deletes bin/ and build/
Every verb is a bare word that is only read as a verb while no
conflicting verb is active; after one, the same word is a glob. The
guard on `clean` checked the test and lint verbs and then itself, where
the format verb was meant. `crafter-build format clean`, which asks for
the rules matching "clean", therefore removed both trees and exited 0
without formatting anything.

The guard now names format, like the three beside it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-20 19:08:26 +02:00
53add32f0f
docs(packaging): add the 2.2.0 release entry
All checks were successful
Flatpak / flatpak (push) Successful in 15m0s
CI takes the newest <release> as the notes of a tag and refuses a tag
that does not match it. 2.2.0 covers what changed since 2.1.0: Shader
entries with per-entry defines and a named output variant, sysroot
compiles against the sysroot's own libc++ headers, a project.so stamp
that hashes every part of the host build, and the vendored json.hpp
removed.

The variant API is what catcrafts.net's deploy needs: Crafter.Graphics
master has used it since 2026-10-06, and a stable Flatpak that lacks it
cannot build the site.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-16 23:55:58 +02:00
4af9280e79
removed json.hpp
All checks were successful
Flatpak / flatpak (push) Successful in 11m49s
2026-10-06 17:05:42 +02:00
fe38cc1f66
docs: sysroot compiles use the sysroot's libc++ headers
Some checks failed
Flatpak / flatpak (pull_request) Successful in 7m54s
Flatpak / flatpak (push) Has been cancelled
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
4ca8f14c70
test(sysroot): the compile command forces the sysroot's libc++
GetCompileCommand is checked as text, without a compiler: the forced
include directory is present for linux-gnu and linux-musl targets with
a sysroot, and absent without a sysroot, for wasm32 and for mingw.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
3c96d78a7d
fix(sysroot): compile against the sysroot's libc++ headers
clang's driver ranks the host toolchain's own libc++ headers
(<install>/../include/c++/v1) above --sysroot. BuildStdPcm has forced
the sysroot's copy for the std module for that reason, but ordinary
compiles did not. A unit that only does `import std;` never noticed. A
unit that includes a standard header textually got the host's libc++,
configured for the host's libc: cross-building Crafter.Graphics for
aarch64 musl, Crafter.Asset's vendored gdeflate died on <chrono> with
"unknown rune table for this platform". Against a glibc sysroot the
same unit would compile against the wrong libc++ release and say
nothing.

Every non-wasm, non-mingw compile with a sysroot now gets
`-nostdinc++ -isystem <sysroot>/usr/include/c++/v1`. wasm has its own
sysroot handling and mingw is a libstdc++ target.

-nostdinc++ leaves -stdlib=libc++ unused on each compile, which clang
reports once per unit, so the command also carries
-Wno-unused-command-line-argument. The cost is that the driver's
unused-argument diagnostics are silent on sysroot builds, including for
a user's own misplaced flag (a -l or -Wl, option in compileFlags).

Not addressed: external cmake dependencies are still configured
without the target or the sysroot, and objects already built in an
existing sysroot build directory are not invalidated by the new flags.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
51d87a990f
test(shader): variants through the Shader API and through Build()
ShaderCompile registers one source twice and checks that the variant
gets its own output name, takes the other #ifdef branch and receives
the value of a NAME=VALUE define. Staleness is pinned per output: while
only the plain output exists the variant must still be due, or Build()
would never compile a variant added to a built project. Malformed
defines and a variant with a path separator must be refused by name.

ShaderVariantBuild drives the same through Build(), which is where the
entries compile in parallel into one directory: a variant added to an
already built configuration is compiled into its own file, and two
entries with one output name fail the build without touching the
existing output. Its fixture tests the macro with `#if`, which pins
that a bare define is 1 and not empty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
ddaab5a32a
feat(shader): per-entry defines and a named output variant
Crafter.Graphics has to ship each of its compute shaders in two builds
of one source: one that indexes VK_EXT_descriptor_heap and one that
reads the same slots from a descriptor set, for drivers without the
extension. A Shader was a path, an entry point and a stage, and its
output name was derived from the path alone, so one source could only
be compiled once.

Shader gains `defines`, spelled like the compiler's -D ("NAME" is
defined to 1, "NAME=VALUE" to VALUE) and handed to glslang as a
preamble, and `variant`, which names the output: "ui.comp.glsl" with
variant "sets" becomes "ui.comp.sets.spv". OutputName() is the one
place that name is formed; Check, Compile and the forwarding of a
dependency's shaders into the consumer's bin dir all use it.

Two fields make one mistake easy: defines set, variant forgotten. Both
entries then compiled in parallel into the same file, the build
reported success with whichever finished last, and Check() called both
up to date from then on. The same has always been true of a.vert next
to a.frag. Build() now refuses a configuration in which two entries
share an output name and says which.

A define without a name, or one spanning lines, and a variant with a
path separator are rejected by name. Left to glslang the first two
surface as "-1:<line>" with no file, and the last as a failed open
after the whole compile.

Not addressed: Check() still compares only the source's mtime, so a
change to the defines, or to an included file, does not recompile
(#35).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
e985f2724b
test(host-cache): changed declarations recompile project.so
HostCacheSourceStamp already edits a module source with a backdated
mtime and expects the PCM cache to be rebuilt. Its own header names the
real danger, a project.so compiled against declarations that no longer
match the process it is loaded into, but it never looked at the
project.so. It now records the file's mtime before the edit and expects
a newer one after it, with project.cpp untouched, so only the stamp can
force the rebuild.

Fails without the previous commit: the stamp was the version string
alone and the project.so was reused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
e280c955f7
fix(host-cache): hash every part of the project.so stamp
The stamp that decides whether a compiled project file can be reused
was meant to cover the crafter-build version, the toolchain key, the
module-source stamp and the compile command. Only the version went in.
The four parts were joined by a format string with "\0" between the
fields, and a format string is read as a C string: it ended at the
first NUL, so std::format saw "{}" and dropped the other arguments
without a diagnostic.

A project.so is loaded into the crafter-build process and fills in
Configuration objects by their layout. With the stamp blind to the
declarations, one built against other module sources was loaded as
current whenever the version string was unchanged, which is every
working-tree build (`git describe --dirty` does not follow the tree's
contents) and every local Flatpak build. Growing a class that project
files construct then corrupts the heap: a project.so built while Shader
was 56 bytes, loaded after it had become 104, aborted with "free():
invalid size" as soon as it declared a shader. CI builds were spared
only because each carries its own version.

The parts are now appended to a std::string with NUL separators, the
way the other stamps in this file are built. Every existing project.so
carries a stamp of the version alone, so each is recompiled once.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 16:50:41 +02:00
c6b0bbaf25
readme change
All checks were successful
Flatpak / flatpak (push) Successful in 13m43s
2026-10-03 21:16:01 +02:00
7abbdf503b
readme change
Some checks failed
Flatpak / flatpak (push) Has been cancelled
2026-10-03 21:14:30 +02:00
486935bce8
docs(packaging): add the 2.1.0 release entry
All checks were successful
Flatpak / flatpak (push) Successful in 19m32s
CI takes the newest <release> as the notes of a tag and refuses a tag
that does not match it. 2.1.0 covers what changed since 2.0.0: the
bundled LLVM 23.1.2, build outputs kept per compiler, and the std
module source chosen by the sysroot's libc++ release.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 20:09:22 +02:00
03728fd4a7
docs: describe the output directory name and what a compiler update does
The README still gave the output path as bin/<name>-<target>-<march>/,
which has not been its shape since the variant hash was introduced. It
now shows the real one and says what the hash stands for, including
that a compiler update starts a new directory and `clean` removes the
old. AGENTS.md lists the toolchain among VariantId()'s inputs and notes
that a glob over variant directories matches two after an update.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 20:09:22 +02:00
995a40e7dd
test(incremental): a compiler update perturbs VariantId
Two toolchains that differ only in the clang release, as the Flatpak's
do across an update, must give one Configuration two variants; so must
the same release installed under another prefix. The no-argument form
is checked to be the variant under the running toolchain, and BuildDir
to be named for it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 20:09:22 +02:00
b5a366d060
fix(incremental): key build outputs on the compiler
A project built with one clang kept its outputs when crafter-build
started using another. The output directory is named by VariantId(),
which hashed every input that changes what is compiled except the
compiler itself, and staleness is decided by mtimes, which a compiler
update leaves alone. A PCM can only be read by the clang release that
wrote it, so after the bundled LLVM moved from 22 to 23 the first build
found nothing stale, and the first edit failed with "module file
Greeter.pcm uses an older format that is no longer supported" until the
project was cleaned by hand. A project with a `#pragma crafter
dependency` did not get that far: its project file imports the
dependency's PCM, so it failed to load, and for a git dependency that
PCM lives in the cache, where `crafter-build clean` does not reach. The
same happens natively when the distro updates clang.

The toolchain's key now goes into the hash, as the host PCM cache was
already keyed on it. An update starts an empty variant directory and
rebuilds into it; the previous one stays until `clean`, as after any
other change of variant, and two crafter-build versions can work in one
checkout without disturbing each other. VariantId() gains an overload
taking the toolchain so the dependence can be tested in-process, the
way ResolveToolchain and SysrootStdModule take theirs.

External cmake builds are not covered: their directory is fixed per
clone and keyed on the compiler's path, not its release. Objects of two
clang releases link together on the host, so nothing fails there, but
they are not rebuilt either.

Verified by replaying the update: three projects built with the 2.0.0
Flatpak, then built, edited and run with this one without cleaning,
then built with 2.0.0 again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 20:09:22 +02:00
6e9eedfd21
fix(packaging): check out the WASI sysroot's LLVM tree from git
All checks were successful
Flatpak / flatpak (push) Successful in 19m22s
The previous commit made the LLVM tree inside the wasi-sdk checkout a
git repository with `git init`, so that wasi-sdk's patch step would
apply. That was half the fix: wasi-sdk also runs `git rev-parse HEAD`
in that tree to write the sysroot's VERSION file, which on an empty
repository raises a traceback during configure and leaves VERSION
without its llvm lines. The build went on regardless, so the suite
passed and the published beta carries the truncated file.

Fetching the tree as a git source pinned to the release tag gives
wasi-sdk what it has upstream, a submodule checkout at a commit, and
removes the shell step instead of stacking a second workaround on it.
The llvm module keeps the release tarball: it is the same commit, and a
tarball is the cheaper download for a tree that is only compiled.
flatpak-builder keeps a mirror of the clone in its state directory, so
the fetch costs the cold run about 300 MB once.

Verified locally: no traceback, VERSION lists llvm 85ac56026243 and
llvm-version 23.1.2, the eight patch steps apply, the suite passes in
the sandbox and a wasm32-wasip1 program runs under the bundled wasmtime.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 19:04:47 +02:00
4e5d17b9e9
docs: describe how a sysroot build finds its std module source
All checks were successful
Flatpak / flatpak (push) Successful in 1h30m57s
The README said a cross build precompiles the sysroot's own std.cppm,
which was never the whole story for sysroots that do not ship one. It
now states the order (the sysroot's, the toolchain's for the same major
release, the sources crafter-build carries for other releases), what
the failure looks like when nothing matches and how to resolve it, and
lists the new directory in the install layout.

AGENTS.md records the rule for the next LLVM bump: the release the
bundled toolchain leaves behind stays available as a module-source
module, because stable distros remain on it long after.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 17:01:21 +02:00
4159ee4117
feat(packaging): ship libc++ 22's std module source for sysroots
With LLVM 23 bundled, a sysroot still on libc++ 22 without its own
module source (Alpine v3.24) is the mirror image of the case that
started this: the toolchain's source is a release ahead of the
sysroot's headers. The new libcxx-22-modules module installs libc++
22.1.8's module source under share/crafter-build/libc++-modules/22,
where SysrootStdModule looks for the release a sysroot is on.

It only configures libc++'s runtimes build and installs the cxx-modules
component: std.cppm as configure generates it and the .inc files from
the source tree, nothing compiled, about 600 KB installed. The
component is staged and copied because it also writes
lib/libc++.modules.json, which belongs to the bundled libc++.

Verified in the built Flatpak: `crafter-build --version` lists release
22, and a cross build against an Alpine v3.24 aarch64 sysroot (libc++
22.1.3) precompiles that source with clang 23, links and runs under
qemu.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 17:01:21 +02:00
8dccb88b78
feat(packaging): bundle LLVM 23.1.2
LLVM 23 has been the stable series since September, and the distros
crafter-build cross-builds for have moved to it: Alpine edge ships
libc++ 23.1.2, which is what surfaced the std module mismatch fixed in
the previous commits. wasi-sdk 34 already builds against a newer LLVM
than 22 upstream, so the WASI sysroot loses nothing by following.

The bump exposed a patch step that never ran. wasi-sdk patches its LLVM
tree with `git apply` in src/llvm-project, which upstream is a
submodule. Here that directory is a tarball unpacked inside the
wasi-sdk checkout, and git, finding the outer repository, treats every
patched path as outside the current directory, prints "Skipped patch"
and exits 0. LLVM 22 built without the three patches; libc++ 23 does
not compile for WASI without the one that keeps the z/OS locale shim
out of other platforms' builds (strtod_l redeclared with an abi_tag).
`git init` on the unpacked tree makes it a repository of its own, so
wasi-sdk's step applies whatever patch list the pinned tag carries, as
it does upstream, and fails loudly when a patch no longer fits.

Built locally: the whole manifest, the suite inside the sandbox (26
passed, WasmHelloRuns under the bundled wasmtime included), and aarch64
musl cross builds against Alpine edge and v3.24 sysroots that run under
qemu. The native development loop is unaffected and stays on whatever
clang the machine has.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 17:01:21 +02:00
cbf2b94f06
test(sysroot): cover the std module source choice for a sysroot build
SysrootStdModule runs over staged layouts that need no compiler: a
sysroot with its own module source, one on a point release of the
toolchain's libc++, one on the next major release with a shipped
source, one nothing was shipped for, one without libc++ at all, and the
two cases where a release cannot be read. The last block drives Build()
against a sysroot claiming libc++ 99 and expects the one-line
explanation back instead of compiler output, which is the failure the
fix replaces.

ToolchainResolve additionally checks that the release is read from the
libc++ the module source comes from (the prefix's, an explicit libc++
install's, the system's) and where the shipped sources are looked up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 16:37:00 +02:00
e15cd3c14d
fix(sysroot): take the std module source from the sysroot's libc++ release
A cross build against a sysroot whose libc++ differs from the
toolchain's failed while precompiling the std module, with a page of
"no member named 'runtime_format' in namespace 'std'". The module source
is a list of using-declarations for what one libc++ release declares, so
it only compiles against that release's headers. BuildStdPcm wants the
sysroot's own <sysroot>/usr/share/libc++/v1/std.cppm, but Alpine
packages the headers without it, and the fallback was the toolchain's
copy whatever its release. That held while the compiler came from the
same distro as the sysroot; with the Flatpak's libc++ 22.1.8 and Alpine
edge on 23.1.2 it no longer does.

The release is now read from _LIBCPP_VERSION in the sysroot's and the
toolchain's <__config>, and SysrootStdModule picks in this order: the
sysroot's own source, the toolchain's when both are the same major
release (a release branch only takes fixes), the copy crafter-build
ships for that release under share/crafter-build/libc++-modules/<major>.
When none fits the build stops with one line naming both releases and
the directory the matching source belongs in, before clang is run. If
either release cannot be read the toolchain's source is tried as
before, since there is nothing to compare.

Matching sources live in crafter-build's own share directory rather
than the LLVM prefix: they are data this tool ships for sysroots, not
part of any one compiler install, and "sources ship" already describes
that directory. `--version` prints the toolchain's libc++ release and
the releases sources are shipped for, the first thing to look at when
such a build fails.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 16:37:00 +02:00
adc91ed9e2
fix(packaging): keep the per-commit version out of every module's cache key
flatpak-builder printed "Building module llvm" on every CI run instead
of "Cache hit". The build environment, CRAFTER_BUILD_VERSION included,
sat in the manifest's top-level build options, and those are part of
every module's cache key; the version changes with each commit, so no
module ever hit. Runs still took twelve minutes rather than eighty only
because ccache made the LLVM rebuild cheap.

The variables only matter to crafter-build's own module, so they move
there, and CI rewrites the version on that module. LLVM, the WASI
sysroot and wasmtime are now keyed on nothing that changes between
commits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 16:04:43 +02:00
c7f984b95d
fix(ci): name master builds after the release they follow
All checks were successful
Flatpak / flatpak (push) Successful in 20m44s
Every non-tag build called itself 0.0.0-dev+<hash>, a placeholder that
said nothing about which release it came after and put a `+` in the
bundle's download URL. A master build now uses `git describe`, as a
local checkout already does: 2.0.0-3-g21746b1 is three commits past
2.0.0. The placeholder remains only for history that has no tag yet.

Three more things the tag path needed before its first run. The
forgejo-release action takes the notes text in `release-notes` and a
path in `release-notes-file`; the workflow passed a path to the former,
so the release body would have been the file name. The tag filter is
spelled with globs so it reads the same as a workflow pattern or a plain
glob. Python, used to extract the notes from the metainfo, was only in
the container as somebody else's dependency and is now listed. The
2.0.0 entry carries the date it is tagged on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 15:41:57 +02:00
21746b188d
fix(ci): initialise the published repo when its directory is empty
All checks were successful
Flatpak / flatpak (push) Successful in 25m45s
The publish step created the OSTree repo only when the directory was
missing. On the runner host the web root exists before the first
publish, as an empty directory, so the init was skipped and
build-commit-from failed with `opendir(objects)`. The check now looks
for the repo's config file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 14:40:09 +02:00
b55fe31ddf
docs: describe the dedicated flatpak runner
Some checks failed
Flatpak / flatpak (push) Failing after 21m20s
The privileged runner that already existed is registered to another
repository, so its labels cannot serve this one. CI runs on a separate
privileged runner registered to Crafter.Build only, with four volumes
rather than three now that the signing key is mounted from the host.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 14:17:24 +02:00
7eb837fd55
fix(ci): sign with a key kept on the runner host
Some checks failed
Flatpak / flatpak (push) Failing after 1h23m15s
The runner is our own machine, so the repo signing key does not need to
pass through the forge as a secret. It is generated once under
/etc/crafter-flatpak on the host and bind-mounted read-only into the job
container, where each run imports it into a throwaway keyring. One less
secret to manage, and the private key never leaves the box that uses it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 23:52:13 +02:00
46 changed files with 2732 additions and 25617 deletions

View file

@ -7,15 +7,19 @@ name: Flatpak
# tag X.Y.Z -> publish as `stable` and create release X.Y.Z # tag X.Y.Z -> publish as `stable` and create release X.Y.Z
# #
# The `flatpak` runner label is the privileged runner on the Hetzner box # The `flatpak` runner label is the privileged runner on the Hetzner box
# (bubblewrap needs it) with three volumes allowed: the Sdk install, the # (bubblewrap needs it) with four volumes allowed: the Sdk install, the
# flatpak-builder state (ccache and the LLVM module, so only crafter-build # flatpak-builder state (ccache and the LLVM module, so only crafter-build
# itself rebuilds on a normal run) and the published repo's web root. # itself rebuilds on a normal run), the published repo's web root, and the
# repo signing key, which lives on the runner host and never in the forge.
on: on:
pull_request: pull_request:
branches: [master] branches: [master]
push: push:
branches: [master] branches: [master]
tags: ['[0-9]+.[0-9]+.[0-9]+'] # Bare semver tags. Spelled with globs so it means the same whether the
# forge reads it as a workflow pattern or as a plain glob; `latest`, the
# rolling tag this workflow moves itself, does not start with a digit.
tags: ['[0-9]*.[0-9]*.[0-9]*']
workflow_dispatch: workflow_dispatch:
env: env:
@ -35,13 +39,14 @@ jobs:
-v crafter-flatpak-system:/var/lib/flatpak -v crafter-flatpak-system:/var/lib/flatpak
-v crafter-flatpak-builder:/root/.cache/flatpak-builder -v crafter-flatpak-builder:/root/.cache/flatpak-builder
-v /var/www/flatpak:/srv/flatpak -v /var/www/flatpak:/srv/flatpak
-v /etc/crafter-flatpak:/run/crafter-flatpak:ro
steps: steps:
- name: Install tools - name: Install tools
run: | run: |
pacman-key --init pacman-key --init
pacman-key --populate archlinux pacman-key --populate archlinux
pacman -Sy --noconfirm --needed archlinux-keyring pacman -Sy --noconfirm --needed archlinux-keyring
pacman -Syu --noconfirm --needed flatpak flatpak-builder ccache git nodejs jq gnupg go-yq ostree pacman -Syu --noconfirm --needed flatpak flatpak-builder ccache git nodejs jq gnupg go-yq ostree python
git config --global --add safe.directory '*' git config --global --add safe.directory '*'
- name: Sdk - name: Sdk
@ -72,7 +77,15 @@ jobs:
exit 1 exit 1
fi fi
else else
version="0.0.0-dev+$(git rev-parse --short HEAD)" # A master build names itself after the release it follows, the
# way a local checkout does: 2.0.0-3-g21746b1 is three commits
# past 2.0.0. Before the first tag there is nothing to describe
# from and the bare hash gets a placeholder in front.
described=$(git describe --tags --always --exclude=latest)
case "$described" in
[0-9]*.[0-9]*.[0-9]*) version="$described" ;;
*) version="0.0.0-dev+$described" ;;
esac
branch=beta branch=beta
fi fi
echo "version=$version" >> "$GITHUB_OUTPUT" echo "version=$version" >> "$GITHUB_OUTPUT"
@ -86,7 +99,7 @@ jobs:
VERSION: ${{ steps.meta.outputs.version }} VERSION: ${{ steps.meta.outputs.version }}
run: | run: |
set -eu set -eu
yq '."build-options".env.CRAFTER_BUILD_VERSION = strenv(VERSION)' "$MANIFEST" > "$CI_MANIFEST" yq '(.modules[] | select(.name == "crafter-build") | ."build-options".env.CRAFTER_BUILD_VERSION) = strenv(VERSION)' "$MANIFEST" > "$CI_MANIFEST"
grep -n 'CRAFTER_BUILD_VERSION' "$CI_MANIFEST" grep -n 'CRAFTER_BUILD_VERSION' "$CI_MANIFEST"
flatpak-builder --force-clean --ccache --disable-rofiles-fuse \ flatpak-builder --force-clean --ccache --disable-rofiles-fuse \
--state-dir=/root/.cache/flatpak-builder \ --state-dir=/root/.cache/flatpak-builder \
@ -118,16 +131,23 @@ jobs:
- name: Publish to the Catcrafts repo - name: Publish to the Catcrafts repo
if: github.event_name != 'pull_request' if: github.event_name != 'pull_request'
env:
FLATPAK_GPG_PRIVATE_KEY: ${{ secrets.FLATPAK_GPG_PRIVATE_KEY }}
FLATPAK_GPG_KEY_ID: ${{ secrets.FLATPAK_GPG_KEY_ID }}
run: | run: |
set -eu set -eu
# The signing key is generated once on the runner host
# (/etc/crafter-flatpak: signing-key.asc and key-id) and mounted
# read-only; it is imported into a throwaway keyring per run.
if [ ! -s /run/crafter-flatpak/signing-key.asc ] || [ ! -s /run/crafter-flatpak/key-id ]; then
echo "no signing key under /etc/crafter-flatpak on the runner host; generate it as described in AGENTS.md (CI and publishing)" >&2
exit 1
fi
export GNUPGHOME=$(mktemp -d) export GNUPGHOME=$(mktemp -d)
printf '%s' "$FLATPAK_GPG_PRIVATE_KEY" | gpg --batch --quiet --import gpg --batch --quiet --import /run/crafter-flatpak/signing-key.asc
FLATPAK_GPG_KEY_ID=$(cat /run/crafter-flatpak/key-id)
# The live repo is the canonical copy; builds are promoted into it, # The live repo is the canonical copy; builds are promoted into it,
# never built into it. # never built into it.
[ -d "$REPO_DIR" ] || ostree init --repo="$REPO_DIR" --mode=archive # Keyed on the repo's config file, not the directory: the web root
# may exist as an empty directory before the first publish.
[ -f "$REPO_DIR/config" ] || ostree init --repo="$REPO_DIR" --mode=archive
flatpak build-commit-from --src-repo=repo \ flatpak build-commit-from --src-repo=repo \
--gpg-sign="$FLATPAK_GPG_KEY_ID" --gpg-homedir="$GNUPGHOME" \ --gpg-sign="$FLATPAK_GPG_KEY_ID" --gpg-homedir="$GNUPGHOME" \
"$REPO_DIR" "app/$APP_ID/x86_64/${{ steps.meta.outputs.branch }}" "$REPO_DIR" "app/$APP_ID/x86_64/${{ steps.meta.outputs.branch }}"
@ -180,6 +200,6 @@ jobs:
repo: ${{ github.repository }} repo: ${{ github.repository }}
tag: ${{ github.ref_name }} tag: ${{ github.ref_name }}
title: crafter-build ${{ github.ref_name }} title: crafter-build ${{ github.ref_name }}
release-notes: release-notes.md release-notes-file: release-notes.md
release-dir: dist release-dir: dist
token: ${{ secrets.GITHUB_TOKEN }} token: ${{ secrets.GITHUB_TOKEN }}

View file

@ -10,7 +10,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj
|---|---| |---|---|
| `interfaces/Crafter.Build-<Partition>.cppm` | Module interface partitions. `Crafter.Build.cppm` is the umbrella that re-exports them. | | `interfaces/Crafter.Build-<Partition>.cppm` | Module interface partitions. `Crafter.Build.cppm` is the umbrella that re-exports them. |
| `implementations/Crafter.Build-<Partition>.cpp` | Implementation units, one per partition, named `Crafter.Build:<Partition>_impl`. `main.cpp` is a thin wrapper around `Crafter::Run`. | | `implementations/Crafter.Build-<Partition>.cpp` | Implementation units, one per partition, named `Crafter.Build:<Partition>_impl`. `main.cpp` is a thin wrapper around `Crafter::Run`. |
| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations and the lint rules. | | `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations, the lint rules and the `flatpak` and `install` scripts. |
| `tests/<Name>/main.cpp` | One self-test per directory; `tests/<Name>/fixture/` holds the sample project it builds. | | `tests/<Name>/main.cpp` | One self-test per directory; `tests/<Name>/fixture/` holds the sample project it builds. |
| `examples/` | Standalone user-facing projects, MIT licensed. | | `examples/` | Standalone user-facing projects, MIT licensed. |
| `build.sh` | Bootstrap script. It compiles everything by hand, without crafter-build. | | `build.sh` | Bootstrap script. It compiles everything by hand, without crafter-build. |
@ -21,13 +21,14 @@ crafter-build is a C++26 modules build system. A project is described by a `proj
Partitions and what they own: Partitions and what they own:
- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. - `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release.
- `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint and test data types. - `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types.
- `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code. - `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code.
- `:Interface` and `:Implementation` model module units, their staleness checks and their compiles. - `:Interface` and `:Implementation` model module units, their staleness checks and their compiles.
- `:External` handles git + cmake dependencies, `GitProject` and `LocalProject`. - `:External` handles git + cmake dependencies, `GitProject` and `LocalProject`.
- `:Test` holds `AddTest`, `RunTests` and `TestRunner`. - `:Test` holds `AddTest`, `RunTests` and `TestRunner`.
- `:Lint` is the lint/format driver plus the libclang token and AST layer. - `:Lint` is the lint/format driver plus the libclang token and AST layer.
- `:Script` is the driver behind `crafter-build run`: it looks a name up in `cfg.scripts`, calls the callback and turns what it returns into the exit code.
- `:Progress` is console output. `:Shader` compiles shaders through glslang. `:Asset` is the hook into Crafter.Asset. - `:Progress` is console output. `:Shader` compiles shaders through glslang. `:Asset` is the hook into Crafter.Asset.
## Dev environment ## Dev environment
@ -47,7 +48,7 @@ Three things to know about this loop:
- Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. - Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added.
- A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol. - A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol.
`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. `./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`.
## Build, test and style commands ## Build, test and style commands
@ -60,6 +61,8 @@ Three things to know about this loop:
./bin/crafter-build lint # dry style gate, exit 1 on any finding ./bin/crafter-build lint # dry style gate, exit 1 on any finding
./bin/crafter-build format # apply the transform rules in place ./bin/crafter-build format # apply the transform rules in place
./bin/crafter-build format --check # CI-style dry run ./bin/crafter-build format --check # CI-style dry run
./bin/crafter-build run # list the scripts project.cpp defines
./bin/crafter-build run <name> [args...] # run one; everything after the name is the script's
./bin/crafter-build clean ./bin/crafter-build clean
``` ```
@ -110,7 +113,7 @@ Beyond the rules:
- There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests. - There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests.
- A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`. - A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`.
- Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones. - Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones.
- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, wasm variant flags and uninterpreted project args. - Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, wasm variant flags, uninterpreted project args and the toolchain (`Toolchain::Key()`).
## Design rules ## Design rules
@ -142,21 +145,44 @@ These were decided deliberately. Do not undo them without the maintainer's say-s
- `project.cpp` is compiled and loaded into the crafter-build process, and every `GitProject`, `#pragma crafter dependency` and `ExternalDependency` clones a repository and runs its build on the host. Treat a dependency URL as code you are choosing to run. This repo's own dependencies are sibling Catcrafts repos tracking their default branch on purpose; pin with `commit` when adding anything else, and whenever reproducibility matters. - `project.cpp` is compiled and loaded into the crafter-build process, and every `GitProject`, `#pragma crafter dependency` and `ExternalDependency` clones a repository and runs its build on the host. Treat a dependency URL as code you are choosing to run. This repo's own dependencies are sibling Catcrafts repos tracking their default branch on purpose; pin with `commit` when adding anything else, and whenever reproducibility matters.
- Unpinned dependencies `git pull` on every invocation, so builds and tests need network. A missing or private repo surfaces as a clone failure before anything is built. - Unpinned dependencies `git pull` on every invocation, so builds and tests need network. A missing or private repo surfaces as a clone failure before anything is built.
- `--runner=cmd:<command>` executes the named wrapper around every test binary. Wrappers come from `--runner=` or `CRAFTER_BUILD_RUNNER_*`; do not hardcode hosts, wrappers or credentials in `project.cpp` or tests. - `--runner=cmd:<command>` executes the named wrapper around every test binary. Wrappers come from `--runner=` or `CRAFTER_BUILD_RUNNER_*`; do not hardcode hosts, wrappers or credentials in `project.cpp` or tests.
- `crafter-build run <name>` executes whatever the script does, with your permissions. Only the scripts of the project crafter-build was pointed at can be run, never a dependency's; keep it that way.
- The linter loads `libclang` at runtime by name. `CRAFTER_BUILD_LIBCLANG` pins an exact path and must fail loudly when wrong, never fall back to another library. - The linter loads `libclang` at runtime by name. `CRAFTER_BUILD_LIBCLANG` pins an exact path and must fail loudly when wrong, never fall back to another library.
- CI force-pushes the rolling `latest` tag and publishes a prerelease on every push to `master` with the workflow's own token, and signs the Flatpak repo with the key in the `FLATPAK_GPG_PRIVATE_KEY` secret. Do not add other secrets to the workflow or echo the environment in it. - CI force-pushes the rolling `latest` tag and publishes a prerelease on every push to `master` with the workflow's own token, and signs the Flatpak repo with the key mounted from the runner host. Do not add secrets to the workflow or echo the environment in it.
- Tests write only under `fs::temp_directory_path()` and the fixture's own `bin/` and `build/`. Keep it that way. - Tests write only under `fs::temp_directory_path()` and the fixture's own `bin/` and `build/`. Keep it that way.
## CI and publishing ## CI and publishing
`.forgejo/workflows/ci.yaml` runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box (bubblewrap inside the job container needs it). Its `config.yml` allows three volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change) and `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`. Secrets: `FLATPAK_GPG_PRIVATE_KEY` (armored, no passphrase) and `FLATPAK_GPG_KEY_ID`. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it. `.forgejo/workflows/ci.yaml` runs on the `flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change; a cold run takes about 80 minutes on 8 cores, a warm one a fraction of that), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it.
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. Local builds: `flatpak-builder --user --force-clean --ccache --repo=repo build-dir packaging/flatpak/net.catcrafts.CrafterBuild.yaml`, then `flatpak-builder --run build-dir <manifest> crafter-build --version`. The key is made once, as root on the runner host, and the step fails with a message pointing here when it is missing:
```bash
install -d -m 700 /etc/crafter-flatpak/gnupg && export GNUPGHOME=/etc/crafter-flatpak/gnupg
gpg --batch --pinentry-mode loopback --passphrase '' --quick-generate-key 'Catcrafts Flatpak Repository <flatpak@catcrafts.net>' ed25519 sign never
KEYID=$(gpg --list-secret-keys --with-colons | awk -F: '/^sec/{print $5; exit}')
gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armor "$KEYID" > /etc/crafter-flatpak/signing-key.asc
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id
```
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry.
Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive.
```bash
./bin/crafter-build run flatpak [--jobs=<N>] # build the Flatpak, then print its version from inside the sandbox
./bin/crafter-build run install # install that build as branch `local` and make it current
```
`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart.
`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone.
## Gotchas ## Gotchas
- A new partition is listed in four places, in import order: `PARTITIONS` in `build.sh`, `CrafterBuildModules` in `Crafter.Build-Platform.cpp`, both arrays in `project.cpp`, and the umbrella `Crafter.Build.cppm`. Then `./build.sh`.
- A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away. - A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away.
- A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard. - A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard.
- `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache. - `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache.
- The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop. - The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop.
- A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone. - A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. - A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot.
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both.

View file

@ -1,6 +1,6 @@
# Crafter Build # Crafter Build
A C++26 modules build system. Project descriptions are written in C++ with no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it; Crafter Build compiles, loads, and executes it. A C++26 modules build system. Project descriptions are written in C++ with no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it. Crafter Build compiles, loads, and executes it.
## Install ## Install
@ -14,19 +14,6 @@ alias crafter-build='flatpak run net.catcrafts.CrafterBuild'
Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host. Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host.
## Building from source (Linux)
The development loop is native. It needs `clang`, `cmake`, `git`, `lld` and `libc++` with its std module source:
```bash
./build.sh # produces bin/crafter-build
./bin/crafter-build # rebuild via project.cpp
```
The compiler is resolved once per run: `CRAFTER_BUILD_LLVM_DIR` if set, else a `clang++` installed beside crafter-build (`<prefix>/bin`), else PATH. `crafter-build --version` prints the version and the compiler in use; `crafter-build --help` lists the other overrides (`CRAFTER_BUILD_LIBCXX_DIR`, `CRAFTER_BUILD_LIBCLANG`, `CRAFTER_BUILD_WASI_SYSROOT`, `CRAFTER_MINGW_DIR`). A `Configuration` targets the host triple unless `cfg.target` says otherwise.
The bootstrap targets the toolchain's own triple (`clang++ -print-target-triple`); override with `CRAFTER_BUILD_TARGET=`. The Flatpak is built from `packaging/flatpak/net.catcrafts.CrafterBuild.yaml`, which runs this same bootstrap, the self-host build and the test suite inside the sandbox against the bundled toolchain (`CRAFTER_BUILD_LLVM_DIR=/app`).
## Writing a project.cpp ## Writing a project.cpp
Crafter Build loads a `project.cpp` from the current directory. The file exports one function that returns a populated `Configuration`: Crafter Build loads a `project.cpp` from the current directory. The file exports one function that returns a populated `Configuration`:
@ -52,7 +39,7 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
} }
``` ```
Run `crafter-build` in that directory; outputs land at `bin/myapp-<target>-<march>/myapp` and intermediates at `build/myapp-<target>-<march>/`. Add `-r` to run the produced executable straight after a successful build. Run `crafter-build` in that directory, outputs land at `bin/myapp-<target>-<march>-<mtune>-<hash>/myapp` and intermediates at `build/myapp-<target>-<march>-<mtune>-<hash>/`. The hash covers everything that changes what is compiled (build type, defines, flags, sysroot, the compiler itself), so two such builds never share a directory. After a compiler update, a new crafter-build release included, the first build starts a fresh directory and recompiles. `crafter-build clean` removes the ones left behind. Add `-r` to run the produced executable straight after a successful build.
## Dependencies ## Dependencies
@ -73,7 +60,7 @@ glslang.includeDirs = { "" };
glslang.libs = { "SPIRV", "glslang", "OSDependent" }; glslang.libs = { "SPIRV", "glslang", "OSDependent" };
``` ```
Clones live at `~/.cache/crafter.build/external/<name>-<hash>/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes. Clones live at `~/.cache/crafter.build/external/<name>-<hash>/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes. Submodules are checked out at the revisions the fetched commit records, after every clone, switch, checkout and pull, so a pinned commit builds against its own submodules and not the default branch's.
**Header-only git deps** — same as cmake deps but `builder = ExternalBuilder::None`. Just clones and propagates `-I` flags. **Header-only git deps** — same as cmake deps but `builder = ExternalBuilder::None`. Just clones and propagates `-I` flags.
@ -88,7 +75,7 @@ Configuration* sibling = LocalProject({ .projectFile = "../Crafter.Math/project.
cfg.dependencies = { math }; cfg.dependencies = { math };
``` ```
Clones share the external cache and identical specs share one `Configuration`. An unpinned source is `git pull`ed on every invocation; set `.commit` for reproducible, offline-after-first-fetch builds. Clones share the external cache and identical specs share one `Configuration`. An unpinned source is `git pull`ed on every invocation, set `.commit` for reproducible, offline-after-first-fetch builds.
A project loaded this way runs its `CrafterBuildProject` as a dependency, and `IsTopLevelProject()` is `false` while it does (`true` in the project crafter-build was pointed at, and outside any project file). Wire test-only dependencies behind it: a library whose test links crafter-build itself would otherwise load crafter-build's `project.cpp` from inside crafter-build and recurse. A project loaded this way runs its `CrafterBuildProject` as a dependency, and `IsTopLevelProject()` is `false` while it does (`true` in the project crafter-build was pointed at, and outside any project file). Wire test-only dependencies behind it: a library whose test links crafter-build itself would otherwise load crafter-build's `project.cpp` from inside crafter-build and recurse.
@ -107,22 +94,6 @@ import Crafter.LintRules;
#pragma crafter dependency "<source>" [branch "<name>" | commit "<sha>"] [project "<file>"] #pragma crafter dependency "<source>" [branch "<name>" | commit "<sha>"] [project "<file>"]
``` ```
## Install layout
Inside the Flatpak:
```
/app/bin/crafter-build # the executable
/app/bin/clang++ lld llvm-ar wasmtime # the bundled toolchain the executable resolves first
/app/lib/libcrafter-build.a # static archive for downstream consumers
/app/share/crafter-build/*.cppm # module sources (rebuilt on the user's machine on first run)
/app/share/wasi-sysroot/ # WASI sysroot for wasm32 targets
~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/<target>-<march>-<clang>-<hash>/
# std.pcm + Crafter.Build*.pcm, built locally per toolchain
```
A source build uses the same shape under the checkout (`bin/`, `lib/`, `share/crafter-build/`) with the system compiler and `~/.cache/crafter.build/`.
## Tests ## Tests
Tests live under `tests/<Name>/`. The simplest case is a single C++ file: Tests live under `tests/<Name>/`. The simplest case is a single C++ file:
@ -141,7 +112,7 @@ Test conventions:
- **Exit codes**: `0` = ✅ pass, nonzero = ❌ fail, **`77`** = ⏭ skipped (autoconf convention; use it for runtime opt-outs like "tool not on PATH"). - **Exit codes**: `0` = ✅ pass, nonzero = ❌ fail, **`77`** = ⏭ skipped (autoconf convention; use it for runtime opt-outs like "tool not on PATH").
- Each test runs in its own subprocess — a segfault doesn't take the runner down. - Each test runs in its own subprocess — a segfault doesn't take the runner down.
- Default timeout is 60 s per test; runs in parallel up to `hardware_concurrency()`. - Default timeout is 60 s per test, runs in parallel up to `hardware_concurrency()`.
- Stdout+stderr is captured and printed atomically on completion. Failed/crashed/timed-out tests also write to `build/test-logs/<name>.log`. - Stdout+stderr is captured and printed atomically on completion. Failed/crashed/timed-out tests also write to `build/test-logs/<name>.log`.
- A broken fixture `project.cpp` is reported as a single failed test, not a fatal error — other tests still run. - A broken fixture `project.cpp` is reported as a single failed test, not a fatal error — other tests still run.
@ -159,7 +130,7 @@ crafter-build test --timeout=5 # override per-t
Outcomes: ✅ pass, ❌ fail, 💥 crash (with signal name), ⏱ timeout, ⏭ skipped. The runner exits 0 only if every non-skipped test passed. Outcomes: ✅ pass, ❌ fail, 💥 crash (with signal name), ⏱ timeout, ⏭ skipped. The runner exits 0 only if every non-skipped test passed.
`--target=` filters the run to tests whose `cfg.target` matches; default is the host triple. Per-target tests parse `--target=...` from their own args span and build for it; tests with hardcoded targets (e.g. host-only meta-tests) are filtered out under non-matching `--target=`. `--target=` filters the run to tests whose `cfg.target` matches, default is the host triple. Per-target tests parse `--target=...` from their own args span and build for it, tests with hardcoded targets (e.g. host-only meta-tests) are filtered out under non-matching `--target=`.
### Cross-target test runners ### Cross-target test runners
@ -174,9 +145,9 @@ TestRunner::Cmd("wine") // local exec via prefix co
For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_PREFIX` for sysroots), construct `TestRunner` directly — `copy`/`exec`/`cleanup`/`probe` are public string fields with `{bundle}`, `{remote_bundle}`, `{bin}`, `{bin_name}`, `{args}` placeholders. For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_PREFIX` for sysroots), construct `TestRunner` directly — `copy`/`exec`/`cleanup`/`probe` are public string fields with `{bundle}`, `{remote_bundle}`, `{bin}`, `{bin_name}`, `{args}` placeholders.
For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=<path>` is passed to clang, and `<sysroot>/usr/share/libc++/v1/std.cppm` is precompiled instead of the host's. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root <rootfs> --config <alarm-pacman.conf> --dbpath <rootfs>/var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=<path>` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. Ordinary compiles are pointed at the same headers (`-nostdinc++ -isystem <sysroot>/usr/include/c++/v1`), so a unit that `#include`s a standard header gets the sysroot's libc++ and not the host toolchain's. The module source has to come from the same libc++ release as those headers, so it is taken from `<sysroot>/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules/<major>/`, `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases, installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root <rootfs> --config <alarm-pacman.conf> --dbpath <rootfs>/var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages).
For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported; there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags.
### Configuring runners ### Configuring runners
@ -205,7 +176,7 @@ local
cmd:<command> # cmd:wine, cmd:qemu-aarch64, cmd:valgrind, ... cmd:<command> # cmd:wine, cmd:qemu-aarch64, cmd:valgrind, ...
``` ```
Each runner carries a `probe` command (e.g. `which wine`); if it fails, every test using that runner is reported as ⏭ skipped and the rest of the suite continues. Skipped tests count toward the summary but don't fail the suite (`AllPassed()` returns true on skip-only runs). Each runner carries a `probe` command (e.g. `which wine`), if it fails, every test using that runner is reported as ⏭ skipped and the rest of the suite continues. Skipped tests count toward the summary but don't fail the suite (`AllPassed()` returns true on skip-only runs).
### Linking the parent project's library ### Linking the parent project's library
@ -232,7 +203,7 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view>)
## Linting & formatting ## Linting & formatting
Lint rules are C++ in `project.cpp` with no `.lintrc`, no YAML, and no built-in rules. A rule is a named callback that runs once per source file; register it on the `Configuration` before returning: Lint rules are C++ in `project.cpp` with no `.lintrc`, no YAML, and no built-in rules. A rule is a named callback that runs once per source file, register it on the `Configuration` before returning:
```cpp ```cpp
cfg.AddLintRule("no-tabs", [](Crafter::LintContext& ctx) { cfg.AddLintRule("no-tabs", [](Crafter::LintContext& ctx) {
@ -273,7 +244,7 @@ Lint findings print compiler-style — `file:line: warning: message [rule]` —
| `Extension()` | file extension, e.g. `".cppm"` | | `Extension()` | file extension, e.g. `".cppm"` |
| `CommentStripped()` | `content` with comments and string/char literal bodies blanked to spaces — newlines kept, so line numbers stay valid (raw strings not recognized yet) | | `CommentStripped()` | `content` with comments and string/char literal bodies blanked to spaces — newlines kept, so line numbers stay valid (raw strings not recognized yet) |
| `Report(line, msg)` | record a finding (`line == 0` for whole-file) | | `Report(line, msg)` | record a finding (`line == 0` for whole-file) |
| `SetContent(str)` | replace the file's content — makes the rule a transform; `lines` re-splits, `CommentStripped()` re-derives. Build the new string, call once at the end; don't assign `content` directly | | `SetContent(str)` | replace the file's content — makes the rule a transform, `lines` re-splits, `CommentStripped()` re-derives. Build the new string, call once at the end, don't assign `content` directly |
Suppression is comment-based : Suppression is comment-based :
@ -285,6 +256,50 @@ Suppression is comment-based :
// lint-disable-file — all rules, whole file // lint-disable-file — all rules, whole file
``` ```
## Scripts
A project has chores that are not part of building it: packaging, installing, publishing, generating documentation. A script gives one a name. Scripts are C++ in `project.cpp` like everything else, a name and a callback pushed onto `cfg.scripts` before returning the `Configuration`:
```cpp
cfg.scripts.push_back({"docs", [](Crafter::ScriptContext& ctx) {
return ctx.Run("python3 tools/gen-docs.py");
}});
```
```bash
crafter-build run # list the project's scripts
crafter-build run docs # run one
crafter-build run docs --open api # everything after the name is handed to the script
```
The callback returns the exit status, and `crafter-build run` exits with it. `ctx.Run(command)` runs a shell command with the terminal attached, so its output appears as it is produced and it can ask questions, and returns the command's exit code (128 plus the signal number when it was killed). A script with several steps stops at the first failure by returning that code:
```cpp
cfg.scripts.push_back({"package", [](Crafter::ScriptContext& ctx) {
const std::string dist = Crafter::ShellQuote((ctx.projectDir / "dist").string());
if (const std::int32_t code = ctx.Run(std::format("mkdir -p {}", dist)); code != 0) return code;
return ctx.Run(std::format("tar -czf {}/myapp.tar.gz -C {} .", dist, Crafter::ShellQuote(ctx.project.BinDir().string())));
}});
```
`ScriptContext` gives each script:
| Member | Meaning |
|---|---|
| `project` | the `Configuration` that `project.cpp` returned. Read the project through this; a `cfg` captured by reference is gone by the time a script runs |
| `projectDir` | absolute directory of the project file |
| `args` | everything typed after the script's name, untouched |
| `Run(command)` | run a shell command with the terminal attached and return its exit code |
Things to know:
- Everything after the script's name belongs to the script, including what looks like an option of crafter-build (`--help`, `-v`). Options and project args for crafter-build go before the name: `crafter-build --debug run package`.
- `run` loads `project.cpp` but does not build the project. A script that needs the build output runs after `crafter-build`.
- Commands run in the directory crafter-build was started from. Build paths from `ctx.projectDir`, and quote anything that is not a literal with `Crafter::ShellQuote`.
- Only the scripts of the project you run crafter-build in are available. A dependency's scripts are not.
- Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host.
- A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed.
## Examples ## Examples
[examples/](examples/) contains progressively larger self-contained projects: [examples/](examples/) contains progressively larger self-contained projects:
@ -304,7 +319,7 @@ This project is licensed under the GNU Lesser General Public License, version 3
The example code under [examples/](examples/) is licensed under the MIT license (see [examples/LICENSE](examples/LICENSE)), so it can be freely copied into your own projects. The example code under [examples/](examples/) is licensed under the MIT license (see [examples/LICENSE](examples/LICENSE)), so it can be freely copied into your own projects.
The [lib/](lib/) directory contains vendored third-party code (nlohmann/json, glslang's DirStackFileIncluder) that remains under its own permissive licenses; see the headers of those files. The [lib/](lib/) directory contains vendored third-party code (nlohmann/json, glslang's DirStackFileIncluder) that remains under its own permissive licenses, see the headers of those files.
## Copyright ## Copyright

View file

@ -48,7 +48,7 @@ cp wasi-runtime/index.html.in share/crafter-build/wasi-runtime/
# Partition order is the import order: every unit is listed before the # Partition order is the import order: every unit is listed before the
# units that import it, and the umbrella last. # units that import it, and the umbrella last.
PARTITIONS="Toolchain Shader Platform Interface Implementation External Clang Test Lint Progress Asset" PARTITIONS="Toolchain Shader Platform Interface Implementation External Clang Test Lint Script Progress Asset"
cp interfaces/Crafter.Build.cppm share/crafter-build/ cp interfaces/Crafter.Build.cppm share/crafter-build/
for p in $PARTITIONS; do for p in $PARTITIONS; do

View file

@ -12,6 +12,7 @@ import :Platform;
import :Toolchain; import :Toolchain;
import :Test; import :Test;
import :Lint; import :Lint;
import :Script;
import :Progress; import :Progress;
import :Asset; import :Asset;
namespace fs = std::filesystem; namespace fs = std::filesystem;
@ -296,6 +297,21 @@ CompileCommand Crafter::GetCompileCommand(const Configuration& config) {
if (!config.sysroot.empty()) { if (!config.sysroot.empty()) {
out.command += std::format(" --sysroot={}", config.sysroot); out.command += std::format(" --sysroot={}", config.sysroot);
// Same reason as in BuildStdPcm: the driver ranks the host
// toolchain's own libc++ headers (<install>/../include/c++/v1) above
// --sysroot. A unit that only does `import std;` never notices, but
// one that includes a standard header textually (vendored C++ such as
// Crafter.Asset's gdeflate) would get host libc++ configured for the
// host libc, which fails on a musl sysroot with "unknown rune table
// for this platform". Force the sysroot's copy. -nostdinc++ leaves
// -stdlib=libc++ unused on every compile ("argument unused during
// compilation"), hence the silencer; the price is that it also hides
// that diagnostic for a user's own misplaced flag (a -l or -Wl,
// option in compileFlags) on sysroot builds. Not for mingw, which
// uses libstdc++ and has no libc++ in its sysroot.
if (!isWasm && config.target != "x86_64-w64-mingw32") {
out.command += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1 -Wno-unused-command-line-argument", config.sysroot);
}
} }
if (isWasm) { if (isWasm) {
// -mllvm is consumed by codegen but not the link driver, which is the // -mllvm is consumed by codegen but not the link driver, which is the
@ -502,13 +518,31 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
std::string buildError; std::string buildError;
std::atomic<bool> buildCancelled{false}; std::atomic<bool> buildCancelled{false};
// Two entries with the same output name would compile in parallel into
// one file: whichever finished last won, the build reported success, and
// Check() then called both up to date. The usual way in is one source
// registered twice with different defines and no variant; a.vert next to
// a.frag is another.
{
std::unordered_map<std::string, const Shader*> outputs;
for (const Shader& shader : config.shaders) {
const auto [existing, inserted] = outputs.emplace(shader.OutputName().string(), &shader);
if (inserted) continue;
buildError = std::format("shaders {} and {} both compile to {}: give one of them a distinct Shader::variant", existing->second->path.string(), shader.path.string(), existing->first);
buildCancelled.store(true);
break;
}
}
for (const Shader& shader : config.shaders) { for (const Shader& shader : config.shaders) {
if (shader.Check(outputDir)) continue; if (buildCancelled.load(std::memory_order_relaxed)) break;
threads.emplace_back([&shader, &outputDir, &shaderIncludeDirs, &buildError, &buildCancelled]() { if (shader.Check(outputDir, buildDir, shaderIncludeDirs)) continue;
Progress::Task task(std::format("Compiling shader {}", shader.path.filename().string())); threads.emplace_back([&shader, &outputDir, &buildDir, &shaderIncludeDirs, &buildError, &buildCancelled]() {
// A source registered more than once shows up once per variant;
// name the variant so the lines can be told apart.
Progress::Task task(shader.variant.empty() ? std::format("Compiling shader {}", shader.path.filename().string()) : std::format("Compiling shader {} ({})", shader.path.filename().string(), shader.variant));
if (buildCancelled.load(std::memory_order_relaxed)) return; if (buildCancelled.load(std::memory_order_relaxed)) return;
std::string result = shader.Compile(outputDir, shaderIncludeDirs); std::string result = shader.Compile(outputDir, buildDir, shaderIncludeDirs);
if (result.empty()) return; if (result.empty()) return;
bool expected = false; bool expected = false;
@ -665,7 +699,13 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
std::string stdPcmResult; std::string stdPcmResult;
{ {
Progress::Task task(std::format("Building std PCM ({}-{})", config.target, config.march)); Progress::Task task(std::format("Building std PCM ({}-{})", config.target, config.march));
stdPcmResult = BuildStdPcm(config, stdPcmDir/"std.pcm"); // The shader and external threads are running: an exception leaving
// here would destroy them joinable, which terminates the process.
try {
stdPcmResult = BuildStdPcm(config, stdPcmDir/"std.pcm");
} catch (const std::exception& e) {
stdPcmResult = std::format("building the std module for {}: {}", config.target, e.what());
}
} }
if(!stdPcmResult.empty()) { if(!stdPcmResult.empty()) {
buildCancelled.store(true); buildCancelled.store(true);
@ -694,8 +734,8 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
const bool isWasm = config.target.starts_with("wasm32"); const bool isWasm = config.target.starts_with("wasm32");
std::string files; std::string files;
std::unordered_set<std::string> libSet; std::set<std::string> libSet;
std::unordered_set<std::string> publicFlagSet; std::set<std::string> publicFlagSet;
std::mutex fileMutex; std::mutex fileMutex;
std::vector<std::thread> depThreads; std::vector<std::thread> depThreads;
depThreads.reserve(config.dependencies.size()); depThreads.reserve(config.dependencies.size());
@ -770,6 +810,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
: std::format(" -march={} -mtune={}", config.march, config.mtune); : std::format(" -march={} -mtune={}", config.march, config.mtune);
if (!config.sysroot.empty()) { if (!config.sysroot.empty()) {
cArchFlags += std::format(" --sysroot={}", config.sysroot); cArchFlags += std::format(" --sysroot={}", config.sysroot);
} else if (config.target == "x86_64-w64-mingw32" && !GetToolchain().mingwSysroot.empty()) {
// The C++ side gets it through GetBaseCommand.
cArchFlags += std::format(" --sysroot={}", GetToolchain().mingwSysroot.string());
} }
if (cIsWasm) { if (cIsWasm) {
// Matches the C++ path's wasi flag set so any libc shim defines // Matches the C++ path's wasi flag set so any libc shim defines
@ -878,7 +921,7 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
if (!seen.insert(dep).second) return; if (!seen.insert(dep).second) return;
fs::path depBinDir = dep->BinDir(); fs::path depBinDir = dep->BinDir();
for (const Shader& shader : dep->shaders) { for (const Shader& shader : dep->shaders) {
fs::path src = depBinDir / shader.path.filename().replace_extension("spv"); fs::path src = depBinDir / shader.OutputName();
if (!fs::exists(src)) continue; if (!fs::exists(src)) continue;
copyTree(src, outputDir / src.filename()); copyTree(src, outputDir / src.filename());
} }
@ -1115,8 +1158,48 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
} }
} }
// The exact command the link (or archive) step runs, kept beside the
// objects once it has succeeded. A different one relinks even with no
// object newer than the output: a link flag changed in project.cpp, or a
// dependency's library moved, as an ExternalDependency's does when its
// options change and it is built into a new cache directory.
std::string linkCommand;
if (config.type == ConfigurationType::Executable) {
if (config.target.starts_with("wasm32")) {
linkCommand = std::format("{}{} -o {}.wasm -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras);
} else {
linkCommand = std::format("{}{} -o {} -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras);
}
} else if (config.type == ConfigurationType::LibraryStatic) {
// ThinLTO emits LLVM bitcode objects; plain `ar` writes an archive
// index that omits their symbols, so a consumer's lld link can't
// pull the needed members. llvm-ar writes a bitcode-aware index
// and handles plain ELF and COFF objects just as well, so every
// build uses it.
linkCommand = std::format("{} rcs {}.a {}", GetToolchain().llvmAr.string(), (outputDir/fs::path(std::string("lib")+config.outputName)).string(), files);
} else if (config.target == "x86_64-w64-mingw32") {
// LibraryDynamic. Output names follow each target's convention so
// consumers can link via the standard linker search:
// mingw: <name>.dll + lib<name>.dll.a (lld --out-implib)
// unix: lib<name>.so (rpath $ORIGIN)
fs::path dll = outputDir / std::format("{}.dll", config.outputName);
fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName);
linkCommand = std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras);
} else {
linkCommand = std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras);
}
const fs::path linkRecord = buildDir / std::format("{}.link", config.outputName);
if (!buildResult.repack) {
std::ifstream recorded(linkRecord, std::ios::binary);
const std::string previous((std::istreambuf_iterator<char>(recorded)), std::istreambuf_iterator<char>());
if (previous != linkCommand) buildResult.repack = true;
}
if(buildResult.repack) { if(buildResult.repack) {
Progress::Task task(std::format("Linking {}", config.outputName)); Progress::Task task(std::format("Linking {}", config.outputName));
// The record goes first, so a link that fails leaves it stale.
std::error_code removed;
fs::remove(linkRecord, removed);
if(config.type == ConfigurationType::Executable) { if(config.type == ConfigurationType::Executable) {
if(config.target == "x86_64-w64-mingw32") { if(config.target == "x86_64-w64-mingw32") {
try { try {
@ -1147,30 +1230,10 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
return {e.what(), false, {}}; return {e.what(), false, {}};
} }
} }
if (config.target.starts_with("wasm32")) { }
buildResult.result = RunBuildCommand(std::format("{}{} -o {}.wasm -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras)); buildResult.result = RunBuildCommand(linkCommand);
} else { if (buildResult.result.empty()) {
buildResult.result = RunBuildCommand(std::format("{}{} -o {} -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras)); std::ofstream(linkRecord, std::ios::binary | std::ios::trunc) << linkCommand;
}
} else if(config.type == ConfigurationType::LibraryStatic) {
// ThinLTO emits LLVM bitcode objects; plain `ar` writes an archive
// index that omits their symbols, so a consumer's lld link can't
// pull the needed members. llvm-ar writes a bitcode-aware index
// and handles plain ELF and COFF objects just as well, so every
// build uses it.
buildResult.result = RunBuildCommand(std::format("{} rcs {}.a {}", GetToolchain().llvmAr.string(), (outputDir/fs::path(std::string("lib")+config.outputName)).string(), files));
} else {
// LibraryDynamic. Output names follow each target's convention so
// consumers can link via the standard linker search:
// mingw: <name>.dll + lib<name>.dll.a (lld --out-implib)
// unix: lib<name>.so (rpath $ORIGIN)
if (config.target == "x86_64-w64-mingw32") {
fs::path dll = outputDir / std::format("{}.dll", config.outputName);
fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName);
buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras));
} else {
buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras));
}
} }
} }
@ -1537,6 +1600,20 @@ static void PrintVersion() {
return; return;
} }
std::println("compiler {} ({})", tc.clangVersion, tc.clangxx.string()); std::println("compiler {} ({})", tc.clangVersion, tc.clangxx.string());
if (tc.libcxxVersion != 0) {
// A sysroot on another libc++ release builds only when its std
// module source is in the sysroot or among these; printed here
// because that is the first question when such a build fails.
std::vector<std::string> shipped;
std::error_code ec;
for (const fs::directory_entry& entry : fs::directory_iterator(tc.stdModulesDir, ec)) {
if (fs::exists(entry.path() / "std.cppm")) shipped.push_back(entry.path().filename().string());
}
std::ranges::sort(shipped);
std::string also;
for (const std::string& release : shipped) also += std::format(", {}", release);
std::println("libc++ {}{}", LibcxxVersionString(tc.libcxxVersion), also.empty() ? std::string() : std::format(" (std module sources for sysroots also for{})", also.substr(1)));
}
std::println("host {}", tc.hostTriple); std::println("host {}", tc.hostTriple);
} }
@ -1547,6 +1624,7 @@ R"(Usage:
{0} test [test-options] [globs...] Build and run the project's tests {0} test [test-options] [globs...] Build and run the project's tests
{0} lint [lint-options] [globs...] Run the project's lint rules over its sources {0} lint [lint-options] [globs...] Run the project's lint rules over its sources
{0} format [format-options] [globs...] Apply the project's transform rules (rewrites files) {0} format [format-options] [globs...] Apply the project's transform rules (rewrites files)
{0} run [script [args...]] Run a script the project defines, or list them
{0} clean Delete the project's bin/ and build/ trees {0} clean Delete the project's bin/ and build/ trees
{0} help | -h | --help Show this help {0} help | -h | --help Show this help
{0} --version Print the version and the compiler in use {0} --version Print the version and the compiler in use
@ -1599,6 +1677,23 @@ Format options (after the `format` subcommand):
transform. `format` rewrites changed files in place; `lint` reports the transform. `format` rewrites changed files in place; `lint` reports the
same transforms as would-reformat findings without writing. same transforms as would-reformat findings without writing.
Run (after the `run` subcommand):
<script> [args...] Run the named script. Everything after the name is
handed to the script untouched, including what
looks like an option of crafter-build itself, so
-v, -q, --project= and project args go before it.
--list List the project's scripts. A bare `run` does too.
Scripts are defined in project.cpp, as entries of cfg.scripts:
cfg.scripts.push_back({{"docs", [](Crafter::ScriptContext& ctx) {{
return ctx.Run("python3 tools/gen-docs.py");
}}}});
The callback returns the exit status. ctx.Run runs a shell command with the
terminal attached and returns its status; ctx.args holds the arguments and
ctx.projectDir the project's directory. Commands run in the directory
crafter-build was started from. Only the scripts of the project being run
are available, never a dependency's.
Clean (after the `clean` subcommand): Clean (after the `clean` subcommand):
Removes bin/ and build/ next to the project file. Does not load project.cpp, Removes bin/ and build/ next to the project file. Does not load project.cpp,
so it works when the project itself no longer compiles. Every target, variant so it works when the project itself no longer compiles. Every target, variant
@ -1623,13 +1718,18 @@ Environment:
instead of the compiler's own. instead of the compiler's own.
CRAFTER_BUILD_LIBCLANG Exact libclang to load for lint. CRAFTER_BUILD_LIBCLANG Exact libclang to load for lint.
CRAFTER_BUILD_WASI_SYSROOT WASI sysroot for wasm32 targets. CRAFTER_BUILD_WASI_SYSROOT WASI sysroot for wasm32 targets.
CRAFTER_MINGW_DIR mingw-w64 sysroot, passed as --sysroot. CRAFTER_MINGW_DIR mingw-w64 prefix (holding x86_64-w64-mingw32/
and lib/gcc/), passed as --sysroot. Default:
the mingw gcc on PATH; in the Flatpak, the
host's /usr when it has mingw-w64.
CRAFTER_BUILD_HOME Directory holding the module sources. CRAFTER_BUILD_HOME Directory holding the module sources.
Exit status: Exit status:
0 success / all non-skipped tests passed / lint clean / format applied 0 success / all non-skipped tests passed / lint clean / format applied
1 build failure, one or more tests failed, lint findings, no rules defined, 1 build failure, one or more tests failed, lint findings, no rules defined,
format write errors, or `format --check` found files that would change format write errors, or `format --check` found files that would change
`run` exits with the status its script returned, and with 1 when there is no
such script, when the script throws, or when it returns a value outside 0..255
)", argv0); )", argv0);
} }
@ -1645,14 +1745,29 @@ std::int32_t Crafter::Run(std::int32_t argc, char** argv) {
bool runLint = false; bool runLint = false;
bool runFormat = false; bool runFormat = false;
bool runClean = false; bool runClean = false;
bool runScript = false;
bool runAfterBuild = false; bool runAfterBuild = false;
RunTestsOptions testOpts; RunTestsOptions testOpts;
RunLintOptions lintOpts; RunLintOptions lintOpts;
RunScriptOptions scriptOpts;
Progress::Verbosity verbosity = Progress::Verbosity::Default; Progress::Verbosity verbosity = Progress::Verbosity::Default;
for (std::int32_t i = 1; i < argc; ++i) { for (std::int32_t i = 1; i < argc; ++i) {
std::string_view arg = argv[i]; std::string_view arg = argv[i];
if (arg == "--version") { // The two `run` branches come before everything else on purpose.
// What follows a script's name belongs to the script, whatever
// it looks like: `run package --help` must not print this
// tool's help, `run package clean` must not delete build/, and
// none of it may reach projectArgs, where it would change the
// variant hash and so the directory the project builds into.
if (scriptOpts.named) {
scriptOpts.args.emplace_back(arg);
} else if (runScript && !arg.starts_with("-")) {
// The first word after `run`, even one that is a verb
// elsewhere. Options before it keep their usual meaning.
scriptOpts.name = arg;
scriptOpts.named = true;
} else if (arg == "--version") {
// Answered here, before the project file is looked for, so // Answered here, before the project file is looked for, so
// it works in any directory and is never forwarded to // it works in any directory and is never forwarded to
// project.cpp as an unknown argument. // project.cpp as an unknown argument.
@ -1665,11 +1780,15 @@ std::int32_t Crafter::Run(std::int32_t argc, char** argv) {
runTests = true; runTests = true;
} else if (!runTests && !runFormat && !runClean && arg == "lint") { } else if (!runTests && !runFormat && !runClean && arg == "lint") {
runLint = true; runLint = true;
} else if (!runTests && !runLint && !runClean && arg == "clean") { } else if (!runTests && !runLint && !runFormat && arg == "clean") {
runClean = true; runClean = true;
} else if (!runTests && !runLint && !runClean && arg == "format") { } else if (!runTests && !runLint && !runClean && arg == "format") {
runFormat = true; runFormat = true;
lintOpts.mode = LintMode::Apply; lintOpts.mode = LintMode::Apply;
} else if (!runTests && !runLint && !runFormat && !runClean && arg == "run") {
runScript = true;
} else if (runScript && arg == "--list") {
scriptOpts.listOnly = true;
} else if (runFormat && arg == "--check") { } else if (runFormat && arg == "--check") {
lintOpts.mode = LintMode::Check; lintOpts.mode = LintMode::Check;
} else if (arg == "-r") { } else if (arg == "-r") {
@ -1731,6 +1850,11 @@ std::int32_t Crafter::Run(std::int32_t argc, char** argv) {
Configuration config = LoadProject(projectFile, projectArgs); Configuration config = LoadProject(projectFile, projectArgs);
SetParentProject(&config); SetParentProject(&config);
if (runScript) {
scriptOpts.projectFile = fs::absolute(projectFile).lexically_normal();
return RunScript(config, scriptOpts);
}
if (runLint || runFormat) { if (runLint || runFormat) {
// lexically_normal: fs::absolute keeps a "./" component, which // lexically_normal: fs::absolute keeps a "./" component, which
// would leak into displayed paths and the project-root derivation. // would leak into displayed paths and the project-root derivation.

View file

@ -21,16 +21,6 @@ std::string DeriveName(const GitSource& source) {
return slash == std::string::npos ? url : url.substr(slash + 1); return slash == std::string::npos ? url : url.substr(slash + 1);
} }
std::string ShellQuote(std::string_view s) {
std::string out = "'";
for (char c : s) {
if (c == '\'') out += "'\\''";
else out += c;
}
out += "'";
return out;
}
std::string JoinOptions(std::span<const std::string> options) { std::string JoinOptions(std::span<const std::string> options) {
std::string out; std::string out;
for (const std::string& opt : options) { for (const std::string& opt : options) {
@ -46,6 +36,18 @@ std::string FetchGit(const GitSource& source, const fs::path& cloneDir) {
if (r.exitCode != 0) return std::format("`{}` failed (exit {}): {}", cmd, r.exitCode, r.output); if (r.exitCode != 0) return std::format("`{}` failed (exit {}): {}", cmd, r.exitCode, r.output);
return std::nullopt; return std::nullopt;
}; };
// Submodules follow the checkout only when asked to: after every clone,
// switch, checkout or pull, so they match the revision that was fetched
// instead of whichever one was checked out before, from the URLs that
// revision's .gitmodules gives (sync). A no-op, and offline, when they
// already match, which puts back submodules moved by hand; and no git at
// all without any.
auto updateSubmodules = [&runGit](const fs::path& dir) -> std::optional<std::string> {
if (!fs::exists(dir / ".gitmodules")) return std::nullopt;
const std::string quoted = ShellQuote(dir.string());
if (auto err = runGit(std::format("git -C {} submodule sync --recursive", quoted))) return err;
return runGit(std::format("git -C {} submodule update --init --recursive", quoted));
};
if (fs::exists(cloneDir)) { if (fs::exists(cloneDir)) {
CommandResult remote = RunCommandChecked(std::format("git -C {} remote get-url origin", ShellQuote(cloneDir.string()))); CommandResult remote = RunCommandChecked(std::format("git -C {} remote get-url origin", ShellQuote(cloneDir.string())));
@ -61,6 +63,9 @@ std::string FetchGit(const GitSource& source, const fs::path& cloneDir) {
if (auto err = runGit(std::format("git -C {} fetch origin", ShellQuote(cloneDir.string())))) return *err; if (auto err = runGit(std::format("git -C {} fetch origin", ShellQuote(cloneDir.string())))) return *err;
if (auto err = runGit(std::format("git -C {} checkout {}", ShellQuote(cloneDir.string()), ShellQuote(source.commit)))) return *err; if (auto err = runGit(std::format("git -C {} checkout {}", ShellQuote(cloneDir.string()), ShellQuote(source.commit)))) return *err;
} }
// Also when the commit already matched: a clone made before
// submodules were updated has them at the default branch's.
if (auto err = updateSubmodules(cloneDir)) return *err;
return ""; return "";
} else { } else {
// No commit pinned — pull latest. The user opted into "track latest" // No commit pinned — pull latest. The user opted into "track latest"
@ -71,6 +76,7 @@ std::string FetchGit(const GitSource& source, const fs::path& cloneDir) {
} else { } else {
if (auto err = runGit(std::format("git -C {} pull", ShellQuote(cloneDir.string())))) return *err; if (auto err = runGit(std::format("git -C {} pull", ShellQuote(cloneDir.string())))) return *err;
} }
if (auto err = updateSubmodules(cloneDir)) return *err;
return ""; return "";
} }
} }
@ -83,7 +89,9 @@ std::string FetchGit(const GitSource& source, const fs::path& cloneDir) {
fs::path tmpDir = cloneDir; fs::path tmpDir = cloneDir;
tmpDir += std::format(".tmp.{:08x}{:08x}", rd(), rd()); tmpDir += std::format(".tmp.{:08x}{:08x}", rd(), rd());
if (auto err = runGit(std::format("git clone --recursive {} {}", ShellQuote(source.url), ShellQuote(tmpDir.string())))) { // Not --recursive: that checks out the default branch's submodules, and
// the requested revision's come once it is checked out.
if (auto err = runGit(std::format("git clone {} {}", ShellQuote(source.url), ShellQuote(tmpDir.string())))) {
if (fs::exists(tmpDir)) fs::remove_all(tmpDir); if (fs::exists(tmpDir)) fs::remove_all(tmpDir);
return *err; return *err;
} }
@ -99,6 +107,10 @@ std::string FetchGit(const GitSource& source, const fs::path& cloneDir) {
return *err; return *err;
} }
} }
if (auto err = updateSubmodules(tmpDir)) {
fs::remove_all(tmpDir);
return *err;
}
std::error_code ec; std::error_code ec;
fs::rename(tmpDir, cloneDir, ec); fs::rename(tmpDir, cloneDir, ec);
@ -134,6 +146,10 @@ std::string BuildInjectedCMakeFlags(const fs::path& cmakeBuildDir, std::string_v
out += " -DCMAKE_SYSTEM_NAME=Windows"; out += " -DCMAKE_SYSTEM_NAME=Windows";
out += std::format(" -DCMAKE_C_COMPILER_TARGET={}", target); out += std::format(" -DCMAKE_C_COMPILER_TARGET={}", target);
out += std::format(" -DCMAKE_CXX_COMPILER_TARGET={}", target); out += std::format(" -DCMAKE_CXX_COMPILER_TARGET={}", target);
// The libstdc++ and the mingw-w64 install the project's own compiles
// use (GetBaseCommand).
out += " -DCMAKE_CXX_FLAGS=-stdlib=libstdc++";
if (!tc.mingwSysroot.empty()) out += std::format(" -DCMAKE_SYSROOT={}", ShellQuote(tc.mingwSysroot.string()));
} }
fs::path absBuildDir = fs::absolute(cmakeBuildDir); fs::path absBuildDir = fs::absolute(cmakeBuildDir);
out += std::format(" -DCMAKE_ARCHIVE_OUTPUT_DIRECTORY={}", ShellQuote(absBuildDir.string())); out += std::format(" -DCMAKE_ARCHIVE_OUTPUT_DIRECTORY={}", ShellQuote(absBuildDir.string()));
@ -237,6 +253,9 @@ ExternalBuildResult Crafter::BuildExternal(const ExternalDependency& dep, std::s
} }
fs::path cloneDir = ExternalCloneDir(dep, target); fs::path cloneDir = ExternalCloneDir(dep, target);
// Builds running side by side (tests built in parallel, two
// crafter-builds) share the clone: one fetches and builds it at a time.
FileLock lock(fs::path(std::format("{}.lock", cloneDir.string())));
std::string fetchErr = FetchGit(dep.source, cloneDir); std::string fetchErr = FetchGit(dep.source, cloneDir);
if (!fetchErr.empty()) { if (!fetchErr.empty()) {
@ -283,13 +302,20 @@ ExternalBuildResult Crafter::BuildExternal(const ExternalDependency& dep, std::s
result.linkFlags.push_back(std::format("-l{}", lib)); result.linkFlags.push_back(std::format("-l{}", lib));
} }
if (fs::exists(cmakeBuildDir)) { // The newest library in every directory the link searches: the build
for (const auto& entry : fs::directory_iterator(cmakeBuildDir)) { // directory and the libDirs, where projects like msquic put theirs,
if (!entry.is_regular_file()) continue; // versioned names (libmsquic.so.2) included.
const fs::path& p = entry.path(); auto isLibrary = [](const std::string& name) {
if (p.extension() != ".a" && p.extension() != ".so") continue; return name.ends_with(".a") || name.ends_with(".so") || name.contains(".so.") || name.ends_with(".dll");
};
std::vector<fs::path> searched = { cmakeBuildDir };
for (const fs::path& libDir : dep.libDirs) searched.push_back(libDir.is_absolute() ? libDir : cmakeBuildDir / libDir);
for (const fs::path& dir : searched) {
std::error_code listEc;
for (fs::directory_iterator it(dir, listEc), end; !listEc && it != end; it.increment(listEc)) {
std::error_code ec; std::error_code ec;
fs::file_time_type t = entry.last_write_time(ec); if (!it->is_regular_file(ec) || !isLibrary(it->path().filename().string())) continue;
fs::file_time_type t = it->last_write_time(ec);
if (!ec && t > result.latestArtifact) result.latestArtifact = t; if (!ec && t > result.latestArtifact) result.latestArtifact = t;
} }
} }
@ -365,6 +391,8 @@ Configuration* Crafter::GitProject(const GitProjectSpec& spec) {
fs::path cloneDir = externalRoot / std::format("{}-{}", name, srcHash); fs::path cloneDir = externalRoot / std::format("{}-{}", name, srcHash);
{ {
// One fetch of a clone at a time, as in BuildExternal.
FileLock lock(fs::path(std::format("{}.lock", cloneDir.string())));
const bool exists = fs::exists(cloneDir); const bool exists = fs::exists(cloneDir);
Progress::Task task(std::format("{} {}", exists ? "Updating" : "Cloning", name)); Progress::Task task(std::format("{} {}", exists ? "Updating" : "Cloning", name));
if (std::string err = FetchGit(spec.source, cloneDir); !err.empty()) { if (std::string err = FetchGit(spec.source, cloneDir); !err.empty()) {
@ -406,6 +434,8 @@ fs::path Crafter::GitFetch(const GitSource& source) {
fs::path cloneDir = externalRoot / std::format("{}-{:016x}", name, key); fs::path cloneDir = externalRoot / std::format("{}-{:016x}", name, key);
{ {
// One fetch of a clone at a time, as in BuildExternal.
FileLock lock(fs::path(std::format("{}.lock", cloneDir.string())));
const bool exists = fs::exists(cloneDir); const bool exists = fs::exists(cloneDir);
Progress::Task task(std::format("{} {}", exists ? "Updating" : "Cloning", name)); Progress::Task task(std::format("{} {}", exists ? "Updating" : "Cloning", name));
if (std::string err = FetchGit(source, cloneDir); !err.empty()) { if (std::string err = FetchGit(source, cloneDir); !err.empty()) {

View file

@ -33,27 +33,10 @@ namespace {
// pass through the lock before it reaches the compile steps that *read* those // pass through the lock before it reaches the compile steps that *read* those
// PCMs, so a reader never observes a half-written file. // PCMs, so a reader never observes a half-written file.
// //
// The lock lives in the OS, keyed on a `.lock` file in the cache dir, and is // A FileLock on a `.lock` file in the cache dir.
// released automatically when the descriptor/handle closes — including on struct CacheLock {
// process death — so an interrupted build can never leave a stale lock. The FileLock lock;
// descriptor is close-on-exec / non-inheritable so spawned clang processes explicit CacheLock(const fs::path& cacheDir) : lock(cacheDir / ".lock") {}
// don't keep the lock alive past our own release.
class CacheLock {
std::int32_t fd_ = -1;
public:
explicit CacheLock(const fs::path& cacheDir) {
fd_ = ::open((cacheDir / ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0644);
if (fd_ >= 0) {
while (::flock(fd_, LOCK_EX) != 0 && errno == EINTR) {}
}
}
~CacheLock() {
if (fd_ >= 0) {
::close(fd_); // closing the descriptor drops the flock
}
}
CacheLock(const CacheLock&) = delete;
CacheLock& operator=(const CacheLock&) = delete;
}; };
// The cached Crafter.Build PCMs are keyed by `<target>-<march>` alone, so // The cached Crafter.Build PCMs are keyed by `<target>-<march>` alone, so
@ -120,6 +103,51 @@ namespace {
buffer << in.rdbuf(); buffer << in.rdbuf();
return buffer.str(); return buffer.str();
} }
// The string value of `"key": "..."` in `object`, if it has one.
std::optional<std::string> JsonString(std::string_view object, std::string_view key) {
const std::string quoted = std::format("\"{}\"", key);
std::size_t at = object.find(quoted);
if (at == std::string_view::npos) return std::nullopt;
at = object.find_first_not_of(" \t\r\n", at + quoted.size());
if (at == std::string_view::npos || object[at] != ':') return std::nullopt;
at = object.find_first_not_of(" \t\r\n", at + 1);
if (at == std::string_view::npos || object[at] != '"') return std::nullopt;
std::string value;
for (std::size_t i = at + 1; i < object.size(); ++i) {
if (object[i] == '"') return value;
if (object[i] == '\\' && i + 1 < object.size()) ++i;
value += object[i];
}
return std::nullopt;
}
// The std module's "source-path" in a libstdc++.modules.json: the object
// in its "modules" array whose "logical-name" is "std". The manifest is
// a flat list of such objects, so each one is the text between its
// braces.
std::optional<std::string> StdModuleSourcePath(std::string_view manifest) {
for (std::size_t open = manifest.find('{', manifest.find("\"modules\"")); open != std::string_view::npos; open = manifest.find('{', open + 1)) {
const std::size_t close = manifest.find('}', open);
if (close == std::string_view::npos) break;
const std::string_view object = manifest.substr(open, close - open);
if (JsonString(object, "logical-name") == "std") return JsonString(object, "source-path");
}
return std::nullopt;
}
}
Crafter::FileLock::FileLock(const fs::path& path) {
fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0644);
if (fd >= 0) {
while (::flock(fd, LOCK_EX) != 0 && errno == EINTR) {}
}
}
Crafter::FileLock::~FileLock() {
if (fd >= 0) {
::close(fd); // closing the descriptor drops the flock
}
} }
fs::path Crafter::GetCrafterBuildHome() { fs::path Crafter::GetCrafterBuildHome() {
@ -464,7 +492,7 @@ namespace {
std::mutex depMutex; std::mutex depMutex;
std::unordered_set<Configuration*> forced; std::unordered_set<Configuration*> forced;
std::unordered_set<Configuration*> walked; std::unordered_set<Configuration*> walked;
std::unordered_set<std::string> libs; std::set<std::string> libs;
for (const PragmaDependency& pragma : pragmas) { for (const PragmaDependency& pragma : pragmas) {
Configuration* cfg = nullptr; Configuration* cfg = nullptr;
@ -552,6 +580,35 @@ bool Crafter::IsTopLevelProject() {
return LoadingFrames.size() <= 1; return LoadingFrames.size() <= 1;
} }
namespace {
// The one reading of a wait status (pclose, system) for every helper
// that runs a command: a signal death is reported the way a shell
// reports it, as 128+signal.
void DecodeWaitStatus(std::int32_t status, CommandResult& result) {
if (WIFEXITED(status)) {
result.exitCode = WEXITSTATUS(status);
} else if (WIFSIGNALED(status)) {
result.signal = WTERMSIG(status);
result.exitCode = 128 + result.signal;
result.crashed = true;
} else {
result.exitCode = -1;
}
}
}
std::string Crafter::ShellQuote(std::string_view s) {
std::string out;
out.reserve(s.size() + 2);
out.push_back('\'');
for (char c : s) {
if (c == '\'') out += "'\\''";
else out.push_back(c);
}
out.push_back('\'');
return out;
}
CommandResult Crafter::RunCommandChecked(std::string_view cmd) { CommandResult Crafter::RunCommandChecked(std::string_view cmd) {
std::array<char, 128> buffer; std::array<char, 128> buffer;
CommandResult result{}; CommandResult result{};
@ -564,19 +621,27 @@ CommandResult Crafter::RunCommandChecked(std::string_view cmd) {
result.output += buffer.data(); result.output += buffer.data();
} }
std::int32_t status = pclose(pipe); DecodeWaitStatus(pclose(pipe), result);
if (WIFEXITED(status)) {
result.exitCode = WEXITSTATUS(status);
} else if (WIFSIGNALED(status)) {
result.signal = WTERMSIG(status);
result.exitCode = 128 + result.signal;
result.crashed = true;
} else {
result.exitCode = -1;
}
return result; return result;
} }
std::int32_t Crafter::RunCommandAttached(std::string_view cmd) {
// The child writes straight to the terminal. Anything of ours that is
// still pending — the in-place status line, buffered stdout when it is
// a pipe — has to be out before the child's first byte, or it lands in
// the middle of the child's output or after it.
Progress::Clear();
std::fflush(stdout);
std::fflush(stderr);
const std::string command(cmd);
const std::int32_t status = std::system(command.c_str());
if (status == -1) throw std::runtime_error(std::format("could not start a shell for: {}", cmd));
CommandResult result{};
DecodeWaitStatus(status, result);
return result.exitCode;
}
CommandResult Crafter::RunCommandWithTimeout(std::string_view cmd, std::chrono::seconds timeout) { CommandResult Crafter::RunCommandWithTimeout(std::string_view cmd, std::chrono::seconds timeout) {
std::array<char, 128> buffer; std::array<char, 128> buffer;
CommandResult result{}; CommandResult result{};
@ -592,25 +657,16 @@ CommandResult Crafter::RunCommandWithTimeout(std::string_view cmd, std::chrono::
result.output += buffer.data(); result.output += buffer.data();
} }
std::int32_t status = pclose(pipe); DecodeWaitStatus(pclose(pipe), result);
if (WIFEXITED(status)) { // What comes back is `timeout`'s own exit, and it speaks through the
std::int32_t code = WEXITSTATUS(status); // code: 124 for a command it had to stop, 128+signal for one that died.
if (code == 124) { if (!result.crashed) {
if (result.exitCode == 124) {
result.timedOut = true; result.timedOut = true;
result.exitCode = 124; } else if (result.exitCode >= 128) {
} else if (code >= 128) { result.signal = result.exitCode - 128;
result.signal = code - 128;
result.exitCode = code;
result.crashed = true; result.crashed = true;
} else {
result.exitCode = code;
} }
} else if (WIFSIGNALED(status)) {
result.signal = WTERMSIG(status);
result.exitCode = 128 + result.signal;
result.crashed = true;
} else {
result.exitCode = -1;
} }
return result; return result;
} }
@ -618,23 +674,52 @@ CommandResult Crafter::RunCommandWithTimeout(std::string_view cmd, std::chrono::
std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) {
CacheLock lock(stdPcm.parent_path()); CacheLock lock(stdPcm.parent_path());
if(config.target == "x86_64-w64-mingw32") { if(config.target == "x86_64-w64-mingw32") {
std::vector<std::string> folders; // libstdc++ 15 and newer install libstdc++.modules.json beside the
// library, naming the std module's source relative to itself. It is
// Iterate through the directory and collect all subdirectories // looked for in the library directories the compiler searches for
for (const auto& entry : fs::directory_iterator(GetToolchain().mingwSysroot / "include" / "c++")) { // this target, as it would look for libstdc++ itself, but only in
if (entry.is_directory()) { // those that belong to the mingw toolchain: the list also has the
folders.push_back(entry.path().filename().string()); // generic <prefix>/lib, where the host's own (Linux) libstdc++ keeps
// its manifest. A toolchain that isn't there is a build error, not
// an exception: Build() runs this with its worker threads started.
const Toolchain& tc = GetToolchain();
const std::string where = tc.mingwSysroot.empty()
? std::string("where clang finds it by the x86_64-w64-mingw32 gcc on PATH")
: std::format("under {}", tc.mingwSysroot.string());
const std::string missing = std::format("no mingw-w64 libstdc++ with the std module (libstdc++ 15 or newer) {}. Install mingw-w64, or point CRAFTER_MINGW_DIR at its prefix, the directory holding x86_64-w64-mingw32/ and lib/gcc/", where);
// Asked once a run for each compiler and target: every mingw
// configuration comes through here, up to date or not, and the answer
// costs a clang.
static std::mutex probeMutex;
static std::unordered_map<std::string, CommandResult> probes;
const std::string probe = std::format("{} --target={} -print-search-dirs", GetBaseCommand(config), config.target);
CommandResult dirs;
{
std::lock_guard guard(probeMutex);
auto found = probes.find(probe);
if (found == probes.end()) found = probes.emplace(probe, RunCommandChecked(probe)).first;
dirs = found->second;
}
if (dirs.exitCode != 0) return std::format("{}: {}", missing, dirs.output);
fs::path manifestPath;
constexpr std::string_view librariesLine = "libraries: =";
if (const std::size_t at = dirs.output.find(librariesLine); at != std::string::npos) {
const std::size_t end = dirs.output.find('\n', at);
const std::string list = dirs.output.substr(at + librariesLine.size(), end == std::string::npos ? std::string::npos : end - at - librariesLine.size());
for (const auto part : std::views::split(list, ':')) {
const fs::path dir(std::string_view(part.begin(), part.end()));
const bool mingwOwn = std::ranges::contains(dir, fs::path("x86_64-w64-mingw32"));
if (mingwOwn && fs::exists(dir / "libstdc++.modules.json")) {
manifestPath = dir / "libstdc++.modules.json";
break;
}
} }
} }
if (manifestPath.empty()) return missing;
// Sort the folders by version in descending order const std::optional<std::string> source = StdModuleSourcePath(FileBytes(manifestPath));
std::sort(folders.begin(), folders.end(), [](const std::string& a, const std::string& b) { if (!source) return std::format("{} names no std module", manifestPath.string());
return std::lexicographical_compare(b.begin(), b.end(), a.begin(), a.end()); const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal();
}); if (!fs::exists(stdCc)) return std::format("{}, which {} names as the std module, is missing", stdCc.string(), manifestPath.string());
std::string mingWversion = folders.front();
fs::path stdCc = GetToolchain().mingwSysroot / "include" / "c++" / mingWversion / "bits" / "std.cc";
// -femulated-tls keeps PCM TLS access matching libstdc++'s emutls // -femulated-tls keeps PCM TLS access matching libstdc++'s emutls
// definitions; mismatch surfaces as undefined std::__once_callable. // definitions; mismatch surfaces as undefined std::__once_callable.
@ -663,14 +748,13 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) {
} else if (config.sysroot.empty()) { } else if (config.sysroot.empty()) {
stdCppm = GetToolchain().stdCppm.string(); stdCppm = GetToolchain().stdCppm.string();
} else { } else {
stdCppm = std::format("{}/usr/share/libc++/v1/std.cppm", config.sysroot); // The source has to belong to the libc++ release whose headers
// not every distro ships the std module sources (Alpine's // the sysroot carries. Handing clang the toolchain's own copy
// libc++-dev doesn't); the host copy is interchangeable as long // regardless used to end in a page of "no member named ... in
// as the libc++ versions line up, which the compile of the PCM // namespace 'std'" once the two releases differed.
// against the sysroot's headers verifies anyway StdModuleSource source = SysrootStdModule(GetToolchain(), config.sysroot);
if (!fs::exists(stdCppm) && fs::exists(GetToolchain().stdCppm)) { if (!source.error.empty()) return source.error;
stdCppm = GetToolchain().stdCppm.string(); stdCppm = source.stdCppm.string();
}
} }
std::string sysrootFlag = config.sysroot.empty() std::string sysrootFlag = config.sysroot.empty()
? std::string() ? std::string()
@ -724,9 +808,11 @@ std::string Crafter::GetBaseCommand(const Configuration& config) {
const Toolchain& tc = GetToolchain(); const Toolchain& tc = GetToolchain();
std::string command = tc.clangxx.string(); std::string command = tc.clangxx.string();
if (config.target == "x86_64-w64-mingw32") { if (config.target == "x86_64-w64-mingw32") {
// libstdc++ from the mingw toolchain, found by clang's own mingw // libstdc++ from the mingw toolchain, said outright: a clang built to
// discovery unless the sysroot was given explicitly. // default to libc++ (the Flatpak's) would otherwise reach for that.
if (tc.mingwSysrootExplicit) command += std::format(" --sysroot={}", tc.mingwSysroot.string()); // Found by clang's own mingw discovery unless there is a sysroot.
command += " -stdlib=libstdc++";
if (!tc.mingwSysroot.empty()) command += std::format(" --sysroot={}", tc.mingwSysroot.string());
return command; return command;
} }
command += " -stdlib=libc++"; command += " -stdlib=libc++";
@ -737,7 +823,7 @@ std::string Crafter::GetBaseCommand(const Configuration& config) {
} }
namespace { namespace {
constexpr std::array<std::string_view, 12> CrafterBuildModules = { constexpr std::array<std::string_view, 13> CrafterBuildModules = {
"Crafter.Build-Toolchain", "Crafter.Build-Toolchain",
"Crafter.Build-Shader", "Crafter.Build-Shader",
"Crafter.Build-Platform", "Crafter.Build-Platform",
@ -747,6 +833,7 @@ namespace {
"Crafter.Build-Clang", "Crafter.Build-Clang",
"Crafter.Build-Test", "Crafter.Build-Test",
"Crafter.Build-Lint", "Crafter.Build-Lint",
"Crafter.Build-Script",
"Crafter.Build-Progress", "Crafter.Build-Progress",
"Crafter.Build-Asset", "Crafter.Build-Asset",
"Crafter.Build", "Crafter.Build",
@ -846,7 +933,18 @@ Configuration Crafter::LoadProject(const fs::path& projectFile, std::span<const
// in for the executable: project.so resolves Crafter:: symbols from the // in for the executable: project.so resolves Crafter:: symbols from the
// running binary, so only the declarations (the .cppm files) matter, // running binary, so only the declarations (the .cppm files) matter,
// and an implementation-only rebuild of crafter-build leaves it valid. // and an implementation-only rebuild of crafter-build leaves it valid.
const std::string projectStamp = HashStamp(std::format("{}\0{}\0{}\0{}", CrafterBuildVersion(), GetToolchain().Key(), sourcesStamp, compileCmd)); // The separators are appended, not written into a format string: a
// format string is read as a C string, so one with "\0" in it ends at the
// first NUL and everything after the version silently dropped out of the
// stamp. A project.so compiled against other declarations (a Shader with
// a different layout, say) was then loaded as current and corrupted the
// heap.
std::string projectStampInput(CrafterBuildVersion());
for (std::string_view part : {std::string_view(GetToolchain().Key()), std::string_view(sourcesStamp), std::string_view(compileCmd)}) {
projectStampInput += '\0';
projectStampInput += part;
}
const std::string projectStamp = HashStamp(projectStampInput);
if (ProjectFileStale(soPath, absProject, projectStamp, preDeps)) { if (ProjectFileStale(soPath, absProject, projectStamp, preDeps)) {
std::string result = RunBuildCommand(compileCmd); std::string result = RunBuildCommand(compileCmd);

View file

@ -0,0 +1,104 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
module Crafter.Build:Script_impl;
import std;
import :Script;
import :Clang;
import :Platform;
import :Progress;
namespace fs = std::filesystem;
using namespace Crafter;
namespace {
std::string JoinNames(std::span<const Script> scripts) {
std::string out;
for (const Script& script : scripts) {
if (!out.empty()) out += ", ";
out += script.name;
}
return out;
}
}
std::int32_t ScriptContext::Run(std::string_view command) const {
// Echoed by default, unlike a compile step: a script's commands are the
// only account of what it is doing to the machine.
if (Progress::GetVerbosity() != Progress::Verbosity::Quiet) std::println("$ {}", command);
return RunCommandAttached(command);
}
std::int32_t Crafter::RunScript(Configuration& projectCfg, const RunScriptOptions& opts) {
// Loading the project may have left an in-place status line behind.
Progress::Clear();
const std::span<const Script> scripts = projectCfg.scripts;
if (scripts.empty()) {
std::println(std::cerr,
R"msg(No scripts defined.
Add them in project.cpp before returning the Configuration:
cfg.scripts.push_back({{"docs", [](Crafter::ScriptContext& ctx) {{
return ctx.Run("python3 tools/gen-docs.py");
}}}});
`crafter-build run docs [args...]` then runs it and exits with what it
returned; ctx.args holds the arguments.)msg");
return 1;
}
// Checked for every script on every run, the list included, so a bad
// declaration is reported the first time anyone uses `run` and not only
// when that one name is asked for.
std::unordered_set<std::string_view> seen;
for (const Script& script : scripts) {
if (script.name.empty() || script.name.starts_with('-')) {
std::println(std::cerr, "script name '{}' cannot be run: a name must not be empty or start with '-'", script.name);
return 1;
}
// Running the first or the last of two would both be a guess, and
// the usual cause is a pasted line whose name was never changed.
if (!seen.insert(script.name).second) {
std::println(std::cerr, "script '{}' is defined twice", script.name);
return 1;
}
if (!script.run) {
std::println(std::cerr, "script '{}' has no callback", script.name);
return 1;
}
}
if (opts.listOnly || !opts.named) {
for (const Script& script : scripts) std::println("{}", script.name);
return 0;
}
const auto found = std::ranges::find(scripts, opts.name, &Script::name);
if (found == scripts.end()) {
std::println(std::cerr, "No script named '{}'. Scripts: {}", opts.name, JoinNames(scripts));
return 1;
}
ScriptContext ctx{
.project = projectCfg,
// lexically_normal: fs::absolute keeps a "./" component.
.projectDir = opts.projectFile.empty() ? fs::absolute(projectCfg.path).lexically_normal() : opts.projectFile.parent_path(),
.args = opts.args,
};
try {
const std::int32_t code = found->run(ctx);
if (code >= 0 && code <= 255) return code;
std::println(std::cerr, "script '{}' returned {}, which is not an exit status (0..255); exiting with 1", found->name, code);
return 1;
} catch (const std::exception& e) {
// Never let a script's exception unwind across the project.so
// boundary and out of Run: report it as this script's failure.
Progress::Clear();
std::println(std::cerr, "script '{}' failed: {}", found->name, e.what());
return 1;
} catch (...) {
Progress::Clear();
std::println(std::cerr, "script '{}' failed: it threw something that isn't a std::exception", found->name);
return 1;
}
}

View file

@ -8,6 +8,7 @@ module;
#include "../lib/DirStackFileIncluder.h" #include "../lib/DirStackFileIncluder.h"
module Crafter.Build:Shader_impl; module Crafter.Build:Shader_impl;
import :Shader; import :Shader;
import :Platform;
import std; import std;
namespace fs = std::filesystem; namespace fs = std::filesystem;
@ -31,17 +32,72 @@ namespace {
} }
return EShLangVertex; return EShLangVertex;
} }
// A path as a Make prerequisite, escaped the way NewestPrerequisite reads it.
std::string MakeEscaped(const std::string& path) {
std::string escaped;
for (const char c : path) {
if (c == ' ' || c == '\t' || c == '#') escaped += '\\';
if (c == '$') escaped += '$';
escaped += c;
}
return escaped;
}
std::string ReadFile(const fs::path& file) {
std::ifstream stream(file, std::ios::binary);
return std::string(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
}
// What a compile's output depends on besides files' contents, one line
// each: two entries of one source differ here, and an #include searched
// for from another source directory or in other directories can find
// another file.
std::string Settings(const Crafter::Shader& shader, std::span<const fs::path> includeDirs) {
std::string settings = std::format("source {}\nstage {}\nentry {}\n", fs::absolute(shader.path).string(), static_cast<int>(shader.type), shader.entrypoint);
for (const std::string& define : shader.defines) settings += std::format("define {}\n", define);
for (const fs::path& dir : includeDirs) settings += std::format("include {}\n", dir.string());
return settings;
}
// Where Compile records, for Check, what the output in the bin dir was
// built from: beside the configuration's objects, as their depfiles are.
fs::path Record(const Crafter::Shader& shader, const fs::path& buildDir, std::string_view kind) {
return buildDir / std::format("{}.{}", shader.OutputName().string(), kind);
}
} }
namespace Crafter { namespace Crafter {
Shader::Shader(fs::path&& path, std::string&& entrypoint, ShaderType type) : path(std::move(path)), entrypoint(std::move(entrypoint)), type(type) { Shader::Shader(fs::path&& path, std::string&& entrypoint, ShaderType type) : path(std::move(path)), entrypoint(std::move(entrypoint)), type(type) {
} }
bool Shader::Check(const fs::path& outputDir) const { fs::path Shader::OutputName() const {
fs::path spv = outputDir / path.filename().replace_extension("spv"); // replace_extension swaps only the last extension, so a stage suffix
return fs::exists(spv) && fs::last_write_time(path) < fs::last_write_time(spv); // survives: "ui.comp.glsl" becomes "ui.comp.spv", or "ui.comp.sets.spv"
// for the variant "sets".
fs::path name = path.filename();
name.replace_extension(variant.empty() ? std::string("spv") : std::format("{}.spv", variant));
return name;
} }
std::string Shader::Compile(const fs::path& outputDir, std::span<const fs::path> includeDirs) const { bool Shader::Check(const fs::path& outputDir, const fs::path& buildDir, std::span<const fs::path> includeDirs) const {
std::error_code ec;
const fs::file_time_type built = fs::last_write_time(outputDir / OutputName(), ec);
if (ec) return false;
// The source and every file it #included, as the last compile
// recorded them: without a record, or with an included file since
// deleted, NewestPrerequisite reads max(), which is stale.
if (std::max(fs::last_write_time(path), NewestPrerequisite(Record(*this, buildDir, "d"))) >= built) return false;
return ReadFile(Record(*this, buildDir, "settings")) == Settings(*this, includeDirs);
}
std::string Shader::Compile(const fs::path& outputDir, const fs::path& buildDir, std::span<const fs::path> includeDirs) const {
// The records go first, so a compile that fails leaves the output
// stale, whatever it wrote.
const fs::path depFile = Record(*this, buildDir, "d");
const fs::path settingsFile = Record(*this, buildDir, "settings");
std::error_code removed;
fs::remove(depFile, removed);
fs::remove(settingsFile, removed);
EShLanguage glslangType = ToEShLanguage(type); EShLanguage glslangType = ToEShLanguage(type);
glslang::InitializeProcess(); glslang::InitializeProcess();
@ -52,6 +108,9 @@ namespace Crafter {
auto fail = [&](std::string_view stage, std::string log) { auto fail = [&](std::string_view stage, std::string log) {
glslang::FinalizeProcess(); glslang::FinalizeProcess();
std::string out = path.string(); std::string out = path.string();
// Several variants of one source compile in parallel; without the
// variant the message would not say which of them failed.
if (!variant.empty()) out += std::format(" ({})", variant);
out += ": "; out += ": ";
out += stage; out += stage;
if (!log.empty()) { if (!log.empty()) {
@ -75,8 +134,34 @@ namespace Crafter {
const char* shaderSource = src.data(); const char* shaderSource = src.data();
const std::int32_t shaderSourceLen = static_cast<std::int32_t>(src.size()); const std::int32_t shaderSourceLen = static_cast<std::int32_t>(src.size());
// A variant is one component of a file name. Caught here, because a
// separator would otherwise surface as "failed to open SPIR-V output"
// after the whole compile.
if (variant.find_first_of("/\\") != std::string::npos) {
return fail(std::format("shader variant '{}' must not contain a path separator", variant), {});
}
// One #define line per entry, split like the compiler's -D: "NAME"
// defines it to 1, "NAME=VALUE" to VALUE. glslang places the preamble
// ahead of the whole source (it finds #version by scanning first), so
// line numbers in the user's file are unaffected. It keeps the
// pointer it is handed instead of copying, so the string is declared
// here to outlive parse() below.
std::string preamble;
for (const std::string& define : defines) {
const std::size_t eq = define.find('=');
const std::string name = define.substr(0, eq);
// An error inside the preamble is reported as "-1:<line>" with no
// file and no macro name, so reject the malformed ones by name.
if (name.empty() || define.find_first_of("\r\n") != std::string::npos) {
return fail(std::format("invalid shader define '{}': expected NAME or NAME=VALUE on one line", define), {});
}
preamble += std::format("#define {} {}\n", name, eq == std::string::npos ? std::string("1") : define.substr(eq + 1));
}
glslang::TShader shader(glslangType); glslang::TShader shader(glslangType);
shader.setStringsWithLengthsAndNames(&shaderSource, &shaderSourceLen, fileNameList, 1); shader.setStringsWithLengthsAndNames(&shaderSource, &shaderSourceLen, fileNameList, 1);
if (!preamble.empty()) shader.setPreamble(preamble.c_str());
shader.setEntryPoint(entrypoint.c_str()); shader.setEntryPoint(entrypoint.c_str());
shader.setSourceEntryPoint(entrypoint.c_str()); shader.setSourceEntryPoint(entrypoint.c_str());
shader.setEnvTarget(glslang::EShTargetSpv, glslang::EShTargetSpv_1_4); shader.setEnvTarget(glslang::EShTargetSpv, glslang::EShTargetSpv_1_4);
@ -110,7 +195,7 @@ namespace Crafter {
glslang::GlslangToSpv(*intermediate, spirv, &logger); glslang::GlslangToSpv(*intermediate, spirv, &logger);
std::string spvLog = logger.getAllMessages(); std::string spvLog = logger.getAllMessages();
fs::path filename = path.filename().replace_extension("spv"); fs::path filename = OutputName();
std::ofstream file(outputDir/filename, std::ios::binary); std::ofstream file(outputDir/filename, std::ios::binary);
if (!file) { if (!file) {
return fail("failed to open SPIR-V output", (outputDir/filename).string()); return fail("failed to open SPIR-V output", (outputDir/filename).string());
@ -120,6 +205,25 @@ namespace Crafter {
if (!spvLog.empty()) { if (!spvLog.empty()) {
return fail("SPIR-V codegen reported issues", std::move(spvLog)); return fail("SPIR-V codegen reported issues", std::move(spvLog));
} }
// What Check compares the output against next time: the source and
// the files it included, as absolute paths so that reading them back
// doesn't depend on the directory a build runs from, and the settings.
std::ofstream dep(depFile, std::ios::binary);
dep << MakeEscaped(fs::absolute(outputDir / filename).string()) << ": " << MakeEscaped(fs::absolute(path).string());
for (const std::string& included : includeDir.getIncludedFiles()) {
dep << " \\\n " << MakeEscaped(fs::absolute(included).string());
}
dep << '\n';
dep.close();
std::ofstream settings(settingsFile, std::ios::binary);
settings << Settings(*this, includeDirs);
settings.close();
if (!dep || !settings) {
fs::remove(depFile, removed);
fs::remove(settingsFile, removed);
return fail("failed to record what the shader was built from", buildDir.string());
}
glslang::FinalizeProcess(); glslang::FinalizeProcess();
return {}; return {};
} }

View file

@ -23,23 +23,11 @@ namespace {
// MatchGlob/MatchAny live in :Platform — shared with the lint verb. // MatchGlob/MatchAny live in :Platform — shared with the lint verb.
std::string ShellQuoteSh(std::string_view s) {
std::string out;
out.reserve(s.size() + 2);
out.push_back('\'');
for (char c : s) {
if (c == '\'') out += "'\\''";
else out.push_back(c);
}
out.push_back('\'');
return out;
}
std::string JoinAndQuoteArgs(std::span<const std::string> args) { std::string JoinAndQuoteArgs(std::span<const std::string> args) {
std::string out; std::string out;
for (const auto& a : args) { for (const auto& a : args) {
if (!out.empty()) out.push_back(' '); if (!out.empty()) out.push_back(' ');
out += ShellQuoteSh(a); out += ShellQuote(a);
} }
return out; return out;
} }
@ -273,7 +261,7 @@ TestResult Crafter::RunSingleTest(const Test& test, const fs::path& binary, std:
if (test.runner.exec.empty()) { if (test.runner.exec.empty()) {
// Local runner: spawn the binary directly through the host shell. // Local runner: spawn the binary directly through the host shell.
std::string cmd = std::format("{} {}", ShellQuoteSh(binary.string()), ph["{args}"]); std::string cmd = std::format("{} {}", ShellQuote(binary.string()), ph["{args}"]);
r = RunCommandWithTimeout(cmd, timeout); r = RunCommandWithTimeout(cmd, timeout);
} else { } else {
// Prefix runner (qemu-user, wasmtime, wine, ...): templated exec // Prefix runner (qemu-user, wasmtime, wine, ...): templated exec

View file

@ -112,15 +112,19 @@ Toolchain Crafter::ResolveToolchain(const ToolchainInputs& in) {
// 3. libc++ and its std module source. The compiler's own copy lives at // 3. libc++ and its std module source. The compiler's own copy lives at
// <prefix>/share/libc++/v1 when the prefix carries libc++; the system // <prefix>/share/libc++/v1 when the prefix carries libc++; the system
// copy is the last resort either way. // copy is the last resort either way. The headers of the same install
const fs::path systemStdCppm = "/usr/share/libc++/v1/std.cppm"; // say which release the source belongs to.
fs::path libcxxRoot = "/usr";
if (std::string_view dir = Env(in, "CRAFTER_BUILD_LIBCXX_DIR"); !dir.empty()) { if (std::string_view dir = Env(in, "CRAFTER_BUILD_LIBCXX_DIR"); !dir.empty()) {
tc.libcxxDir = fs::path(dir); tc.libcxxDir = fs::path(dir);
tc.stdCppm = tc.libcxxDir / "share" / "libc++" / "v1" / "std.cppm"; libcxxRoot = tc.libcxxDir;
} else if (!tc.prefix.empty() && fs::exists(tc.prefix / "share" / "libc++" / "v1" / "std.cppm")) { } else if (!tc.prefix.empty() && fs::exists(tc.prefix / "share" / "libc++" / "v1" / "std.cppm")) {
tc.stdCppm = tc.prefix / "share" / "libc++" / "v1" / "std.cppm"; libcxxRoot = tc.prefix;
} else { }
tc.stdCppm = systemStdCppm; tc.stdCppm = libcxxRoot / "share" / "libc++" / "v1" / "std.cppm";
tc.libcxxVersion = LibcxxVersion(libcxxRoot / "include" / "c++" / "v1");
if (!in.crafterPrefix.empty()) {
tc.stdModulesDir = in.crafterPrefix / "share" / "crafter-build" / "libc++-modules";
} }
// 4. Sysroots for the cross targets that have a conventional location. // 4. Sysroots for the cross targets that have a conventional location.
@ -133,9 +137,10 @@ Toolchain Crafter::ResolveToolchain(const ToolchainInputs& in) {
} }
if (std::string_view dir = Env(in, "CRAFTER_MINGW_DIR"); !dir.empty()) { if (std::string_view dir = Env(in, "CRAFTER_MINGW_DIR"); !dir.empty()) {
tc.mingwSysroot = fs::path(dir); tc.mingwSysroot = fs::path(dir);
tc.mingwSysrootExplicit = true; } else if (!in.hostRoot.empty() && fs::exists(in.hostRoot / "usr" / "x86_64-w64-mingw32")) {
} else { // The sandbox has no mingw-w64 of its own, and no mingw gcc on PATH
tc.mingwSysroot = "/usr/x86_64-w64-mingw32"; // for clang to find one by.
tc.mingwSysroot = in.hostRoot / "usr";
} }
return tc; return tc;
} }
@ -153,6 +158,7 @@ const Toolchain& Crafter::GetToolchain() {
try { try {
in.crafterPrefix = GetCrafterBuildHome().parent_path().parent_path(); in.crafterPrefix = GetCrafterBuildHome().parent_path().parent_path();
} catch (const std::exception&) {} } catch (const std::exception&) {}
if (fs::exists("/.flatpak-info")) in.hostRoot = "/run/host";
Toolchain tc = ResolveToolchain(in); Toolchain tc = ResolveToolchain(in);
if (!tc.error.empty()) return tc; if (!tc.error.empty()) return tc;
@ -180,6 +186,56 @@ const Toolchain& Crafter::RequireToolchain() {
return tc; return tc;
} }
std::int32_t Crafter::LibcxxVersion(const fs::path& includeDir) {
// "# define _LIBCPP_VERSION 230102", the one place libc++ states its
// release in a form the preprocessor does not have to be run for.
constexpr std::string_view Marker = "define _LIBCPP_VERSION ";
std::ifstream config(includeDir / "__config");
std::string line;
while (std::getline(config, line)) {
const std::size_t at = line.find(Marker);
if (at == std::string::npos) continue;
const std::string_view digits = std::string_view(line).substr(at + Marker.size());
std::int32_t version = 0;
if (std::from_chars(digits.data(), digits.data() + digits.size(), version).ec == std::errc{}) return version;
}
return 0;
}
std::string Crafter::LibcxxVersionString(std::int32_t version) {
return std::format("{}.{}.{}", version / 10000, version / 100 % 100, version % 100);
}
StdModuleSource Crafter::SysrootStdModule(const Toolchain& tc, const fs::path& sysroot) {
const fs::path own = sysroot / "usr" / "share" / "libc++" / "v1" / "std.cppm";
if (fs::exists(own)) return {own, {}};
const fs::path includeDir = sysroot / "usr" / "include" / "c++" / "v1";
if (!fs::exists(includeDir / "__config")) {
return {{}, std::format("sysroot {} has no libc++ headers at {}; install its libc++ development package", sysroot.string(), includeDir.string())};
}
const std::int32_t wanted = LibcxxVersion(includeDir);
const bool comparable = wanted != 0 && tc.libcxxVersion != 0;
// Releases are matched on the major version: a release branch only takes
// fixes, so 22.1.3 headers and the 22.1.8 module source name the same
// declarations.
if (comparable && wanted / 10000 == tc.libcxxVersion / 10000 && fs::exists(tc.stdCppm)) return {tc.stdCppm, {}};
if (wanted != 0 && !tc.stdModulesDir.empty()) {
const fs::path shipped = tc.stdModulesDir / std::to_string(wanted / 10000) / "std.cppm";
if (fs::exists(shipped)) return {shipped, {}};
}
// One of the two releases is unknown, so there is nothing to compare:
// hand over the toolchain's source and let the compile decide, which is
// what happened for every sysroot before releases were compared.
if (!comparable && fs::exists(tc.stdCppm)) return {tc.stdCppm, {}};
const std::string sysrootRelease = wanted != 0 ? LibcxxVersionString(wanted) : std::string("of an unknown release");
const std::string toolchainSource = !fs::exists(tc.stdCppm)
? std::string("the toolchain has no std module source at all")
: std::format("the toolchain's is for libc++ {}", LibcxxVersionString(tc.libcxxVersion));
return {{}, std::format("sysroot {} carries libc++ {} without its std module source, and crafter-build has none for that release ({}); install the share/libc++/v1 directory of that libc++ release (std.cppm and std/) as {}", sysroot.string(), sysrootRelease, toolchainSource, own.parent_path().string())};
}
std::string_view Crafter::CrafterBuildVersion() { std::string_view Crafter::CrafterBuildVersion() {
return CRAFTER_BUILD_VERSION; return CRAFTER_BUILD_VERSION;
} }

View file

@ -7,18 +7,22 @@ import :Shader;
import :Interface; import :Interface;
import :Implementation; import :Implementation;
import :External; import :External;
import :Toolchain;
namespace fs = std::filesystem; namespace fs = std::filesystem;
export namespace Crafter { export namespace Crafter {
struct BuildResult { struct BuildResult {
std::string result; std::string result;
bool repack; bool repack;
std::unordered_set<std::string> libs; // Sets in order: the commands they go into are compared with the ones
// recorded last time, so they come out the same however the
// dependency threads that fill them happen to finish.
std::set<std::string> libs;
// Compile flags (typically -I include paths) the dep wants its // Compile flags (typically -I include paths) the dep wants its
// consumers to see — sourced from its external dependencies' include // consumers to see — sourced from its external dependencies' include
// dirs so headers a dep exposes in its public modules are reachable // dirs so headers a dep exposes in its public modules are reachable
// when a consumer #includes them directly. Propagates transitively. // when a consumer #includes them directly. Propagates transitively.
std::unordered_set<std::string> publicCompileFlags; std::set<std::string> publicCompileFlags;
}; };
struct Define { struct Define {
@ -355,6 +359,45 @@ export namespace Crafter {
bool needsAst = false; bool needsAst = false;
}; };
struct Configuration;
// What a Script's callback is handed when `crafter-build run <name>`
// starts it.
struct ScriptContext {
// The Configuration project.cpp returned. Read the project through
// this rather than capturing the local in CrafterBuildProject by
// reference: that local is gone by the time a script runs.
Configuration& project;
// Absolute directory of the project file. Commands run in the
// directory crafter-build was started from, like everything else it
// does, so a script names its files through this.
fs::path projectDir;
// Everything the user typed after the script's name, untouched:
// crafter-build interprets none of it.
std::span<const std::string> args;
// Run `command` through sh with the terminal attached — output
// appears as it is produced, the command can prompt, Ctrl-C reaches
// it — after echoing it as `$ command` (not under -q). Returns its
// exit code, 128+signal when it was killed; return that from the
// script to make it the exit code of crafter-build. Quote anything
// that is not a literal with ShellQuote. Out-of-line like
// LintContext's members: the callback runs from project.so.
std::int32_t Run(std::string_view command) const;
};
// A project script: a chore that belongs to the project but is not part
// of building it (packaging, installing, publishing, serving), run by
// name with `crafter-build run <name> [args...]`. `run` returns the
// process exit code. It is only stored when project.cpp is loaded, which
// happens for every verb and for every project that depends on this one,
// so whatever a script needs to probe it probes when it runs. Only the
// scripts of the project crafter-build was pointed at can be run, never
// a dependency's.
struct Script {
std::string name;
std::function<std::int32_t(ScriptContext&)> run;
};
// The host target triple: what the resolved toolchain's clang++ prints // The host target triple: what the resolved toolchain's clang++ prints
// for -print-target-triple (see GetToolchain). Used as the default for // for -print-target-triple (see GetToolchain). Used as the default for
// Configuration::target so projects don't have to hardcode it for the // Configuration::target so projects don't have to hardcode it for the
@ -445,6 +488,12 @@ export namespace Crafter {
std::vector<Test> tests; std::vector<Test> tests;
// Lint rules for `crafter-build lint`. Populate via AddLintRule. // Lint rules for `crafter-build lint`. Populate via AddLintRule.
std::vector<LintRule> lintRules; std::vector<LintRule> lintRules;
// Scripts for `crafter-build run <name>`, a plain list like defines:
// cfg.scripts.push_back({"docs", [](ScriptContext& ctx) {
// return ctx.Run("python3 tools/gen-docs.py");
// }});
// Not part of VariantId: a script changes nothing that is compiled.
std::vector<Script> scripts;
void GetInterfacesAndImplementations(std::span<fs::path> interfaces, std::span<fs::path> implementations); void GetInterfacesAndImplementations(std::span<fs::path> interfaces, std::span<fs::path> implementations);
// Retry the `import X;` names GetInterfacesAndImplementations could not // Retry the `import X;` names GetInterfacesAndImplementations could not
// place, against the dependency DAG as it stands now. Sources are // place, against the dependency DAG as it stands now. Sources are
@ -488,10 +537,21 @@ export namespace Crafter {
void AddAstLintRule(std::string name, std::function<void(LintContext&)> check); void AddAstLintRule(std::string name, std::function<void(LintContext&)> check);
// Suffix that uniquely identifies this Configuration's compile state. // Suffix that uniquely identifies this Configuration's compile state.
// target+march+mtune are spelled out for readability; the rest // target+march+mtune are spelled out for readability; the rest
// (type, debug, sysroot, defines, compileFlags) collapse into a short // (type, debug, sysroot, defines, compileFlags, the toolchain)
// hash so two Configurations sharing a path but with different // collapse into a short hash so two Configurations sharing a path but
// compile state can't clobber each other's outputs. // with different compile state can't clobber each other's outputs.
std::string VariantId() const { std::string VariantId() const { return VariantId(GetToolchain()); }
// The variant under an explicit toolchain; VariantId() is this for
// the running one. The compiler is compile state like any flag: a
// PCM can only be read by the clang release that wrote it, and
// staleness is decided by mtimes, which a compiler update does not
// touch. Before the toolchain was hashed, a project built with clang
// 22 kept its PCMs when crafter-build moved to clang 23, and the
// first edit afterwards failed with "module file ... uses an older
// format that is no longer supported" until the project was cleaned.
// Keyed here, an update starts an empty directory and rebuilds; the
// old one stays until `crafter-build clean`.
std::string VariantId(const Toolchain& toolchain) const {
std::string compileKey; std::string compileKey;
compileKey += std::to_string(static_cast<std::int32_t>(type)); compileKey += std::to_string(static_cast<std::int32_t>(type));
compileKey += '|'; compileKey += '|';
@ -521,6 +581,8 @@ export namespace Crafter {
compileKey += "|A:"; compileKey += "|A:";
compileKey += a; compileKey += a;
} }
compileKey += "|T:";
compileKey += toolchain.Key();
std::size_t configHash = std::hash<std::string>{}(compileKey); std::size_t configHash = std::hash<std::string>{}(compileKey);
return std::format("{}-{}-{}-{}-{:08x}", name, target, march, mtune, configHash); return std::format("{}-{}-{}-{}-{:08x}", name, target, march, mtune, configHash);
} }

View file

@ -14,6 +14,21 @@ namespace Crafter {
bool timedOut = false; bool timedOut = false;
std::int32_t signal = 0; std::int32_t signal = 0;
}; };
// An exclusive lock on the file `path` (created if missing), held from
// construction to destruction. An flock, so it keeps out the other
// threads of this process (each opens the file anew) and other
// crafter-build processes alike. The OS drops it when the descriptor
// closes, process death included, so an interrupted build never leaves a
// stale lock; the descriptor is close-on-exec, so the tools a holder
// spawns don't keep it past its release.
class FileLock {
std::int32_t fd = -1;
public:
explicit FileLock(const fs::path& path);
~FileLock();
FileLock(const FileLock&) = delete;
FileLock& operator=(const FileLock&) = delete;
};
std::string BuildStdPcm(const Configuration& config, fs::path stdPcm); std::string BuildStdPcm(const Configuration& config, fs::path stdPcm);
fs::path GetCacheDir(); fs::path GetCacheDir();
CommandResult RunCommandChecked(std::string_view command); CommandResult RunCommandChecked(std::string_view command);
@ -29,6 +44,18 @@ namespace Crafter {
// therefore keep the `if (!result.empty())` shape they already had. // therefore keep the `if (!result.empty())` shape they already had.
std::string RunBuildCommand(std::string_view command); std::string RunBuildCommand(std::string_view command);
export CommandResult RunCommandWithTimeout(std::string_view command, std::chrono::seconds timeout); export CommandResult RunCommandWithTimeout(std::string_view command, std::chrono::seconds timeout);
// Runs `command` through sh with this process's own stdin, stdout and
// stderr, so its output appears as it is produced and it can prompt.
// Every helper above captures instead, which suits a compile step and
// not a command that runs for half an hour. Returns the exit code, or
// 128+signal when the command was killed; a nonzero exit is the caller's
// to judge and never throws. While it runs, Ctrl-C goes to the command
// and not to crafter-build, which then sees it as exit code 130.
// Project scripts reach this through ScriptContext::Run.
std::int32_t RunCommandAttached(std::string_view command);
// `s` as one single-quoted sh word, safe to splice into a command line
// whatever it contains: paths with spaces, arguments a user typed.
export std::string ShellQuote(std::string_view s);
std::string GetBaseCommand(const Configuration& config); std::string GetBaseCommand(const Configuration& config);
export Configuration LoadProject(const fs::path& projectFile, std::span<const std::string_view> args); export Configuration LoadProject(const fs::path& projectFile, std::span<const std::string_view> args);
// A dependency the project file currently running its CrafterBuildProject // A dependency the project file currently running its CrafterBuildProject

View file

@ -0,0 +1,40 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
export module Crafter.Build:Script;
import std;
import :Clang;
export namespace Crafter {
struct RunScriptOptions {
// The script to run; read only when `named` is set.
std::string name;
// A name was given. Kept apart from `name` because an empty string
// is a name someone can type (`run ""`): it has to be looked up and
// refused, not read as a bare `run`, which lists.
bool named = false;
// What followed the name on the command line, handed to the script
// as ScriptContext::args.
std::vector<std::string> args;
// Print the script names instead of running one.
bool listOnly = false;
// Absolute path of the loaded project.cpp; its directory becomes
// ScriptContext::projectDir. Empty => fall back to cfg.path.
std::filesystem::path projectFile;
};
// Backs `crafter-build run`. Runs the script `opts` names out of
// `projectCfg.scripts` and returns what it returned, which the CLI uses
// as the process exit code. Without a name, or with listOnly, prints the
// names and returns 0.
//
// Returns 1, with the reason on stderr and without running anything,
// when the project defines no scripts, when a script is declared under a
// name that could never be typed (empty, or starting with '-') or under
// one name twice, and when no script has the requested name. A script
// that throws also yields 1, as does one that returns a nonzero value
// outside 1..255: an exit status is eight bits, and a value such as 256
// (a raw wait status passed along by mistake) would otherwise wrap to
// success.
std::int32_t RunScript(Configuration& projectCfg, const RunScriptOptions& opts);
}

View file

@ -28,8 +28,33 @@ namespace Crafter {
fs::path path; fs::path path;
std::string entrypoint; std::string entrypoint;
ShaderType type; ShaderType type;
// Preprocessor macros for this compile, spelled like the compiler's -D:
// "NAME" (defined to 1) or "NAME=VALUE". They are placed ahead of the
// source, so one file can be registered more than once and take a
// different #ifdef branch each time. Note the spelling differs from
// Configuration::defines, which is a list of {name, value} pairs.
std::vector<std::string> defines;
// Names the output when one source is compiled more than once: empty
// gives "<name>.spv", "sets" gives "<name>.sets.spv". One file-name
// component: no path separators. Every entry of a configuration must
// end up with its own OutputName(); Build() fails if two collide, so
// each extra compile of a source needs its own variant.
std::string variant;
Shader(fs::path&& path, std::string&& entrypoint, ShaderType type); Shader(fs::path&& path, std::string&& entrypoint, ShaderType type);
bool Check(const fs::path& outputDir) const; // File name of the compiled SPIR-V, relative to the output directory.
std::string Compile(const fs::path& outputDir, std::span<const fs::path> includeDirs) const; // The single source of truth for it: Check, Compile and the artifact
// forwarding in Build() all go through here.
fs::path OutputName() const;
// Whether the output in `outputDir` is up to date: newer than the
// source and than every file the source #included, and compiled with
// this entry's stage, entry point and defines, searching
// `includeDirs`. Compile records the last two in `buildDir`; without
// its record an output is stale.
bool Check(const fs::path& outputDir, const fs::path& buildDir, std::span<const fs::path> includeDirs) const;
// Compiles into `outputDir`, searching `includeDirs` for #include, and
// records what Check needs in `buildDir`: a depfile naming the source
// and every file it included, and the settings it was compiled with.
// Returns an error message, empty on success.
std::string Compile(const fs::path& outputDir, const fs::path& buildDir, std::span<const fs::path> includeDirs) const;
}; };
} }

View file

@ -28,13 +28,22 @@ export namespace Crafter {
fs::path libcxxDir; fs::path libcxxDir;
// libc++'s std module source for host builds. // libc++'s std module source for host builds.
fs::path stdCppm; fs::path stdCppm;
// The _LIBCPP_VERSION of the libc++ stdCppm belongs to (220108 for
// 22.1.8); 0 when its <__config> could not be read.
std::int32_t libcxxVersion = 0;
// Std module sources for libc++ releases other than the toolchain's
// own, one <major>/std.cppm each, for sysroots that carry libc++
// without them: <crafter prefix>/share/crafter-build/libc++-modules.
// Empty when the crafter prefix is unknown; need not exist.
fs::path stdModulesDir;
fs::path wasiSysroot; fs::path wasiSysroot;
// Holds include/c++/<ver>/bits/std.cc for the mingw target. Passed to // The mingw-w64 install prefix, the directory that holds
// clang as --sysroot only when it was set explicitly: clang's own // x86_64-w64-mingw32/ and lib/gcc/x86_64-w64-mingw32/ (/usr for a
// mingw discovery handles the distro layout, and a --sysroot pointing // distro package), passed to clang as --sysroot for the mingw target.
// at the triple directory would lose crtbegin.o and libgcc. // Empty means clang's own discovery, which follows the mingw gcc on
// PATH. Set by CRAFTER_MINGW_DIR, else the host's /usr when running
// sandboxed and it has mingw-w64 (the sandbox's own /usr never does).
fs::path mingwSysroot; fs::path mingwSysroot;
bool mingwSysrootExplicit = false;
// `clang++ -print-target-triple` and the first line of // `clang++ -print-target-triple` and the first line of
// `clang++ --version`; both empty when clang++ could not be run. // `clang++ --version`; both empty when clang++ could not be run.
std::string hostTriple; std::string hostTriple;
@ -64,6 +73,9 @@ export namespace Crafter {
std::map<std::string, std::string, std::less<>> env; std::map<std::string, std::string, std::less<>> env;
// The <prefix> of <prefix>/share/crafter-build; empty when unknown. // The <prefix> of <prefix>/share/crafter-build; empty when unknown.
fs::path crafterPrefix; fs::path crafterPrefix;
// Where the host's root shows inside a sandbox (/run/host in a
// Flatpak); empty outside one.
fs::path hostRoot;
}; };
// Pure path resolution: an env override wins, then a toolchain installed // Pure path resolution: an env override wins, then a toolchain installed
// beside crafter-build (<crafterPrefix>/bin/clang++), then bare names on // beside crafter-build (<crafterPrefix>/bin/clang++), then bare names on
@ -74,6 +86,29 @@ export namespace Crafter {
const Toolchain& GetToolchain(); const Toolchain& GetToolchain();
// GetToolchain(), or a std::runtime_error carrying its error. // GetToolchain(), or a std::runtime_error carrying its error.
const Toolchain& RequireToolchain(); const Toolchain& RequireToolchain();
// _LIBCPP_VERSION as the <__config> in `includeDir` (an include/c++/v1)
// defines it, XXYYZZ; 0 when the file is missing or has no such line.
std::int32_t LibcxxVersion(const fs::path& includeDir);
// "23.1.2" for 230102.
std::string LibcxxVersionString(std::int32_t version);
struct StdModuleSource {
fs::path stdCppm;
// One line saying why no source fits; empty when stdCppm is set.
std::string error;
};
// The std module source to precompile for a native (non-wasm) build
// against `sysroot`. The module source is a list of using-declarations
// for names the headers of the same libc++ release declare, so one from
// another release fails on every name that release added or dropped. In
// order: the sysroot's own <sysroot>/usr/share/libc++/v1/std.cppm, the
// toolchain's when both are the same major release, the copy for the
// sysroot's release under toolchain.stdModulesDir. Not every distro
// packages the module sources with its libc++ (Alpine does not), which
// is why the sysroot's own copy cannot be the only answer. When the
// sysroot's release cannot be read, the toolchain's source is returned
// and the compile is the check.
StdModuleSource SysrootStdModule(const Toolchain& toolchain, const fs::path& sysroot);
// The version baked in when crafter-build was built; "0.0.0-dev" for a // The version baked in when crafter-build was built; "0.0.0-dev" for a
// build that was given none. // build that was given none.
std::string_view CrafterBuildVersion(); std::string_view CrafterBuildVersion();

View file

@ -10,5 +10,6 @@ export import :Shader;
export import :External; export import :External;
export import :Test; export import :Test;
export import :Lint; export import :Lint;
export import :Script;
export import :Progress; export import :Progress;
export import :Asset; export import :Asset;

25361
lib/json.hpp

File diff suppressed because it is too large Load diff

View file

@ -25,7 +25,19 @@
</provides> </provides>
<content_rating type="oars-1.1"/> <content_rating type="oars-1.1"/>
<releases> <releases>
<release version="2.0.0" date="2026-10-02"> <release version="2.2.0" date="2026-10-16">
<description>
<p>A Shader entry can carry defines, spelled like the compiler's -D (NAME is defined to 1, NAME=VALUE to VALUE), and a variant that names its output, so one source compiles into several outputs: ui.comp.glsl with variant sets becomes ui.comp.sets.spv. A configuration in which two entries share an output name is refused with both named, as are a malformed define and a variant with a path separator.</p>
<p>Compiles with a sysroot now use the sysroot's own libc++ headers instead of the bundled toolchain's, which fixes cross builds against a musl sysroot that include a standard header textually and stops glibc sysroot builds from silently compiling against the wrong libc++ release. The project.so stamp hashes every part of the host build, so a changed declaration recompiles it. The vendored json.hpp is gone.</p>
</description>
</release>
<release version="2.1.0" date="2026-10-03">
<description>
<p>The bundled toolchain is now LLVM 23.1.2. Build outputs are kept per compiler, so the first build after updating recompiles each project into a fresh directory and nothing has to be cleaned by hand.</p>
<p>Cross builds pick the std module source that matches the sysroot's libc++ release. The source for libc++ 22 ships beside the bundled 23 for sysroots that carry libc++ without one, such as Alpine, and a sysroot nothing matches fails with one line naming both releases. crafter-build --version prints the libc++ release in use.</p>
</description>
</release>
<release version="2.0.0" date="2026-10-03">
<description> <description>
<p>First release distributed as a Flatpak, with a bundled LLVM 22 toolchain and WASI sysroot. Windows host support is removed; crafter-build gains --version and resolves its compiler from one place.</p> <p>First release distributed as a Flatpak, with a bundled LLVM 22 toolchain and WASI sysroot. Windows host support is removed; crafter-build gains --version and resolves its compiler from one place.</p>
</description> </description>

View file

@ -28,15 +28,6 @@ finish-args:
# gdb and friends from the Sdk # gdb and friends from the Sdk
- --allow=devel - --allow=devel
- --env=PATH=/app/bin:/usr/bin - --env=PATH=/app/bin:/usr/bin
build-options:
env:
CRAFTER_BUILD_LLVM_DIR: /app
# The shipped binary runs on any x86_64 from the last decade; user
# projects still build with -march=native by default.
CRAFTER_BUILD_MARCH: x86-64-v2
CRAFTER_BUILD_MTUNE: generic
# CI replaces this with the tag; a local build is a development build.
CRAFTER_BUILD_VERSION: 0.0.0-dev
cleanup: cleanup:
- /share/man - /share/man
- /share/doc - /share/doc
@ -115,14 +106,14 @@ modules:
- /bin/scan-view - /bin/scan-view
sources: sources:
- type: archive - type: archive
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz url: https://github.com/llvm/llvm-project/releases/download/llvmorg-23.1.2/llvm-project-23.1.2.src.tar.xz
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888 sha256: c98bbef08a2b4c2613cd50e9aa9ae7b69b1fe6c16b2c40373bc0ab6116fdf78a
- name: wasi-sysroot - name: wasi-sysroot
# wasi-sdk's sysroot-only mode, driven by the clang built above: # wasi-sdk's sysroot-only mode, driven by the clang built above:
# wasi-libc, compiler-rt builtins (installed into our clang's resource # wasi-libc, compiler-rt builtins (installed into our clang's resource
# dir, where the wasm32 driver looks for them) and libc++ for wasip1 and # dir, where the wasm32 driver looks for them) and libc++ for wasip1 and
# wasip1-threads. The llvm-project tree is the same tarball as above so # wasip1-threads. The llvm-project tree is the same release as above so
# the wasm libc++ matches the native one. # the wasm libc++ matches the native one.
buildsystem: cmake-ninja buildsystem: cmake-ninja
builddir: true builddir: true
@ -163,9 +154,19 @@ modules:
url: https://github.com/WebAssembly/wasi-libc.git url: https://github.com/WebAssembly/wasi-libc.git
commit: 2e6fb9d8ee0cdf9e431fbcabe8af3115de000a13 commit: 2e6fb9d8ee0cdf9e431fbcabe8af3115de000a13
dest: src/wasi-libc dest: src/wasi-libc
- type: archive # A git checkout, where the llvm module above unpacks the release
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz # tarball of the same tag: wasi-sdk treats src/llvm-project as the
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888 # submodule it is upstream. It patches the tree with `git apply` before
# building libc++, and records `git rev-parse HEAD` in the sysroot's
# VERSION file. Unpacked from a tarball the directory is a plain
# subdirectory of the wasi-sdk checkout, where git treats the patched
# paths as outside the current directory, skips every one of them and
# exits 0. LLVM 22 happened to build without the patches; LLVM 23's
# libc++ does not compile for WASI without one of them.
- type: git
url: https://github.com/llvm/llvm-project.git
tag: llvmorg-23.1.2
commit: 85ac560262434c9ccfc0c183ec22d4138ed647fb
dest: src/llvm-project dest: src/llvm-project
- type: file - type: file
path: wasi-host-toolchain.cmake path: wasi-host-toolchain.cmake
@ -180,6 +181,33 @@ modules:
url: https://github.com/bytecodealliance/wasmtime/releases/download/v49.0.2/wasmtime-v49.0.2-x86_64-linux.tar.xz url: https://github.com/bytecodealliance/wasmtime/releases/download/v49.0.2/wasmtime-v49.0.2-x86_64-linux.tar.xz
sha256: a4d6e9e3a5a60f527cf7793d674c48930c80c2e8977995b8a275cad3254b9322 sha256: a4d6e9e3a5a60f527cf7793d674c48930c80c2e8977995b8a275cad3254b9322
- name: libcxx-22-modules
# The std module source of the libc++ release before the bundled one, for
# cross sysroots that carry that libc++ without it: Alpine packages the
# headers and the library but not share/libc++/v1, and a module source
# only compiles against the headers of its own release. crafter-build
# picks <major>/std.cppm here by the sysroot's _LIBCPP_VERSION
# (SysrootStdModule in Crafter.Build:Toolchain). Add a module like this
# one per release worth supporting; the bundled clang has to be able to
# compile that release's headers.
buildsystem: simple
build-commands:
# Configure only. The cxx-modules component is std.cppm as configure
# generates it plus the .inc files straight from the source tree, so
# nothing is compiled, and libc++'s own install rule stays the one
# place that knows what a module source install consists of.
- cmake -S runtimes -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/app/bin/clang -DCMAKE_CXX_COMPILER=/app/bin/clang++ "-DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi" -DLIBCXXABI_USE_LLVM_UNWINDER=OFF -DLIBCXX_INSTALL_MODULES=ON
# Staged, not installed to /app: the component also writes
# lib/libc++.modules.json, which would replace the bundled libc++'s.
- cmake --install build --component cxx-modules --prefix "$PWD/stage"
- grep -q "_LIBCPP_VERSION 22" libcxx/include/__config
- mkdir -p /app/share/crafter-build/libc++-modules
- cp -r stage/share/libc++/v1 /app/share/crafter-build/libc++-modules/22
sources:
- type: archive
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888
- name: crafter-build - name: crafter-build
buildsystem: simple buildsystem: simple
build-options: build-options:
@ -188,6 +216,18 @@ modules:
# (sibling repos tracking their default branch, see AGENTS.md). # (sibling repos tracking their default branch, see AGENTS.md).
build-args: build-args:
- --share=network - --share=network
# On this module, not at the top of the manifest: top-level build
# options are part of every module's cache key, and the version changes
# with every commit, which made flatpak-builder rebuild LLVM each run.
env:
CRAFTER_BUILD_LLVM_DIR: /app
# The shipped binary runs on any x86_64 from the last decade; user
# projects still build with -march=native by default.
CRAFTER_BUILD_MARCH: x86-64-v2
CRAFTER_BUILD_MTUNE: generic
# CI replaces this with the version; a local build is a development
# build.
CRAFTER_BUILD_VERSION: 0.0.0-dev
build-commands: build-commands:
# Bootstrap by hand, then let the result build itself: only the # Bootstrap by hand, then let the result build itself: only the
# self-hosted binary has asset support and ThinLTO. # self-hosted binary has asset support and ThinLTO.

View file

@ -8,6 +8,134 @@ import Crafter.LintRules;
namespace fs = std::filesystem; namespace fs = std::filesystem;
using namespace Crafter; using namespace Crafter;
// The local Flatpak: `crafter-build run flatpak` builds it from this checkout
// and `crafter-build run install` installs that build. The development box
// compiles with its own clang, the shipped artifact with the LLVM bundled in
// the Flatpak, so this is how a change is tried against the real thing.
namespace {
struct FlatpakPaths {
fs::path manifest;
// flatpak-builder's downloads, ccache and per-module cache. The
// LLVM build lives here: about 2 GB, and 25 minutes to rebuild.
fs::path state;
// The OSTree repo the build is exported to; `install` reads it.
fs::path repo;
fs::path buildDir;
};
// Everything under <project>/build/flatpak, and a fixed name on purpose
// instead of a BuildDir(): the Flatpak is compiled by the toolchain
// inside the sandbox, so it does not depend on the host variant, and a
// variant-keyed directory would strand the LLVM cache on every host
// compiler update. The manifest's `skip: build` keeps it out of the
// sources the sandbox copies. flatpak-builder keys its cache on the
// manifest's absolute path, so the manifest is used where it is, never
// through a rewritten copy the way CI stamps the version in; a local
// build therefore reports version 0.0.0-dev.
FlatpakPaths FlatpakLayout(const fs::path& projectDir) {
const fs::path root = projectDir / "build" / "flatpak";
return {
.manifest = projectDir / "packaging" / "flatpak" / "net.catcrafts.CrafterBuild.yaml",
.state = root / "state",
.repo = root / "repo",
.buildDir = root / "build-dir",
};
}
// Both scripts drive the host's flatpak, and the sandbox crafter-build
// itself runs in as a Flatpak has none.
bool RefusedInSandbox(std::string_view script) {
if (!fs::exists("/.flatpak-info")) return false;
std::println(std::cerr, "run {0}: needs the host's flatpak, which the Flatpak build of crafter-build cannot reach. Use a native build: ./build.sh, then ./bin/crafter-build run {0}", script);
return true;
}
// flatpak-builder as a host package, else the org.flatpak.Builder app.
// Empty when neither is installed.
std::string FlatpakBuilder() {
if (RunCommandWithTimeout("flatpak-builder --version", std::chrono::seconds(10)).exitCode == 0) return "flatpak-builder";
if (RunCommandWithTimeout("flatpak info org.flatpak.Builder", std::chrono::seconds(10)).exitCode == 0) return "flatpak run org.flatpak.Builder";
return "";
}
// A repo path as the file:// URL of a remote. Percent-encoded, because
// `remote-modify --url=` stores what it is given as it is.
std::string FileUrl(const fs::path& path) {
std::string url = "file://";
for (const char c : path.string()) {
const bool plain = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || std::string_view("/-._~").contains(c);
if (plain) url.push_back(c);
else url += std::format("%{:02X}", static_cast<std::uint8_t>(c));
}
return url;
}
std::int32_t BuildFlatpak(ScriptContext& ctx) {
if (RefusedInSandbox("flatpak")) return 1;
// Nothing else is passed through: a flag such as --stop-at or
// --download-only would end the run without a build and leave the
// smoke test below running against the previous one.
std::string jobs;
for (const std::string& arg : ctx.args) {
const std::string_view count = arg.starts_with("--jobs=") ? std::string_view(arg).substr(7) : std::string_view();
if (count.empty() || !std::ranges::all_of(count, [](char c) { return c >= '0' && c <= '9'; })) {
std::println(std::cerr, "run flatpak: unknown argument '{}' (only --jobs=<N> is accepted)", arg);
return 1;
}
jobs = std::format(" --jobs={}", count);
}
const std::string builder = FlatpakBuilder();
if (builder.empty()) {
std::println(std::cerr, "run flatpak: neither flatpak-builder nor the org.flatpak.Builder app is installed (flatpak install flathub org.flatpak.Builder)");
return 1;
}
const FlatpakPaths paths = FlatpakLayout(ctx.projectDir);
const std::string buildDir = ShellQuote(paths.buildDir.string());
if (const std::int32_t code = ctx.Run(std::format("{}{} --user --force-clean --ccache --state-dir={} --repo={} --default-branch=local {} {}", builder, jobs, ShellQuote(paths.state.string()), ShellQuote(paths.repo.string()), buildDir, ShellQuote(paths.manifest.string()))); code != 0) return code;
// The smoke test CI runs on its build, but through `flatpak build`
// and not `<builder> --run`: the org.flatpak.Builder app (1.4.9)
// exits 1 when the command it ran succeeded and 0 when it failed,
// and --run also wants the state dir again or it drops a
// .flatpak-builder/ into the working directory.
return ctx.Run(std::format("flatpak build {} crafter-build --version", buildDir));
}
std::int32_t InstallFlatpak(ScriptContext& ctx) {
if (RefusedInSandbox("install")) return 1;
if (!ctx.args.empty()) {
std::println(std::cerr, "run install: takes no arguments (got '{}')", ctx.args.front());
return 1;
}
const FlatpakPaths paths = FlatpakLayout(ctx.projectDir);
// Deliberately not built from here: flatpak-builder never caches a
// `type: dir` module, so every build reruns build.sh and the whole
// suite inside the sandbox. Installing is the cheap half.
if (!fs::exists(paths.repo / "config")) {
std::println(std::cerr, "run install: nothing built yet, run: crafter-build run flatpak");
return 1;
}
const std::string url = ShellQuote(FileUrl(paths.repo));
// Its own branch from its own remote, next to whichever published
// branch is installed. The remote may already exist and point at
// another checkout's repo; --if-not-exists would keep that silently.
if (const std::int32_t code = ctx.Run(std::format("flatpak remote-add --user --if-not-exists --no-gpg-verify crafter-build-local {}", url)); code != 0) return code;
if (const std::int32_t code = ctx.Run(std::format("flatpak remote-modify --user --url={} crafter-build-local", url)); code != 0) return code;
if (const std::int32_t code = ctx.Run("flatpak install --user --noninteractive --or-update crafter-build-local net.catcrafts.CrafterBuild//local"); code != 0) return code;
// Only a first install becomes the branch `flatpak run` starts; an
// update leaves that choice where it was.
if (const std::int32_t code = ctx.Run("flatpak make-current --user net.catcrafts.CrafterBuild local"); code != 0) return code;
std::println(
R"(
`flatpak run net.catcrafts.CrafterBuild` now starts this build.
Back to a published branch, keeping this one installed:
flatpak make-current --user net.catcrafts.CrafterBuild <branch>
Remove it (do this before `crafter-build clean`, which deletes the repo the remote points at):
flatpak uninstall --user net.catcrafts.CrafterBuild//local
flatpak remote-delete --user crafter-build-local)");
return 0;
}
}
extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) { extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
std::vector<std::string> depArgs(args.begin(), args.end()); std::vector<std::string> depArgs(args.begin(), args.end());
Configuration* math = GitProject({ Configuration* math = GitProject({
@ -51,7 +179,7 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
// it reaches clang, which must still see a string literal. // it reaches clang, which must still see a string literal.
CrafterBuildLib->defines.push_back({"CRAFTER_BUILD_VERSION", std::format("\\\"{}\\\"", version)}); CrafterBuildLib->defines.push_back({"CRAFTER_BUILD_VERSION", std::format("\\\"{}\\\"", version)});
{ {
std::array<fs::path, 12> interfaces = { std::array<fs::path, 13> interfaces = {
"interfaces/Crafter.Build", "interfaces/Crafter.Build",
"interfaces/Crafter.Build-Toolchain", "interfaces/Crafter.Build-Toolchain",
"interfaces/Crafter.Build-Shader", "interfaces/Crafter.Build-Shader",
@ -62,10 +190,11 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
"interfaces/Crafter.Build-Clang", "interfaces/Crafter.Build-Clang",
"interfaces/Crafter.Build-Test", "interfaces/Crafter.Build-Test",
"interfaces/Crafter.Build-Lint", "interfaces/Crafter.Build-Lint",
"interfaces/Crafter.Build-Script",
"interfaces/Crafter.Build-Progress", "interfaces/Crafter.Build-Progress",
"interfaces/Crafter.Build-Asset", "interfaces/Crafter.Build-Asset",
}; };
std::array<fs::path, 11> implementations = { std::array<fs::path, 12> implementations = {
"implementations/Crafter.Build-Toolchain", "implementations/Crafter.Build-Toolchain",
"implementations/Crafter.Build-Shader", "implementations/Crafter.Build-Shader",
"implementations/Crafter.Build-Platform", "implementations/Crafter.Build-Platform",
@ -75,6 +204,7 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
"implementations/Crafter.Build-Clang", "implementations/Crafter.Build-Clang",
"implementations/Crafter.Build-Test", "implementations/Crafter.Build-Test",
"implementations/Crafter.Build-Lint", "implementations/Crafter.Build-Lint",
"implementations/Crafter.Build-Script",
"implementations/Crafter.Build-Progress", "implementations/Crafter.Build-Progress",
"implementations/Crafter.Build-Asset", "implementations/Crafter.Build-Asset",
}; };
@ -125,10 +255,19 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
cfg.AddTest("CompilerWarning").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CompilerWarning").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("CleanProject").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CleanProject").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ShaderCompile").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ShaderCompile").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ShaderVariantBuild").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ShaderIncludeChange").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("MingwMissing").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("GitSubmodules").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("LinkFlagsChange").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ExternalArtifactTime").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ConcurrentExternalFetch").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("StandardArgs").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("StandardArgs").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("TestRunnerSpec").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("TestRunnerSpec").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("VariantId").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("VariantId").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("ToolchainResolve").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ToolchainResolve").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("SysrootStdModule").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("SysrootCompileFlags").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("VersionFlag").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("VersionFlag").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("WasiBrowserRuntime").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("WasiBrowserRuntime").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("WasmVariants").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("WasmVariants").Dependencies({ CrafterBuildLib.get() });
@ -147,6 +286,8 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
.LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl"); .LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl");
cfg.AddTest("ProjectDependency").Dependencies({ CrafterBuildLib.get() }) cfg.AddTest("ProjectDependency").Dependencies({ CrafterBuildLib.get() })
.LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl"); .LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl");
cfg.AddTest("Script").Dependencies({ CrafterBuildLib.get() })
.LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl");
cfg.AddTest("ConcurrentDependencyReset").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ConcurrentDependencyReset").Dependencies({ CrafterBuildLib.get() });
cfg.AddTest("Lint").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("Lint").Dependencies({ CrafterBuildLib.get() });
} }
@ -156,5 +297,9 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
// rules also auto-fix under `crafter-build format`. // rules also auto-fix under `crafter-build format`.
LintRules::AddRules(cfg); LintRules::AddRules(cfg);
// See the top of this file.
cfg.scripts.push_back({"flatpak", BuildFlatpak});
cfg.scripts.push_back({"install", InstallFlatpak});
return cfg; return cfg;
} }

View file

@ -29,6 +29,12 @@ namespace {
std::ofstream(root / "keep-me.txt") << "untouched"; std::ofstream(root / "keep-me.txt") << "untouched";
return root; return root;
} }
std::int32_t RunWith(std::vector<std::string> args) {
std::vector<char*> argv;
for (std::string& arg : args) argv.push_back(arg.data());
return Run(static_cast<std::int32_t>(argv.size()), argv.data());
}
} }
int main() { int main() {
@ -71,6 +77,22 @@ int main() {
Check(fs::exists(cwdBin) == cwdBinExisted, "the cwd's bin/ is untouched"); Check(fs::exists(cwdBin) == cwdBinExisted, "the cwd's bin/ is untouched");
} }
{
// `clean` is a verb only where a verb can stand. After `format` it
// is a rule-name glob like any other word, and a glob must never
// delete the build trees. The project file cannot compile, so the
// run fails once it gets as far as loading it — which is the proof
// that it went on to format instead of cleaning and exiting 0.
fs::path root = StageProject("format-glob");
fs::path projectFile = root / "project.cpp";
std::ofstream(projectFile) << "this is not valid C++ at all\n";
const std::int32_t exitCode = RunWith({"crafter-build", "format", "clean", std::format("--project={}", projectFile.string())});
Check(fs::exists(root / "bin" / "app-x86_64-pc-linux-gnu-native-native-0badf00d" / "app"), "`format clean` leaves bin/ alone");
Check(fs::exists(root / "build" / "app-x86_64-pc-linux-gnu-native-native-0badf00d" / "main_impl.o"), "`format clean` leaves build/ alone");
Check(exitCode != 0, "`format clean` goes on to load the project instead of cleaning");
}
if (Failures > 0) { if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures); std::println(std::cerr, "{} assertions failed", Failures);
return 1; return 1;

View file

@ -0,0 +1,95 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// Builds that run at the same time share their dependencies' clones in the
// cache: the tests `crafter-build test --jobs=N` builds side by side each
// declare the same ExternalDependency, and two crafter-builds can run at
// once. Fetching a clone that already exists ran `git pull` (and the CMake
// build) in it with nothing to keep a second fetch out, and the second git
// fails on the first one's index.lock. A clone's fetch and build now hold a
// lock of their own.
//
// Eight threads fetch the same tracked branch of a local repository at
// once, a few rounds over, the branch moving between rounds so every pull
// has work to do; every fetch has to succeed. Everything is local;
// XDG_CACHE_HOME keeps the clone out of the real cache.
extern "C" int setenv(const char* name, const char* value, int overwrite);
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
void Git(const fs::path& dir, std::string_view args) {
const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args);
auto result = RunCommandWithTimeout(command, std::chrono::seconds(60));
if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output));
}
void Commit(const fs::path& repo, std::uint32_t round) {
std::ofstream(repo / "round.txt", std::ios::binary | std::ios::trunc) << round;
Git(repo, "add round.txt");
Git(repo, std::format("commit -q -m round-{}", round));
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-concurrent-external-fetch";
fs::remove_all(scratch);
fs::create_directories(scratch);
setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1);
setenv("GIT_CONFIG_NOSYSTEM", "1", 1);
setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1);
for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1);
for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1);
const fs::path repo = scratch / "shared";
fs::create_directories(repo);
Git(repo, "init -q -b main");
Commit(repo, 0);
const GitSource source { .url = std::format("file://{}", repo.string()), .branch = "main" };
const fs::path clone = GitFetch(source);
constexpr std::uint32_t threads = 8;
constexpr std::uint32_t rounds = 4;
std::vector<std::string> errors;
std::mutex errorsMutex;
for (std::uint32_t round = 1; round <= rounds; ++round) {
Commit(repo, round);
std::vector<std::jthread> fetchers;
for (std::uint32_t i = 0; i < threads; ++i) {
fetchers.emplace_back([&] {
try {
(void)GitFetch(source);
} catch (const std::exception& e) {
std::lock_guard lock(errorsMutex);
errors.push_back(e.what());
}
});
}
}
Check(errors.empty(), std::format("{} fetches at once, {} rounds: every one succeeds ({} failed{}{})", threads, rounds, errors.size(), errors.empty() ? "" : ", the first: ", errors.empty() ? "" : errors.front()));
std::ifstream round(clone / "round.txt");
std::uint32_t last = 0;
round >> last;
Check(last == rounds, std::format("and the clone ends at the last round ({})", last));
std::error_code ec;
fs::remove_all(scratch, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,89 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A CMake dependency's newest library decides when its consumers are stale
// (BuildExternal's latestArtifact). Only files directly in the CMake build
// directory named *.a or *.so were looked at, so msquic's
// bin/Release/libmsquic.so.2, in one of the dependency's libDirs and with a
// version after the .so, never counted: a rebuilt msquic left everything
// built against the old one in place.
//
// The dependency here is a local repository whose CMake project puts a
// versioned shared library in bin/Release, declared as a libDir. Everything
// is local; XDG_CACHE_HOME keeps the clone out of the real cache.
extern "C" int setenv(const char* name, const char* value, int overwrite);
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
void Git(const fs::path& dir, std::string_view args) {
const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args);
auto result = RunCommandWithTimeout(command, std::chrono::seconds(60));
if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output));
}
void Write(const fs::path& file, std::string_view text) {
std::ofstream(file, std::ios::binary | std::ios::trunc) << text;
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-external-artifact-time";
fs::remove_all(scratch);
fs::create_directories(scratch);
setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1);
setenv("GIT_CONFIG_NOSYSTEM", "1", 1);
setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1);
for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1);
for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1);
const fs::path repo = scratch / "versioned";
fs::create_directories(repo);
Write(repo / "lib.c", "int Versioned(void) { return 2; }\n");
Write(repo / "CMakeLists.txt",
"cmake_minimum_required(VERSION 3.16)\n"
"project(versioned C)\n"
"add_library(versioned SHARED lib.c)\n"
"set_target_properties(versioned PROPERTIES VERSION 2.0.0 SOVERSION 2\n"
" LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin/Release)\n");
Git(repo, "init -q -b main");
Git(repo, "add .");
Git(repo, "commit -q -m versioned");
ExternalDependency dep;
dep.name = "versioned";
dep.source.url = std::format("file://{}", repo.string());
dep.builder = ExternalBuilder::CMake;
dep.libDirs = { "bin/Release" };
dep.libs = { "versioned" };
std::atomic<bool> cancelled = false;
const ExternalBuildResult result = BuildExternal(dep, HostTarget(), cancelled);
Check(result.error.empty(), std::format("the dependency builds: {}", result.error));
const fs::path library = ExternalCloneDir(dep, HostTarget()) / "build" / "bin" / "Release" / "libversioned.so.2.0.0";
Check(fs::exists(library), std::format("the library is where its CMake project puts it: {}", library.string()));
if (fs::exists(library)) {
Check(result.latestArtifact == fs::last_write_time(library), "its newest artifact is the versioned library in the libDir");
}
std::error_code ec;
fs::remove_all(scratch, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,128 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A fetched dependency's submodules have to match the revision it was fetched
// at. They used to come from the remote's default branch: a fresh clone was
// `git clone --recursive`, which checks out the default branch's submodules,
// and the `switch` or `checkout` to the requested branch or commit that came
// after it left them where they were. A branch that was pulled forward kept
// the submodules of its old tip as well. So pinning msquic to a release built
// that release against the main branch's quictls.
//
// The test makes a repository whose submodule moves between two of its
// commits, then fetches it four ways through GitFetch:
// - pinned to the older commit;
// - by a branch at the older commit;
// - by a branch that is then moved to the newer commit and fetched again;
// - pinned again after the cached clone's submodule was checked out
// elsewhere, as an older crafter-build left it.
// Everything is local; XDG_CACHE_HOME keeps the clones out of the real cache.
extern "C" int setenv(const char* name, const char* value, int overwrite);
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
// Runs git in `dir`; the output, or an exception naming the command.
std::string Git(const fs::path& dir, std::string_view args) {
const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args);
auto result = RunCommandWithTimeout(command, std::chrono::seconds(60));
if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output));
while (!result.output.empty() && (result.output.back() == '\n' || result.output.back() == '\r')) result.output.pop_back();
return result.output;
}
void Write(const fs::path& file, std::string_view text) {
std::ofstream(file, std::ios::binary | std::ios::trunc) << text;
}
std::string Read(const fs::path& file) {
std::ifstream stream(file, std::ios::binary);
return std::string(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-git-submodules";
fs::remove_all(scratch);
fs::create_directories(scratch);
setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1);
// A known git: no user or system configuration, an identity for the
// commits, and file:// allowed for submodules (git refuses it by default),
// also for the git that GitFetch runs.
setenv("GIT_CONFIG_NOSYSTEM", "1", 1);
setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1);
setenv("GIT_CONFIG_COUNT", "1", 1);
setenv("GIT_CONFIG_KEY_0", "protocol.file.allow", 1);
setenv("GIT_CONFIG_VALUE_0", "always", 1);
for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1);
for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1);
// The submodule: "one", then "two".
const fs::path sub = scratch / "sub";
fs::create_directories(sub);
Git(sub, "init -q -b main");
Write(sub / "value.txt", "one");
Git(sub, "add value.txt");
Git(sub, "commit -q -m one");
const std::string one = Git(sub, "rev-parse HEAD");
Write(sub / "value.txt", "two");
Git(sub, "commit -q -am two");
const std::string two = Git(sub, "rev-parse HEAD");
// The superproject: `older` has the submodule at "one", main at "two".
const fs::path super = scratch / "super";
fs::create_directories(super);
Git(super, "init -q -b main");
Git(super, std::format("submodule --quiet add {} lib", ShellQuote(std::format("file://{}", sub.string()))));
Git(super / "lib", std::format("checkout -q {}", one));
Git(super, "add lib");
Git(super, "commit -q -m older");
const std::string older = Git(super, "rev-parse HEAD");
Git(super, "branch older");
Git(super, "branch track");
Git(super / "lib", std::format("checkout -q {}", two));
Git(super, "add lib");
Git(super, "commit -q -m newer");
const std::string url = std::format("file://{}", super.string());
// ── Pinned to the older commit ───────────────────────────────────────────
const fs::path pinned = GitFetch({ .url = url, .commit = older });
Check(Read(pinned / "lib" / "value.txt") == "one", std::format("a commit pin gets that commit's submodule (lib says {})", Read(pinned / "lib" / "value.txt")));
// ── By a branch at the older commit ──────────────────────────────────────
const fs::path branch = GitFetch({ .url = url, .branch = "older" });
Check(Read(branch / "lib" / "value.txt") == "one", std::format("a branch gets its own tip's submodule (lib says {})", Read(branch / "lib" / "value.txt")));
// ── A tracked branch moving forward ──────────────────────────────────────
const fs::path tracked = GitFetch({ .url = url, .branch = "track" });
Check(Read(tracked / "lib" / "value.txt") == "one", "a tracked branch starts with its tip's submodule");
Git(super, "branch -f track main");
const fs::path pulled = GitFetch({ .url = url, .branch = "track" });
Check(pulled == tracked && Read(pulled / "lib" / "value.txt") == "two", std::format("pulling it forward moves the submodule along (lib says {})", Read(pulled / "lib" / "value.txt")));
// ── A cached clone with its submodule out of step ────────────────────────
Git(pinned / "lib", std::format("checkout -q {}", two));
const fs::path repaired = GitFetch({ .url = url, .commit = older });
Check(repaired == pinned && Read(repaired / "lib" / "value.txt") == "one", std::format("a cached clone at the pin gets its submodule put back (lib says {})", Read(repaired / "lib" / "value.txt")));
std::error_code ec;
fs::remove_all(scratch, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -119,6 +119,17 @@ int main() {
Check(fs::last_write_time(probePcm) == afterCold, "unchanged sources do not rebuild the cache"); Check(fs::last_write_time(probePcm) == afterCold, "unchanged sources do not rebuild the cache");
Check(ReadStamp(moduleCacheDir) == firstStamp, "unchanged sources keep the same stamp"); Check(ReadStamp(moduleCacheDir) == firstStamp, "unchanged sources keep the same stamp");
// The compiled project file is the other half of the same hazard: it is
// loaded into this process and must have been compiled against the
// declarations this process was built from.
fs::path projectSo;
for (const fs::directory_entry& entry : fs::recursive_directory_iterator(projectDir)) {
if (entry.path().filename() == "project.so") projectSo = entry.path();
}
Check(!projectSo.empty(), "the project file was compiled to a project.so");
if (projectSo.empty()) return 1;
const fs::file_time_type soBeforeEdit = fs::last_write_time(projectSo);
// Now the case the mtime rule cannot see. The edit lands in a module the // Now the case the mtime rule cannot see. The edit lands in a module the
// probe does not name, because the cached PCMs import each other: a change // probe does not name, because the cached PCMs import each other: a change
// anywhere in the set invalidates all of them. // anywhere in the set invalidates all of them.
@ -139,6 +150,11 @@ int main() {
Check(fs::last_write_time(probePcm) > afterCold, "a content change rebuilds the cache even when the source's mtime is older"); Check(fs::last_write_time(probePcm) > afterCold, "a content change rebuilds the cache even when the source's mtime is older");
Check(ReadStamp(moduleCacheDir) != firstStamp, "the recorded stamp follows the sources"); Check(ReadStamp(moduleCacheDir) != firstStamp, "the recorded stamp follows the sources");
// project.cpp itself is untouched, so only the stamp can force this. It
// once covered the version string alone (the rest sat behind a NUL in a
// format string): a project.so built against a class of another size was
// reused and corrupted the heap when it filled one in.
Check(fs::last_write_time(projectSo) > soBeforeEdit, "changed declarations recompile project.so even though project.cpp is unchanged");
fs::remove_all(scratch, ec); fs::remove_all(scratch, ec);

View file

@ -0,0 +1,6 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
int main() {
return 0;
}

View file

@ -0,0 +1,119 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A change to what an executable is linked with has to relink it, even when
// no object changed. The relink check compared modification times only, so a
// link flag added in project.cpp, or a dependency's library moving (an
// ExternalDependency whose options changed is fetched and built into a new
// cache directory, and its -L and rpath follow), left the old binary in
// place: Crafter.Network's tests kept loading the msquic they were first
// linked against.
//
// The test adds an rpath to an executable's link flags, takes it away again,
// and then builds once more with nothing changed, which must not relink.
// The flags come out of sets the dependency threads fill as they finish, so
// the recorded command lists them sorted: in the order they happened to land
// in, an unchanged build could look like a changed one.
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
constexpr std::string_view marker = "/crafter-build-link-flags-change-marker";
BuildResult BuildOnce(const fs::path& project, std::vector<std::string> linkFlags) {
Configuration app;
app.path = project;
app.name = "linked";
app.outputName = "linked";
app.target = HostTarget();
app.type = ConfigurationType::Executable;
app.linkFlags = std::move(linkFlags);
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls = { "main" };
app.GetInterfacesAndImplementations(ifaces, impls);
std::unordered_map<fs::path, std::shared_future<BuildResult>> depResults;
std::mutex depMutex;
BuildResult result = Build(app, depResults, depMutex);
return result;
}
Configuration Plain(const fs::path& project) {
Configuration app;
app.path = project;
app.name = "linked";
app.outputName = "linked";
app.target = HostTarget();
app.type = ConfigurationType::Executable;
return app;
}
fs::path Binary(const fs::path& project) {
return Plain(project).BinDir() / "linked";
}
bool HasMarker(const fs::path& binary) {
std::ifstream stream(binary, std::ios::binary);
const std::string bytes((std::istreambuf_iterator<char>(stream)), std::istreambuf_iterator<char>());
return bytes.find(marker) != std::string::npos;
}
}
int main() {
const fs::path project = fs::temp_directory_path() / "crafter-build-link-flags-change";
fs::remove_all(project);
fs::copy(fs::current_path() / "tests" / "LinkFlagsChange" / "fixture", project, fs::copy_options::recursive);
const std::string rpath = std::format("-Wl,-rpath,{}", marker);
const BuildResult first = BuildOnce(project, {});
Check(first.result.empty(), std::format("the plain build succeeds: {}", first.result));
const fs::path binary = Binary(project);
Check(fs::exists(binary) && !HasMarker(binary), "the plain binary has no rpath to the marker");
const BuildResult added = BuildOnce(project, { rpath });
Check(added.result.empty(), std::format("the build with the rpath succeeds: {}", added.result));
Check(HasMarker(binary), "adding a link flag relinks the binary");
const BuildResult removed = BuildOnce(project, {});
Check(removed.result.empty(), std::format("the build without it succeeds: {}", removed.result));
Check(!HasMarker(binary), "taking it away relinks it again");
const fs::file_time_type linked = fs::last_write_time(binary);
const BuildResult again = BuildOnce(project, {});
Check(again.result.empty() && fs::last_write_time(binary) == linked, "an unchanged build doesn't relink");
const std::vector<std::string> flags = { "-Wl,-z,now", "-Wl,-O1", "-Wl,--as-needed", "-Wl,-z,relro", "-Wl,--build-id", "-Wl,--hash-style=gnu" };
const BuildResult many = BuildOnce(project, flags);
Check(many.result.empty(), std::format("the build with six flags succeeds: {}", many.result));
std::ifstream recorded(Plain(project).BuildDir() / "linked.link", std::ios::binary);
const std::string record((std::istreambuf_iterator<char>(recorded)), std::istreambuf_iterator<char>());
std::vector<std::size_t> at;
for (const std::string& flag : flags) at.push_back(record.find(std::format(" {}", flag)));
std::vector<std::string> sorted = flags;
std::ranges::sort(sorted);
bool inOrder = !std::ranges::contains(at, std::string::npos);
for (std::size_t k = 0; inOrder && k + 1 < sorted.size(); ++k) {
const auto index = [&](const std::string& flag) { return at[static_cast<std::size_t>(std::ranges::find(flags, flag) - flags.begin())]; };
inOrder = index(sorted[k]) < index(sorted[k + 1]);
}
Check(inOrder, std::format("the link record lists the flags sorted: {}", record));
std::error_code ec;
fs::remove_all(project, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,9 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
int main() {
std::println("hello from crafter-build");
return 0;
}

View file

@ -0,0 +1,97 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// Building for x86_64-w64-mingw32 without a usable mingw-w64 libstdc++ has to
// fail the build with a message that says what is missing. It used to abort
// the whole process with a bare "libc++abi: terminating": the std module's
// source was found by listing <mingw sysroot>/include/c++ with a throwing
// directory_iterator, the exception unwound through Build() while its worker
// threads were still joinable, and ~thread called std::terminate. That is
// what every mingw build inside the Flatpak did, as the sandbox has no
// /usr/x86_64-w64-mingw32.
//
// The std module's source is now found through libstdc++.modules.json in the
// mingw toolchain's library directories. CRAFTER_MINGW_DIR points the
// toolchain at a scratch prefix, which the test takes through four states:
// - missing;
// - with only a manifest in the prefix's generic lib/, where the host's
// Linux libstdc++ keeps one: not the mingw library, so still missing;
// - with a mingw manifest that names no std module;
// - with one whose std module source isn't there.
extern "C" int setenv(const char* name, const char* value, int overwrite);
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
BuildResult BuildOnce(const fs::path& project) {
Configuration app;
app.path = project;
app.name = "hello";
app.outputName = "hello";
app.target = "x86_64-w64-mingw32";
app.type = ConfigurationType::Executable;
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls = { "main" };
app.GetInterfacesAndImplementations(ifaces, impls);
std::unordered_map<fs::path, std::shared_future<BuildResult>> depResults;
std::mutex depMutex;
return Build(app, depResults, depMutex);
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-mingw-missing";
fs::remove_all(scratch);
fs::create_directories(scratch);
const fs::path project = scratch / "project";
fs::copy(fs::current_path() / "tests" / "MingwMissing" / "fixture", project, fs::copy_options::recursive);
const fs::path sysroot = scratch / "mingw";
// Before anything resolves the toolchain, which reads both once. The std
// module cache goes to the scratch directory too.
setenv("CRAFTER_MINGW_DIR", sysroot.c_str(), 1);
setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1);
const BuildResult missing = BuildOnce(project);
Check(!missing.result.empty(), "a missing mingw prefix fails the build");
Check(missing.result.contains("mingw") && missing.result.contains(sysroot.string()) && missing.result.contains("CRAFTER_MINGW_DIR"), std::format("and says where it looked and how to point elsewhere: {}", missing.result));
auto write = [](const fs::path& file, std::string_view text) {
fs::create_directories(file.parent_path());
std::ofstream(file, std::ios::binary | std::ios::trunc) << text;
};
write(sysroot / "lib" / "libstdc++.modules.json", R"({"version": 1, "modules": [{"logical-name": "std", "source-path": "../include/c++/16/bits/std.cc", "is-std-library": true}]})");
write(sysroot / "include" / "c++" / "16" / "bits" / "std.cc", "// the host's Linux std module\n");
const BuildResult linux = BuildOnce(project);
Check(linux.result.contains("no mingw-w64 libstdc++"), std::format("a manifest outside the mingw toolchain's own directories isn't taken for it: {}", linux.result));
const fs::path manifest = sysroot / "x86_64-w64-mingw32" / "lib" / "libstdc++.modules.json";
write(manifest, R"({"version": 1, "modules": [{"logical-name": "std.compat", "source-path": "../include/c++/15.1.0/bits/std.compat.cc"}]})");
const BuildResult noStd = BuildOnce(project);
Check(noStd.result.contains(manifest.string()) && noStd.result.contains("no std module"), std::format("a manifest without the std module fails the build, naming the manifest: {}", noStd.result));
write(manifest, R"({"version": 1, "modules": [{"source-path": "../include/c++/15.1.0/bits/std.cc", "logical-name": "std"}]})");
const BuildResult noSource = BuildOnce(project);
Check(noSource.result.contains((sysroot / "x86_64-w64-mingw32" / "include" / "c++" / "15.1.0" / "bits" / "std.cc").string()) && noSource.result.contains("missing"),
std::format("a std module source that isn't there fails the build, naming it: {}", noSource.result));
std::error_code ec;
fs::remove_all(scratch, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,30 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
using namespace Crafter;
// Scripts that report what they were handed. The test that loads this file
// drives the real command line and can only observe a script through its
// exit code and through what it leaves in the project directory.
extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
Configuration cfg;
cfg.path = "./";
cfg.name = "script-fixture";
cfg.outputName = "script-fixture";
// Applied so that a script argument which leaked into the project's
// args would be visible: in cfg.projectArgs, or as cfg.debug.
ApplyStandardArgs(cfg, args);
cfg.scripts.push_back({"record", [](ScriptContext& ctx) {
std::ofstream out(ctx.projectDir / "record.txt");
for (const std::string& arg : ctx.args) std::println(out, "arg:{}", arg);
for (const std::string& arg : ctx.project.projectArgs) std::println(out, "projectArg:{}", arg);
std::println(out, "debug:{}", ctx.project.debug);
std::println(out, "project:{}", ctx.project.name);
return 42;
}});
cfg.scripts.push_back({"shell", [](ScriptContext& ctx) { return ctx.Run("exit 7"); }});
return cfg;
}

241
tests/Script/main.cpp Normal file
View file

@ -0,0 +1,241 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// `crafter-build run <name> [args...]` runs a script project.cpp put in
// cfg.scripts and exits with what the script returned.
//
// Two things are easy to get wrong and expensive when they are. The words
// after the script's name belong to the script: read by crafter-build
// instead, `run package clean` deletes the build tree, `run package --help`
// prints the wrong help, and anything else lands in the project's args,
// where it changes the variant hash and so the directory the project builds
// into. And the exit code is the script's whole result: collapsed to 0 or 1
// it loses what a caller branches on, and passed through unchecked a value
// such as 256 wraps to success.
namespace {
std::int32_t Failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++Failures;
}
}
std::int32_t RunWith(std::vector<std::string> args) {
std::vector<char*> argv;
for (std::string& arg : args) argv.push_back(arg.data());
return Run(static_cast<std::int32_t>(argv.size()), argv.data());
}
std::vector<std::string> ReadLines(const fs::path& file) {
std::vector<std::string> lines;
std::ifstream in(file);
for (std::string line; std::getline(in, line);) lines.push_back(line);
return lines;
}
// One script that returns `code`, run by name.
std::int32_t RunReturning(std::int32_t code) {
Configuration cfg;
cfg.scripts.push_back({"it", [code](ScriptContext&) { return code; }});
return RunScript(cfg, {.name = "it", .named = true});
}
// LoadProject compiles a project file against the installed module
// sources; without them the command-line cases cannot run at all.
bool ModuleSourcesInstalled() {
try {
return fs::exists(GetCrafterBuildHome() / "Crafter.Build.cppm");
} catch (const std::exception&) {
return false;
}
}
}
int main() {
const fs::path fixture = fs::current_path() / "tests" / "Script" / "fixture";
const fs::path scratch = fs::temp_directory_path() / "crafter-build-script";
std::error_code ec;
fs::remove_all(scratch, ec);
fs::create_directories(scratch);
fs::copy(fixture, scratch, fs::copy_options::recursive);
// The script gets the project, the project's directory and its
// arguments exactly as given, and its return value comes back.
{
Configuration cfg;
std::int32_t calls = 0;
std::vector<std::string> seen;
const Configuration* seenProject = nullptr;
fs::path seenDir;
cfg.scripts.push_back({"other", [&](ScriptContext&) {
++calls;
return 9;
}});
cfg.scripts.push_back({"pass", [&](ScriptContext& ctx) {
++calls;
seen.assign(ctx.args.begin(), ctx.args.end());
seenProject = &ctx.project;
seenDir = ctx.projectDir;
return 3;
}});
const std::vector<std::string> given = {"a", "--help", "", "two words", "it's"};
const std::int32_t code = RunScript(cfg, {.name = "pass", .named = true, .args = given, .projectFile = scratch / "project.cpp"});
Check(code == 3, std::format("the script's return value is the result (got {})", code));
Check(calls == 1, std::format("only the named script ran (ran {})", calls));
Check(seen == given, "the arguments arrive verbatim, empty ones and option-like ones included");
Check(seenProject == &cfg, "ctx.project is the configuration the script was declared on");
Check(seenDir == scratch, std::format("ctx.projectDir is the project file's directory (got '{}')", seenDir.string()));
// Asking for the list runs nothing, with or without a name.
calls = 0;
Check(RunScript(cfg, {}) == 0, "a run without a name lists and succeeds");
Check(RunScript(cfg, {.name = "pass", .named = true, .listOnly = true}) == 0, "--list succeeds");
Check(calls == 0, "listing runs no script");
// A name nobody declared. The empty string is one: it is a name
// that was given, not the absence of a name.
Check(RunScript(cfg, {.name = "nope", .named = true}) == 1, "an unknown name fails");
Check(RunScript(cfg, {.name = "", .named = true}) == 1, "an empty name is looked up and fails instead of listing");
Check(RunScript(cfg, {.name = "pas", .named = true}) == 1, "names match exactly, not by prefix");
Check(calls == 0, "an unknown name runs no script");
}
// The exit status passes through whole, and only an exit status does.
Check(RunReturning(0) == 0, "0 stays 0");
Check(RunReturning(1) == 1, "1 stays 1");
Check(RunReturning(77) == 77, "77 stays 77");
Check(RunReturning(255) == 255, "255 stays 255");
Check(RunReturning(256) == 1, "256 would wrap to success as an exit status and becomes 1");
Check(RunReturning(-1) == 1, "a negative value becomes 1");
// A script that throws is that script's failure, not a crash.
{
Configuration cfg;
cfg.scripts.push_back({"throws", [](ScriptContext&) -> std::int32_t { throw std::runtime_error("no such tool"); }});
Check(RunScript(cfg, {.name = "throws", .named = true}) == 1, "a throwing script exits 1");
cfg.scripts.push_back({"throwsInt", [](ScriptContext&) -> std::int32_t { throw 7; }});
Check(RunScript(cfg, {.name = "throwsInt", .named = true}) == 1, "and so does one throwing something that isn't a std::exception");
}
// Declarations that cannot work are refused before anything runs, also
// when a different, valid script was asked for.
{
Configuration none;
Check(RunScript(none, {.name = "it", .named = true}) == 1, "a project without scripts fails");
Check(RunScript(none, {}) == 1, "listing a project without scripts fails too");
std::int32_t calls = 0;
const auto count = [&](ScriptContext&) {
++calls;
return 0;
};
Configuration twice;
twice.scripts.push_back({"a", count});
twice.scripts.push_back({"b", count});
twice.scripts.push_back({"a", count});
Check(RunScript(twice, {.name = "b", .named = true}) == 1, "a name declared twice fails the run");
Check(RunScript(twice, {}) == 1, "a name declared twice fails the list");
Configuration dashed;
dashed.scripts.push_back({"ok", count});
dashed.scripts.push_back({"-x", count});
Check(RunScript(dashed, {.name = "ok", .named = true}) == 1, "a name starting with '-' could never be typed and is refused");
Configuration unnamed;
unnamed.scripts.push_back({"", count});
Check(RunScript(unnamed, {.name = "", .named = true}) == 1, "an empty name is refused");
Configuration hollow;
hollow.scripts.push_back({"hollow", nullptr});
Check(RunScript(hollow, {.name = "hollow", .named = true}) == 1, "a script without a callback is refused");
Check(calls == 0, "none of these ran a script");
}
// ctx.Run: the command's real exit code, a signal death the way a shell
// reports it, and ShellQuote carrying awkward text through sh intact.
{
const std::string awkward = "it's a \"test\" $HOME `id` \\ *";
const fs::path echoed = scratch / "echoed file.txt";
Configuration cfg;
cfg.scripts.push_back({"exit", [](ScriptContext& ctx) { return ctx.Run("exit 7"); }});
cfg.scripts.push_back({"signal", [](ScriptContext& ctx) { return ctx.Run("kill -TERM $$"); }});
cfg.scripts.push_back({"quote", [&](ScriptContext& ctx) { return ctx.Run(std::format("printf %s {} > {}", ShellQuote(awkward), ShellQuote(echoed.string()))); }});
// A script is usually several commands: stop at the first failure.
cfg.scripts.push_back({"steps", [&](ScriptContext& ctx) {
if (const std::int32_t code = ctx.Run("exit 5"); code != 0) return code;
return ctx.Run(std::format("touch {}", ShellQuote((scratch / "not-reached").string())));
}});
Check(RunScript(cfg, {.name = "exit", .named = true}) == 7, "ctx.Run returns the command's exit code");
Check(RunScript(cfg, {.name = "signal", .named = true}) == 143, "a command killed by SIGTERM reads as 128+15");
Check(RunScript(cfg, {.name = "quote", .named = true}) == 0, "a quoted command line runs");
const std::vector<std::string> lines = ReadLines(echoed);
Check(lines.size() == 1 && lines[0] == awkward, "ShellQuote passes quotes, spaces and expansions through sh untouched");
Check(RunScript(cfg, {.name = "steps", .named = true}) == 5, "a failed step's code can be returned as the script's");
Check(!fs::exists(scratch / "not-reached"), "and the steps after it do not run");
}
if (!ModuleSourcesInstalled()) {
// A skip must never cover for a failure above.
if (Failures > 0) return 1;
std::println(std::cerr, "SKIP: no module sources to load the fixture project with");
return 77;
}
// The same through the command line, where the parsing lives. Sentinels
// stand in for a built project: a word after `run` that is read as the
// clean verb takes them with it.
const std::string project = std::format("--project={}", (scratch / "project.cpp").string());
const fs::path binSentinel = scratch / "bin" / "sentinel";
const fs::path buildSentinel = scratch / "build" / "sentinel";
fs::create_directories(binSentinel.parent_path());
fs::create_directories(buildSentinel.parent_path());
std::ofstream(binSentinel) << "binary";
std::ofstream(buildSentinel) << "object";
// Everything after the name is the script's, including this tool's own
// options and verbs; what stands before `run` is still the project's.
{
const std::int32_t code = RunWith({"crafter-build", project, "--flavour=x", "run", "record", "a", "--help", "-v", "clean", "--debug", "--version", "run"});
Check(code == 42, std::format("the script ran and its return value is the exit code (got {})", code));
const std::vector<std::string> expected = {
"arg:a", "arg:--help", "arg:-v", "arg:clean", "arg:--debug", "arg:--version", "arg:run",
"projectArg:--flavour=x",
"debug:false",
"project:script-fixture",
};
const std::vector<std::string> record = ReadLines(scratch / "record.txt");
Check(record == expected, "the script got every word after its name and the project args got none of them");
if (record != expected) {
for (const std::string& line : record) std::println(std::cerr, " recorded: {}", line);
}
Check(fs::exists(binSentinel) && fs::exists(buildSentinel), "`clean` as a script argument deleted nothing");
}
// The first word after `run` is the name even when it is a verb.
{
fs::remove(scratch / "record.txt");
Check(RunWith({"crafter-build", project, "run", "clean"}) == 1, "`run clean` looks for a script called clean");
Check(fs::exists(binSentinel) && fs::exists(buildSentinel), "`run clean` deleted nothing");
Check(RunWith({"crafter-build", project, "run", "help"}) == 1, "`run help` looks for a script called help");
}
Check(RunWith({"crafter-build", project, "run", "shell"}) == 7, "a command's exit code reaches the process exit code");
Check(RunWith({"crafter-build", project, "run", "-q", "shell"}) == 7, "an option between `run` and the name is still this tool's");
Check(RunWith({"crafter-build", project, "run"}) == 0, "a bare `run` lists");
Check(RunWith({"crafter-build", project, "run", "--list"}) == 0, "`run --list` lists");
Check(!fs::exists(scratch / "record.txt"), "listing ran no script");
Check(RunWith({"crafter-build", project, "run", "nope"}) == 1, "an unknown script fails");
if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,15 @@
#version 450
// Compiled twice by the ShaderCompile test: once as is, once with VARIANT_RED
// and VARIANT_ALPHA=0.5 defined. The red branch does not compile unless the
// NAME=VALUE define arrives, and the two outputs only differ if the bare one
// does.
layout(location = 0) out vec4 outColor;
void main() {
#ifdef VARIANT_RED
outColor = vec4(1.0, 0.0, 0.0, VARIANT_ALPHA);
#else
outColor = vec4(0.0, 1.0, 0.0, 1.0);
#endif
}

View file

@ -8,7 +8,10 @@ using namespace Crafter;
// Compile a tiny GLSL fragment shader directly via the Shader API. Verifies // Compile a tiny GLSL fragment shader directly via the Shader API. Verifies
// the SPIR-V output lands in the target dir and starts with the SPIR-V magic // the SPIR-V output lands in the target dir and starts with the SPIR-V magic
// number, without dragging in the full Build() pipeline. // number, without dragging in the full Build() pipeline. The second half
// registers one source twice, with and without defines, and checks that the
// variant gets its own output, its own staleness and the other #ifdef branch
// (tests/ShaderVariantBuild drives the same thing through Build()).
int main() { int main() {
fs::path src = fs::current_path() / "tests" / "ShaderCompile" / "fixture" / "triangle.frag"; fs::path src = fs::current_path() / "tests" / "ShaderCompile" / "fixture" / "triangle.frag";
if (!fs::exists(src)) { if (!fs::exists(src)) {
@ -24,7 +27,7 @@ int main() {
Shader shader(fs::path(src), "main", ShaderType::Fragment); Shader shader(fs::path(src), "main", ShaderType::Fragment);
std::array<fs::path, 0> includeDirs = {}; std::array<fs::path, 0> includeDirs = {};
std::string err = shader.Compile(outDir, includeDirs); std::string err = shader.Compile(outDir, outDir, includeDirs);
if (!err.empty()) { if (!err.empty()) {
std::println(std::cerr, "shader compile failed: {}", err); std::println(std::cerr, "shader compile failed: {}", err);
return 1; return 1;
@ -48,9 +51,96 @@ int main() {
} }
// Compiling again should be a no-op since the source isn't newer. // Compiling again should be a no-op since the source isn't newer.
if (!shader.Check(outDir)) { if (!shader.Check(outDir, outDir, includeDirs)) {
std::println(std::cerr, "Shader::Check returned false on an up-to-date spv"); std::println(std::cerr, "Shader::Check returned false on an up-to-date spv");
return 1; return 1;
} }
// Searched elsewhere, an #include could find another file: stale.
const std::array<fs::path, 1> elsewhere = { outDir / "elsewhere" };
if (shader.Check(outDir, outDir, elsewhere)) {
std::println(std::cerr, "Shader::Check called an spv up to date for other include directories");
return 1;
}
// So could a source of the same name moved to another directory, which
// its #includes are searched from first.
const fs::path movedDir = outDir / "moved";
fs::create_directories(movedDir);
fs::copy_file(src, movedDir / src.filename());
fs::last_write_time(movedDir / src.filename(), fs::last_write_time(src));
Shader moved(movedDir / src.filename(), "main", ShaderType::Fragment);
if (moved.Check(outDir, outDir, includeDirs)) {
std::println(std::cerr, "Shader::Check called an spv up to date for a source of the same name elsewhere");
return 1;
}
// One source registered twice: the second entry carries defines and a
// variant name. It must land in its own file instead of overwriting the
// first, and it must take the other #ifdef branch. Crafter.Graphics ships
// each UI shader in a descriptor-heap and a descriptor-set flavour this
// way; if the variant were dropped from the output name, whichever compile
// finished last would silently win.
fs::path variantSrc = fs::current_path() / "tests" / "ShaderCompile" / "fixture" / "variant.frag";
Shader plain(fs::path(variantSrc), "main", ShaderType::Fragment);
Shader red(fs::path(variantSrc), "main", ShaderType::Fragment);
red.defines = {"VARIANT_RED", "VARIANT_ALPHA=0.5"};
red.variant = "red";
if (plain.OutputName() != "variant.spv" || red.OutputName() != "variant.red.spv") {
std::println(std::cerr, "unexpected output names: {} and {}", plain.OutputName().string(), red.OutputName().string());
return 1;
}
for (const Shader* each : {&plain, &red}) {
// Staleness is per output, not per source: while only variant.spv
// exists, the variant is still due. A Check() that ignored the
// variant would call it up to date here, and Build() would then
// never compile a variant added to an already built project.
if (each == &red && red.Check(outDir, outDir, includeDirs)) {
std::println(std::cerr, "Shader::Check called the variant up to date before it was ever compiled");
return 1;
}
std::string variantErr = each->Compile(outDir, outDir, includeDirs);
if (!variantErr.empty()) {
std::println(std::cerr, "variant shader compile failed: {}", variantErr);
return 1;
}
if (!each->Check(outDir, outDir, includeDirs)) {
std::println(std::cerr, "Shader::Check returned false for {}", each->OutputName().string());
return 1;
}
}
// Malformed defines and variants are refused by name. Left to glslang, an
// empty macro name surfaces as "-1:<line>" with no file, and a variant
// with a separator as a failed open after the whole compile.
for (const char* bad : {"", "=1", "A\nB"}) {
Shader malformed(fs::path(variantSrc), "main", ShaderType::Fragment);
malformed.defines = {bad};
malformed.variant = "malformed";
if (!malformed.Compile(outDir, outDir, includeDirs).contains("invalid shader define")) {
std::println(std::cerr, "a malformed define was not rejected by name");
return 1;
}
}
Shader nested(fs::path(variantSrc), "main", ShaderType::Fragment);
nested.variant = "sub/red";
if (!nested.Compile(outDir, outDir, includeDirs).contains("path separator")) {
std::println(std::cerr, "a variant with a path separator was not rejected by name");
return 1;
}
auto readAll = [](const fs::path& file) {
std::ifstream stream(file, std::ios::binary);
return std::string(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
};
std::string plainSpv = readAll(outDir / "variant.spv");
std::string redSpv = readAll(outDir / "variant.red.spv");
if (plainSpv.empty() || redSpv.empty()) {
std::println(std::cerr, "a variant output is missing or empty");
return 1;
}
if (plainSpv == redSpv) {
std::println(std::cerr, "both variants compiled to the same SPIR-V: the defines did not reach the preprocessor");
return 1;
}
return 0; return 0;
} }

View file

@ -0,0 +1 @@
const vec4 kTint = vec4(0.0, 1.0, 0.0, 1.0);

View file

@ -0,0 +1,16 @@
#version 450
#extension GL_GOOGLE_include_directive : require
// Built by the ShaderIncludeChange test, which edits palette.glsl and this
// entry's defines between builds.
#include "palette.glsl"
layout(location = 0) out vec4 outColor;
void main() {
#ifdef LIT_DIM
outColor = kTint * 0.5;
#else
outColor = kTint;
#endif
}

View file

@ -0,0 +1,6 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
int main() {
return 0;
}

View file

@ -0,0 +1,8 @@
#version 450
// Includes nothing: the ShaderIncludeChange test checks it is left alone.
layout(location = 0) out vec4 outColor;
void main() {
outColor = vec4(0.0, 0.0, 1.0, 1.0);
}

View file

@ -0,0 +1,126 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A shader's output is stale when anything that went into it changed, not
// only its own source. Check() used to compare the .spv against the source's
// mtime alone, so:
// - editing a file the shader #includes left the .spv built against the old
// contents, the GLSL twin of IncrementalHeaderChange;
// - changing an entry's defines without touching the source did too.
// Each pass changes one thing and checks both directions: the shader it
// reaches is recompiled, to new contents, and the shader that includes
// nothing is left alone. A build with nothing changed compiles nothing.
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
// The fixture is edited between passes, so work on a copy outside the repo.
fs::path StageFixture() {
fs::path source = fs::current_path() / "tests" / "ShaderIncludeChange" / "fixture";
fs::path staged = fs::temp_directory_path() / "crafter-build-shader-include-change";
fs::remove_all(staged);
fs::copy(source, staged, fs::copy_options::recursive);
return staged;
}
Configuration MakeApp(const fs::path& staged) {
Configuration app;
app.path = staged;
app.name = "lit";
app.outputName = "lit";
app.target = HostTarget();
app.type = ConfigurationType::Executable;
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls = { "main" };
app.GetInterfacesAndImplementations(ifaces, impls);
app.shaders.emplace_back(staged / "lit.frag", std::string("main"), ShaderType::Fragment);
app.shaders.emplace_back(staged / "plain.frag", std::string("main"), ShaderType::Fragment);
// Puts include/ on the shaders' #include path.
app.buildFiles = { staged / "include" / "palette.glsl" };
return app;
}
bool BuildOk(Configuration& app, std::string_view label) {
// A fresh depResults per pass: the map memoizes a Configuration's
// build for the duration of one pass.
std::unordered_map<fs::path, std::shared_future<BuildResult>> depResults;
std::mutex depMutex;
BuildResult r = Build(app, depResults, depMutex);
Check(r.result.empty(), std::format("{} builds: {}", label, r.result));
return r.result.empty();
}
std::string ReadAll(const fs::path& file) {
std::ifstream stream(file, std::ios::binary);
return std::string(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
}
}
int main() {
const fs::path staged = StageFixture();
Configuration app = MakeApp(staged);
const fs::path lit = app.BinDir() / "lit.spv";
const fs::path plain = app.BinDir() / "plain.spv";
const fs::path palette = staged / "include" / "palette.glsl";
// Pass 1: everything from scratch.
if (!BuildOk(app, "the first pass")) return 1;
Check(fs::exists(lit) && fs::exists(plain), "both shaders are in the bin dir");
const fs::file_time_type plainBuilt = fs::last_write_time(plain);
const std::string green = ReadAll(lit);
// Pass 2: nothing changed.
fs::file_time_type litBuilt = fs::last_write_time(lit);
BuildOk(app, "an unchanged project");
Check(fs::last_write_time(lit) == litBuilt && fs::last_write_time(plain) == plainBuilt, "with nothing changed, no shader is recompiled");
// Pass 3: the included file changes. Dated just after the output it must
// outdate, so the rebuilt output is newer than it again.
{
std::ofstream out(palette, std::ios::trunc);
out << "const vec4 kTint = vec4(1.0, 0.0, 0.0, 1.0);\n";
}
fs::last_write_time(palette, litBuilt + std::chrono::milliseconds(1));
BuildOk(app, "an edited include");
const std::string red = ReadAll(lit);
Check(fs::last_write_time(lit) > litBuilt && red != green, "editing an included file recompiles the shader that includes it, to new contents");
Check(fs::last_write_time(plain) == plainBuilt, "and leaves the shader that includes nothing alone");
// Pass 4: the entry's defines change, no file does.
app.shaders[0].defines = { "LIT_DIM" };
litBuilt = fs::last_write_time(lit);
BuildOk(app, "a new define");
const std::string dim = ReadAll(lit);
Check(fs::last_write_time(lit) > litBuilt && dim != red, "a define added to the entry recompiles it, taking the #ifdef branch");
Check(fs::last_write_time(plain) == plainBuilt, "and leaves the other entry alone");
// Pass 5: and back.
app.shaders[0].defines = {};
BuildOk(app, "the define taken out");
Check(ReadAll(lit) == red, "taking the define out again recompiles it as it was");
// Pass 6: nothing changed again; what the passes above recorded holds.
litBuilt = fs::last_write_time(lit);
BuildOk(app, "an unchanged project, again");
Check(fs::last_write_time(lit) == litBuilt && fs::last_write_time(plain) == plainBuilt, "with nothing changed since, no shader is recompiled");
std::error_code ec;
fs::remove_all(staged, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,6 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
int main() {
return 0;
}

View file

@ -0,0 +1,12 @@
#version 450
// Built twice by the ShaderVariantBuild test: as is, and with TINT_RED defined.
layout(location = 0) out vec4 outColor;
void main() {
#if TINT_RED
outColor = vec4(1.0, 0.0, 0.0, 1.0);
#else
outColor = vec4(0.0, 1.0, 0.0, 1.0);
#endif
}

View file

@ -0,0 +1,121 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// One shader source can be registered more than once, each entry with its own
// defines and a variant that names its output ("tint.spv", "tint.red.spv").
// Build() compiles the entries in parallel into one directory, which is where
// the feature can go wrong in ways the Shader API alone does not show:
//
// - both entries must land in the bin dir, with different contents;
// - an entry added later must still be compiled when the other one's output
// is already there and up to date;
// - two entries that would write the same file (defines set, variant
// forgotten) used to race silently: the file held whichever compile
// finished last, the build reported success, and Check() called both up
// to date from then on. That has to be a build error naming the cause.
//
// The fixture's `#if TINT_RED` also pins that a bare define is defined to 1,
// like the compiler's -D: an empty definition would not compile.
namespace {
std::int32_t Failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++Failures;
}
}
// Shader outputs are compared by mtime against the source, and the test
// adds an entry between passes, so work on a fresh copy outside the repo.
fs::path StageFixture() {
fs::path source = fs::current_path() / "tests" / "ShaderVariantBuild" / "fixture";
fs::path staged = fs::temp_directory_path() / "crafter-build-shader-variant-build";
fs::remove_all(staged);
fs::copy(source, staged, fs::copy_options::recursive);
return staged;
}
Configuration MakeApp(const fs::path& staged) {
Configuration app;
app.path = staged;
app.name = "tinted";
app.outputName = "tinted";
app.target = HostTarget();
app.type = ConfigurationType::Executable;
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls = { "main" };
app.GetInterfacesAndImplementations(ifaces, impls);
return app;
}
Shader& AddTint(Configuration& app, const fs::path& staged) {
return app.shaders.emplace_back(staged / "tint.frag", std::string("main"), ShaderType::Fragment);
}
BuildResult BuildOnce(Configuration& app) {
// A fresh depResults per pass: the map memoizes a Configuration's
// build for the duration of one pass.
std::unordered_map<fs::path, std::shared_future<BuildResult>> depResults;
std::mutex depMutex;
return Build(app, depResults, depMutex);
}
std::string ReadAll(const fs::path& file) {
std::ifstream stream(file, std::ios::binary);
return std::string(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
}
}
int main() {
const fs::path staged = StageFixture();
// Pass 1: the plain entry alone.
Configuration app = MakeApp(staged);
AddTint(app, staged);
BuildResult first = BuildOnce(app);
Check(first.result.empty(), std::format("the plain entry builds: {}", first.result));
const fs::path plainSpv = app.BinDir() / "tint.spv";
const fs::path redSpv = app.BinDir() / "tint.red.spv";
Check(fs::exists(plainSpv), "tint.spv is in the bin dir");
Check(!fs::exists(redSpv), "no variant output before the variant is registered");
// Pass 2: the variant is added while tint.spv is already up to date. It
// has its own staleness check and must be compiled, into its own file.
Shader& red = AddTint(app, staged);
red.defines = {"TINT_RED"};
red.variant = "red";
BuildResult second = BuildOnce(app);
Check(second.result.empty(), std::format("plain + variant builds: {}", second.result));
Check(fs::exists(redSpv), "tint.red.spv is in the bin dir");
const std::string plainBytes = ReadAll(plainSpv);
const std::string redBytes = ReadAll(redSpv);
Check(!plainBytes.empty() && !redBytes.empty(), "both outputs have content");
Check(plainBytes != redBytes, "the variant took the other #if branch");
// Pass 3: the same define without a variant. Both entries would write
// tint.spv, so the build has to stop and say so, and must leave the file
// from pass 2 alone.
Configuration clash = MakeApp(staged);
AddTint(clash, staged);
Shader& unnamed = AddTint(clash, staged);
unnamed.defines = {"TINT_RED"};
BuildResult third = BuildOnce(clash);
Check(!third.result.empty(), "two entries with one output name fail the build");
Check(third.result.contains("tint.spv") && third.result.contains("variant"), std::format("the error names the file and the fix: {}", third.result));
Check(ReadAll(plainSpv) == plainBytes, "the existing output is not overwritten by the failed build");
std::error_code ec;
fs::remove_all(staged, ec);
if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,65 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// clang's driver ranks the host toolchain's own libc++ headers
// (<install>/../include/c++/v1) above --sysroot. A unit that only does
// `import std;` never notices, because the std module is precompiled with the
// sysroot's headers forced. A unit that includes a standard header textually
// (vendored C++ in a dependency, say) got the host's libc++ configured for the
// host's libc: against a musl sysroot every such unit died with "unknown rune
// table for this platform", and against a glibc sysroot it compiled against
// the wrong release without a word.
//
// GetCompileCommand therefore has to point every sysroot compile at the
// sysroot's copy. It must not do so where there is no sysroot, for wasm (which
// has its own sysroot handling) or for mingw (libstdc++, no libc++ at all).
// The command is checked as text: no compiler runs.
namespace {
std::int32_t Failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++Failures;
}
}
std::string CommandFor(std::string_view target, std::string_view sysroot) {
Configuration cfg;
cfg.path = "/tmp/crafter-build-sysroot-compile-flags";
cfg.name = "flags";
cfg.outputName = "flags";
cfg.target = std::string(target);
cfg.march = "generic";
cfg.mtune = "generic";
cfg.type = ConfigurationType::Executable;
cfg.sysroot = std::string(sysroot);
return GetCompileCommand(cfg).command;
}
}
int main() {
const std::string sysroot = "/opt/crafter-build-test-sysroot";
const std::string forced = std::format("-nostdinc++ -isystem {}/usr/include/c++/v1", sysroot);
for (std::string_view target : {"aarch64-alpine-linux-musl", "aarch64-unknown-linux-gnu", "x86_64-pc-linux-gnu"}) {
const std::string command = CommandFor(target, sysroot);
Check(command.contains(std::format("--sysroot={}", sysroot)), std::format("{}: the sysroot is passed", target));
Check(command.contains(forced), std::format("{}: the sysroot's libc++ headers are forced", target));
}
Check(!CommandFor("x86_64-pc-linux-gnu", "").contains("/usr/include/c++/v1 -Wno-unused-command-line-argument"), "no sysroot: nothing is forced");
Check(!CommandFor("wasm32-wasip1", sysroot).contains(forced), "wasm: left to its own sysroot handling");
Check(!CommandFor("x86_64-w64-mingw32", sysroot).contains(forced), "mingw: libstdc++ target, libc++ headers are not forced");
if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures);
return 1;
}
return 0;
}

View file

@ -0,0 +1,156 @@
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A sysroot build precompiles the std module from a module source that has to
// belong to the libc++ release whose headers the sysroot carries. Not every
// distro packages that source with its libc++ (Alpine does not), and the
// fallback used to be the toolchain's own copy whatever its release: with a
// libc++ 22 toolchain and a libc++ 23 sysroot the build died on a page of "no
// member named 'runtime_format' in namespace 'std'" that named neither
// release nor what to do about it.
//
// Staged, compiler-free layouts cover the order SysrootStdModule picks in: the
// sysroot's own source, the toolchain's for the same major release, the copy
// crafter-build ships for another release, and otherwise one line naming both
// releases and where the matching source belongs. The last block drives
// Build() against a sysroot nothing matches and expects that line back in
// place of compiler output.
namespace {
std::int32_t Failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++Failures;
}
}
void Write(const fs::path& file, std::string_view content) {
fs::create_directories(file.parent_path());
std::ofstream(file) << content;
}
// What libc++ installs as include/c++/v1/__config, reduced to the line
// that states the release.
void StageHeaders(const fs::path& includeDir, std::int32_t version) {
Write(includeDir / "__config", std::format("#ifndef _LIBCPP___CONFIG\n# define _LIBCPP___CONFIG\n# define _LIBCPP_VERSION {}\n#endif\n", version));
}
fs::path StageSysroot(const fs::path& sysroot, std::int32_t version) {
StageHeaders(sysroot / "usr" / "include" / "c++" / "v1", version);
return sysroot;
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-sysroot-std-module";
std::error_code ec;
fs::remove_all(scratch, ec);
fs::create_directories(scratch);
{
StageHeaders(scratch / "headers", 230102);
Check(LibcxxVersion(scratch / "headers") == 230102, "the release is read from __config");
Check(LibcxxVersion(scratch / "no-such-dir") == 0, "missing headers read as an unknown release");
Write(scratch / "other" / "__config", "#define _LIBCPP_ABI_VERSION 1\n");
Check(LibcxxVersion(scratch / "other") == 0, "a __config without the version line reads as an unknown release");
Check(LibcxxVersionString(230102) == "23.1.2" && LibcxxVersionString(220108) == "22.1.8", "XXYYZZ is spelled as a dotted release");
}
// A toolchain on libc++ 22.1.8 that also ships the sources for 23.
Toolchain tc;
tc.stdCppm = scratch / "toolchain" / "share" / "libc++" / "v1" / "std.cppm";
tc.libcxxVersion = 220108;
tc.stdModulesDir = scratch / "shipped";
Write(tc.stdCppm, "");
Write(tc.stdModulesDir / "23" / "std.cppm", "");
{
// The sysroot's own source matches its headers by construction, so
// it wins whatever the releases are.
const fs::path sysroot = StageSysroot(scratch / "sysroot-own", 240100);
const fs::path own = sysroot / "usr" / "share" / "libc++" / "v1" / "std.cppm";
Write(own, "");
StdModuleSource source = SysrootStdModule(tc, sysroot);
Check(source.error.empty() && source.stdCppm == own, "a sysroot that ships the module source uses its own");
}
{
const fs::path sysroot = StageSysroot(scratch / "sysroot-same-major", 220103);
StdModuleSource source = SysrootStdModule(tc, sysroot);
Check(source.error.empty() && source.stdCppm == tc.stdCppm, "a point release of the toolchain's libc++ uses the toolchain's source");
}
{
// The reported case: libc++ 23 headers, a libc++ 22 toolchain.
const fs::path sysroot = StageSysroot(scratch / "sysroot-next-major", 230102);
StdModuleSource source = SysrootStdModule(tc, sysroot);
Check(source.error.empty() && source.stdCppm == tc.stdModulesDir / "23" / "std.cppm", "another release uses the source shipped for it, not the toolchain's");
}
{
const fs::path sysroot = StageSysroot(scratch / "sysroot-unshipped", 210108);
StdModuleSource source = SysrootStdModule(tc, sysroot);
Check(source.stdCppm.empty() && !source.error.empty(), "a release nothing was shipped for is an error, not a compile against the wrong source");
Check(source.error.contains("21.1.8") && source.error.contains("22.1.8"), std::format("the error names both releases, got '{}'", source.error));
Check(source.error.contains((sysroot / "usr" / "share" / "libc++" / "v1").string()), std::format("the error says where the matching source belongs, got '{}'", source.error));
Check(!source.error.contains('\n'), "the error is one line");
}
{
const fs::path sysroot = scratch / "sysroot-no-libcxx";
fs::create_directories(sysroot / "usr" / "include");
StdModuleSource source = SysrootStdModule(tc, sysroot);
Check(source.stdCppm.empty() && source.error.contains("no libc++ headers"), std::format("a sysroot without libc++ says so, got '{}'", source.error));
}
{
// Nothing to compare when either release is unknown: the toolchain's
// source goes to the compiler, as it did before releases were read.
const fs::path sysroot = scratch / "sysroot-unreadable";
Write(sysroot / "usr" / "include" / "c++" / "v1" / "__config", "#define _LIBCPP_ABI_VERSION 1\n");
Check(SysrootStdModule(tc, sysroot).stdCppm == tc.stdCppm, "an unreadable sysroot release falls back to the toolchain's source");
Toolchain unknown = tc;
unknown.libcxxVersion = 0;
unknown.stdModulesDir.clear();
Check(SysrootStdModule(unknown, StageSysroot(scratch / "sysroot-vs-unknown", 230102)).stdCppm == tc.stdCppm, "an unreadable toolchain release falls back to the toolchain's source");
}
{
Toolchain bare;
bare.stdCppm = scratch / "nowhere" / "std.cppm";
bare.libcxxVersion = 220108;
StdModuleSource source = SysrootStdModule(bare, StageSysroot(scratch / "sysroot-vs-bare", 220108));
Check(source.stdCppm.empty() && source.error.contains("no std module source"), std::format("a toolchain without a module source is an error too, got '{}'", source.error));
}
{
// End to end with the real toolchain: a sysroot on a libc++ release
// that does not exist fails the build with the one-line explanation
// before clang is run. The target is the host's so the std PCM cache
// directory is the one every other test already uses.
const Toolchain& real = GetToolchain();
Check(real.libcxxVersion != 0, "the running toolchain's libc++ release is read from its headers");
const fs::path project = scratch / "project";
Write(project / "main.cpp", "int main() { return 0; }\n");
Configuration app;
app.path = project;
app.name = "sysroot-std-module";
app.outputName = "sysroot-std-module";
app.target = HostTarget();
app.type = ConfigurationType::Executable;
app.sysroot = StageSysroot(scratch / "sysroot-future", 990000).string();
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls = { "main" };
app.GetInterfacesAndImplementations(ifaces, impls);
std::unordered_map<fs::path, std::shared_future<BuildResult>> depResults;
std::mutex depMutex;
BuildResult r = Build(app, depResults, depMutex);
Check(r.result.contains("libc++ 99.0.0"), std::format("the build fails naming the sysroot's release, got '{}'", r.result));
Check(r.result.contains(LibcxxVersionString(real.libcxxVersion)), std::format("the build fails naming the toolchain's release, got '{}'", r.result));
Check(!r.result.contains('\n') && !r.result.contains("error:"), std::format("the failure is one line, not compiler output, got '{}'", r.result));
}
return Failures == 0 ? 0 : 1;
}

View file

@ -40,9 +40,11 @@ int main() {
// A prefix with everything crafter-build looks for beside the compiler, // A prefix with everything crafter-build looks for beside the compiler,
// and one with the compiler alone. // and one with the compiler alone.
const fs::path full = scratch / "full"; const fs::path full = scratch / "full";
for (const char* rel : {"bin/clang++", "bin/clang", "bin/llvm-ar", "lib/libclang.so", "include/clang-c/Index.h", "share/libc++/v1/std.cppm", "share/wasi-sysroot/include/.keep"}) { for (const char* rel : {"bin/clang++", "bin/clang", "bin/llvm-ar", "lib/libclang.so", "include/clang-c/Index.h", "include/c++/v1/__config", "share/libc++/v1/std.cppm", "share/wasi-sysroot/include/.keep"}) {
Touch(full / rel); Touch(full / rel);
} }
// The prefix's libc++ headers state its release.
std::ofstream(full / "include/c++/v1/__config") << "# define _LIBCPP_VERSION 230102\n";
const fs::path bare = scratch / "bare"; const fs::path bare = scratch / "bare";
Touch(bare / "bin/clang++"); Touch(bare / "bin/clang++");
@ -56,8 +58,10 @@ int main() {
Check(!tc.libclang.has_value(), "the PATH fallback searches libclang by name"); Check(!tc.libclang.has_value(), "the PATH fallback searches libclang by name");
Check(tc.llvmIncludeDir.empty(), "the PATH fallback adds no include dir"); Check(tc.llvmIncludeDir.empty(), "the PATH fallback adds no include dir");
Check(tc.stdCppm == "/usr/share/libc++/v1/std.cppm", "the system std module source is the default"); Check(tc.stdCppm == "/usr/share/libc++/v1/std.cppm", "the system std module source is the default");
Check(tc.libcxxVersion == LibcxxVersion("/usr/include/c++/v1"), "the system libc++ headers state the release of the system module source");
Check(tc.stdModulesDir.empty(), "no crafter-build prefix, no shipped module sources");
Check(tc.wasiSysroot == "/usr/share/wasi-sysroot", "the system wasi sysroot is the default"); Check(tc.wasiSysroot == "/usr/share/wasi-sysroot", "the system wasi sysroot is the default");
Check(tc.mingwSysroot == "/usr/x86_64-w64-mingw32" && !tc.mingwSysrootExplicit, "the default mingw dir is not passed as --sysroot"); Check(tc.mingwSysroot.empty(), "outside a sandbox, clang finds mingw-w64 itself: no --sysroot");
Check(tc.LibcxxCompileFlags().empty() && tc.LibcxxLinkFlags().empty(), "the PATH fallback adds no libc++ flags"); Check(tc.LibcxxCompileFlags().empty() && tc.LibcxxLinkFlags().empty(), "the PATH fallback adds no libc++ flags");
} }
{ {
@ -71,6 +75,8 @@ int main() {
Check(tc.libclang == full / "lib/libclang.so", "libclang beside the compiler is pinned"); Check(tc.libclang == full / "lib/libclang.so", "libclang beside the compiler is pinned");
Check(tc.llvmIncludeDir == full / "include", "clang-c headers beside the compiler are added to the include path"); Check(tc.llvmIncludeDir == full / "include", "clang-c headers beside the compiler are added to the include path");
Check(tc.stdCppm == full / "share/libc++/v1/std.cppm", "the prefix's std module source is used"); Check(tc.stdCppm == full / "share/libc++/v1/std.cppm", "the prefix's std module source is used");
Check(tc.libcxxVersion == 230102, "the prefix's libc++ headers state its release");
Check(tc.stdModulesDir == full / "share/crafter-build/libc++-modules", "module sources for other releases are looked up in crafter-build's own share directory");
Check(tc.wasiSysroot == full / "share/wasi-sysroot", "the prefix's wasi sysroot is used"); Check(tc.wasiSysroot == full / "share/wasi-sysroot", "the prefix's wasi sysroot is used");
Check(tc.LibcxxCompileFlags().empty(), "the compiler finds its own libc++ headers without flags"); Check(tc.LibcxxCompileFlags().empty(), "the compiler finds its own libc++ headers without flags");
Check(Contains(tc.LibcxxLinkFlags(), std::format("-L{}", (full / "lib").string())) && Contains(tc.LibcxxLinkFlags(), std::format("-Wl,-rpath,{}", (full / "lib").string())), "binaries get -L and an rpath for the prefix's lib dir"); Check(Contains(tc.LibcxxLinkFlags(), std::format("-L{}", (full / "lib").string())) && Contains(tc.LibcxxLinkFlags(), std::format("-Wl,-rpath,{}", (full / "lib").string())), "binaries get -L and an rpath for the prefix's lib dir");
@ -120,12 +126,15 @@ int main() {
// An explicit libc++ install: headers, library and std module all // An explicit libc++ install: headers, library and std module all
// come from it, whatever the compiler would have used. // come from it, whatever the compiler would have used.
const fs::path libcxx = scratch / "libcxx"; const fs::path libcxx = scratch / "libcxx";
fs::create_directories(libcxx / "include/c++/v1");
std::ofstream(libcxx / "include/c++/v1/__config") << "# define _LIBCPP_VERSION 210108\n";
ToolchainInputs in; ToolchainInputs in;
in.crafterPrefix = full; in.crafterPrefix = full;
in.env["CRAFTER_BUILD_LIBCXX_DIR"] = libcxx.string(); in.env["CRAFTER_BUILD_LIBCXX_DIR"] = libcxx.string();
Toolchain tc = ResolveToolchain(in); Toolchain tc = ResolveToolchain(in);
Check(tc.libcxxDir == libcxx, "CRAFTER_BUILD_LIBCXX_DIR is recorded"); Check(tc.libcxxDir == libcxx, "CRAFTER_BUILD_LIBCXX_DIR is recorded");
Check(tc.stdCppm == libcxx / "share/libc++/v1/std.cppm", "the std module source follows the explicit libc++"); Check(tc.stdCppm == libcxx / "share/libc++/v1/std.cppm", "the std module source follows the explicit libc++");
Check(tc.libcxxVersion == 210108, "the release is read from the explicit libc++, not the compiler's");
Check(Contains(tc.LibcxxCompileFlags(), std::format("-nostdinc++ -isystem {}", (libcxx / "include/c++/v1").string())), "compiles use the explicit libc++ headers"); Check(Contains(tc.LibcxxCompileFlags(), std::format("-nostdinc++ -isystem {}", (libcxx / "include/c++/v1").string())), "compiles use the explicit libc++ headers");
Check(Contains(tc.LibcxxLinkFlags(), std::format("-L{}", (libcxx / "lib").string())) && Contains(tc.LibcxxLinkFlags(), std::format("-Wl,-rpath,{}", (libcxx / "lib").string())), "links use the explicit libc++ library dir, with an rpath"); Check(Contains(tc.LibcxxLinkFlags(), std::format("-L{}", (libcxx / "lib").string())) && Contains(tc.LibcxxLinkFlags(), std::format("-Wl,-rpath,{}", (libcxx / "lib").string())), "links use the explicit libc++ library dir, with an rpath");
} }
@ -135,7 +144,21 @@ int main() {
in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw"; in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw";
Toolchain tc = ResolveToolchain(in); Toolchain tc = ResolveToolchain(in);
Check(tc.wasiSysroot == "/opt/wasi", "CRAFTER_BUILD_WASI_SYSROOT overrides the wasi sysroot"); Check(tc.wasiSysroot == "/opt/wasi", "CRAFTER_BUILD_WASI_SYSROOT overrides the wasi sysroot");
Check(tc.mingwSysroot == "/opt/mingw" && tc.mingwSysrootExplicit, "CRAFTER_MINGW_DIR overrides the mingw dir and is passed as --sysroot"); Check(tc.mingwSysroot == "/opt/mingw", "CRAFTER_MINGW_DIR names the mingw prefix passed as --sysroot");
}
{
// Sandboxed, the host's /usr stands in for the missing mingw-w64,
// unless CRAFTER_MINGW_DIR says otherwise.
const fs::path host = fs::temp_directory_path() / "crafter-build-toolchain-host";
fs::remove_all(host);
ToolchainInputs in;
in.hostRoot = host;
Check(ResolveToolchain(in).mingwSysroot.empty(), "a host without mingw-w64 gives no --sysroot");
fs::create_directories(host / "usr" / "x86_64-w64-mingw32");
Check(ResolveToolchain(in).mingwSysroot == host / "usr", "a host with mingw-w64 gives its /usr as the mingw prefix");
in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw";
Check(ResolveToolchain(in).mingwSysroot == "/opt/mingw", "CRAFTER_MINGW_DIR wins over the host's");
fs::remove_all(host);
} }
{ {
// The cache key separates toolchains that would build different // The cache key separates toolchains that would build different

View file

@ -133,6 +133,33 @@ int main() {
Check(a.BuildDir() != b.BuildDir(), "different VariantId yields different BuildDir"); Check(a.BuildDir() != b.BuildDir(), "different VariantId yields different BuildDir");
Check(a.BinDir() != b.BinDir(), "different VariantId yields different BinDir"); Check(a.BinDir() != b.BinDir(), "different VariantId yields different BinDir");
} }
{
// The compiler is compile state as much as any flag: a PCM can only
// be read by the clang release that wrote it, and nothing in a
// project's sources changes when the compiler does. With the
// toolchain left out of the id, a project built with clang 22 kept
// its PCMs after crafter-build moved to clang 23; the build after
// the update found nothing stale, and the first edit failed with
// "module file ... uses an older format that is no longer supported"
// until the project was cleaned by hand.
Configuration cfg = MakeBase();
Toolchain older;
older.clangxx = "/app/bin/clang++";
older.clangVersion = "clang version 22.1.8";
Toolchain newer = older;
newer.clangVersion = "clang version 23.1.2";
Check(cfg.VariantId(older) != cfg.VariantId(newer), "a compiler update at the same path perturbs VariantId");
Check(cfg.VariantId(older) == cfg.VariantId(older), "VariantId under one toolchain is deterministic");
// Two installs of one release are still two toolchains: their
// libc++ need not be the same build.
Toolchain elsewhere = older;
elsewhere.clangxx = "/usr/bin/clang++";
Check(cfg.VariantId(older) != cfg.VariantId(elsewhere), "the same release installed elsewhere perturbs VariantId");
Check(cfg.VariantId() == cfg.VariantId(GetToolchain()), "VariantId() is the variant under the running toolchain");
Check(cfg.BuildDir().filename() == cfg.VariantId(GetToolchain()), "BuildDir is named for the running toolchain's variant");
}
if (Failures > 0) { if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures); std::println(std::cerr, "{} assertions failed", Failures);