From 2f8f25fd39a35b0086388127dfb4c70ac51a7e3c Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Tue, 6 Oct 2026 18:29:32 +0200 Subject: [PATCH] feat(packaging): `run flatpak` and `run install` for a local Flatpak The local Flatpak build was one long flatpak-builder line in AGENTS.md, with its state wherever the caller happened to stand, and installing the result was not written down anywhere. It is not a rare chore: the development box compiles with the distro's clang and the Flatpak with its bundled LLVM, a major version apart at the moment, so anything toolchain-sensitive has to be tried in the Flatpak too. Both are now scripts in project.cpp, the first users of cfg.scripts. `run flatpak` builds the manifest as it is, from the working tree, and ends with `crafter-build --version` from inside the result. Its state is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in the sandbox, so it does not depend on the host variant, and a variant-keyed directory would strand the LLVM cache on every host compiler update. The manifest already skips build/ when it copies the sources. flatpak-builder keys its cache on the manifest's path, so the manifest is never rewritten into a copy the way CI stamps the version in, and a local build reports 0.0.0-dev. The only argument accepted is --jobs=: anything that ends the run without a build would leave the smoke test checking the previous one. `run install` installs the last export as branch `local` from a user remote of its own and makes it current, leaving the published branch installed. It does not build first: flatpak-builder never caches a `type: dir` module, so every build reruns build.sh and the suite. remote-add --if-not-exists is followed by remote-modify --url, because the remote may exist and point at another checkout. Both refuse to run from the Flatpak build of crafter-build, which has no flatpak to drive. The smoke test goes through `flatpak build` and not ` --run`, which is what CI uses. flatpak-builder 1.4.9 run as the org.flatpak.Builder app exits 1 when the command it ran succeeded and 0 when it failed (`true` gives 1, `exit 3` gives 0), so the script reported a good build as a failure. Measured on the 56-core box with LLVM and the WASI sysroot cached: 260 seconds, the crafter-build module only, 29 tests passing in the sandbox. A second `run install` after a rebuild moved the installed commit. `clean` now also drops that state, about 2 GB and a cold LLVM build. Co-Authored-By: Claude Fable 5.1 --- AGENTS.md | 16 +++++-- project.cpp | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7451d75..ff93b96 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj |---|---| | `interfaces/Crafter.Build-.cppm` | Module interface partitions. `Crafter.Build.cppm` is the umbrella that re-exports them. | | `implementations/Crafter.Build-.cpp` | Implementation units, one per partition, named `Crafter.Build:_impl`. `main.cpp` is a thin wrapper around `Crafter::Run`. | -| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations and the lint rules. | +| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations, the lint rules and the `flatpak` and `install` scripts. | | `tests//main.cpp` | One self-test per directory; `tests//fixture/` holds the sample project it builds. | | `examples/` | Standalone user-facing projects, MIT licensed. | | `build.sh` | Bootstrap script. It compiles everything by hand, without crafter-build. | @@ -48,7 +48,7 @@ Three things to know about this loop: - Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. - A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol. -`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. +`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. ## Build, test and style commands @@ -164,7 +164,17 @@ gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armo printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id ``` -To release: add a `` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. Local builds: `flatpak-builder --user --force-clean --ccache --repo=repo build-dir packaging/flatpak/net.catcrafts.CrafterBuild.yaml`, then `flatpak-builder --run build-dir crafter-build --version`. +To release: add a `` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. +Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive. + +```bash +./bin/crafter-build run flatpak [--jobs=] # build the Flatpak, then print its version from inside the sandbox +./bin/crafter-build run install # install that build as branch `local` and make it current +``` + +`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart. + +`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild ` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone. ## Gotchas diff --git a/project.cpp b/project.cpp index a281e35..ea20bf5 100644 --- a/project.cpp +++ b/project.cpp @@ -8,6 +8,134 @@ import Crafter.LintRules; namespace fs = std::filesystem; using namespace Crafter; +// The local Flatpak: `crafter-build run flatpak` builds it from this checkout +// and `crafter-build run install` installs that build. The development box +// compiles with its own clang, the shipped artifact with the LLVM bundled in +// the Flatpak, so this is how a change is tried against the real thing. +namespace { + struct FlatpakPaths { + fs::path manifest; + // flatpak-builder's downloads, ccache and per-module cache. The + // LLVM build lives here: about 2 GB, and 25 minutes to rebuild. + fs::path state; + // The OSTree repo the build is exported to; `install` reads it. + fs::path repo; + fs::path buildDir; + }; + + // Everything under /build/flatpak, and a fixed name on purpose + // instead of a BuildDir(): the Flatpak is compiled by the toolchain + // inside the sandbox, so it does not depend on the host variant, and a + // variant-keyed directory would strand the LLVM cache on every host + // compiler update. The manifest's `skip: build` keeps it out of the + // sources the sandbox copies. flatpak-builder keys its cache on the + // manifest's absolute path, so the manifest is used where it is, never + // through a rewritten copy the way CI stamps the version in; a local + // build therefore reports version 0.0.0-dev. + FlatpakPaths FlatpakLayout(const fs::path& projectDir) { + const fs::path root = projectDir / "build" / "flatpak"; + return { + .manifest = projectDir / "packaging" / "flatpak" / "net.catcrafts.CrafterBuild.yaml", + .state = root / "state", + .repo = root / "repo", + .buildDir = root / "build-dir", + }; + } + + // Both scripts drive the host's flatpak, and the sandbox crafter-build + // itself runs in as a Flatpak has none. + bool RefusedInSandbox(std::string_view script) { + if (!fs::exists("/.flatpak-info")) return false; + std::println(std::cerr, "run {0}: needs the host's flatpak, which the Flatpak build of crafter-build cannot reach. Use a native build: ./build.sh, then ./bin/crafter-build run {0}", script); + return true; + } + + // flatpak-builder as a host package, else the org.flatpak.Builder app. + // Empty when neither is installed. + std::string FlatpakBuilder() { + if (RunCommandWithTimeout("flatpak-builder --version", std::chrono::seconds(10)).exitCode == 0) return "flatpak-builder"; + if (RunCommandWithTimeout("flatpak info org.flatpak.Builder", std::chrono::seconds(10)).exitCode == 0) return "flatpak run org.flatpak.Builder"; + return ""; + } + + // A repo path as the file:// URL of a remote. Percent-encoded, because + // `remote-modify --url=` stores what it is given as it is. + std::string FileUrl(const fs::path& path) { + std::string url = "file://"; + for (const char c : path.string()) { + const bool plain = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || std::string_view("/-._~").contains(c); + if (plain) url.push_back(c); + else url += std::format("%{:02X}", static_cast(c)); + } + return url; + } + + std::int32_t BuildFlatpak(ScriptContext& ctx) { + if (RefusedInSandbox("flatpak")) return 1; + // Nothing else is passed through: a flag such as --stop-at or + // --download-only would end the run without a build and leave the + // smoke test below running against the previous one. + std::string jobs; + for (const std::string& arg : ctx.args) { + const std::string_view count = arg.starts_with("--jobs=") ? std::string_view(arg).substr(7) : std::string_view(); + if (count.empty() || !std::ranges::all_of(count, [](char c) { return c >= '0' && c <= '9'; })) { + std::println(std::cerr, "run flatpak: unknown argument '{}' (only --jobs= is accepted)", arg); + return 1; + } + jobs = std::format(" --jobs={}", count); + } + const std::string builder = FlatpakBuilder(); + if (builder.empty()) { + std::println(std::cerr, "run flatpak: neither flatpak-builder nor the org.flatpak.Builder app is installed (flatpak install flathub org.flatpak.Builder)"); + return 1; + } + const FlatpakPaths paths = FlatpakLayout(ctx.projectDir); + const std::string buildDir = ShellQuote(paths.buildDir.string()); + if (const std::int32_t code = ctx.Run(std::format("{}{} --user --force-clean --ccache --state-dir={} --repo={} --default-branch=local {} {}", builder, jobs, ShellQuote(paths.state.string()), ShellQuote(paths.repo.string()), buildDir, ShellQuote(paths.manifest.string()))); code != 0) return code; + // The smoke test CI runs on its build, but through `flatpak build` + // and not ` --run`: the org.flatpak.Builder app (1.4.9) + // exits 1 when the command it ran succeeded and 0 when it failed, + // and --run also wants the state dir again or it drops a + // .flatpak-builder/ into the working directory. + return ctx.Run(std::format("flatpak build {} crafter-build --version", buildDir)); + } + + std::int32_t InstallFlatpak(ScriptContext& ctx) { + if (RefusedInSandbox("install")) return 1; + if (!ctx.args.empty()) { + std::println(std::cerr, "run install: takes no arguments (got '{}')", ctx.args.front()); + return 1; + } + const FlatpakPaths paths = FlatpakLayout(ctx.projectDir); + // Deliberately not built from here: flatpak-builder never caches a + // `type: dir` module, so every build reruns build.sh and the whole + // suite inside the sandbox. Installing is the cheap half. + if (!fs::exists(paths.repo / "config")) { + std::println(std::cerr, "run install: nothing built yet, run: crafter-build run flatpak"); + return 1; + } + const std::string url = ShellQuote(FileUrl(paths.repo)); + // Its own branch from its own remote, next to whichever published + // branch is installed. The remote may already exist and point at + // another checkout's repo; --if-not-exists would keep that silently. + if (const std::int32_t code = ctx.Run(std::format("flatpak remote-add --user --if-not-exists --no-gpg-verify crafter-build-local {}", url)); code != 0) return code; + if (const std::int32_t code = ctx.Run(std::format("flatpak remote-modify --user --url={} crafter-build-local", url)); code != 0) return code; + if (const std::int32_t code = ctx.Run("flatpak install --user --noninteractive --or-update crafter-build-local net.catcrafts.CrafterBuild//local"); code != 0) return code; + // Only a first install becomes the branch `flatpak run` starts; an + // update leaves that choice where it was. + if (const std::int32_t code = ctx.Run("flatpak make-current --user net.catcrafts.CrafterBuild local"); code != 0) return code; + std::println( +R"( +`flatpak run net.catcrafts.CrafterBuild` now starts this build. +Back to a published branch, keeping this one installed: + flatpak make-current --user net.catcrafts.CrafterBuild +Remove it (do this before `crafter-build clean`, which deletes the repo the remote points at): + flatpak uninstall --user net.catcrafts.CrafterBuild//local + flatpak remote-delete --user crafter-build-local)"); + return 0; + } +} + extern "C" Configuration CrafterBuildProject(std::span args) { std::vector depArgs(args.begin(), args.end()); Configuration* math = GitProject({ @@ -163,5 +291,9 @@ extern "C" Configuration CrafterBuildProject(std::span a // rules also auto-fix under `crafter-build format`. LintRules::AddRules(cfg); + // See the top of this file. + cfg.scripts.push_back({"flatpak", BuildFlatpak}); + cfg.scripts.push_back({"install", InstallFlatpak}); + return cfg; }