Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 3 Pull requests Projects Releases 4 Packages Wiki Activity Actions

feat(packaging): run flatpak and run install for a local Flatpak

Browse source 
The local Flatpak build was one long flatpak-builder line in AGENTS.md,
with its state wherever the caller happened to stand, and installing
the result was not written down anywhere. It is not a rare chore: the
development box compiles with the distro's clang and the Flatpak with
its bundled LLVM, a major version apart at the moment, so anything
toolchain-sensitive has to be tried in the Flatpak too.

Both are now scripts in project.cpp, the first users of cfg.scripts.

`run flatpak` builds the manifest as it is, from the working tree, and
ends with `crafter-build --version` from inside the result. Its state
is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and
not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in
the sandbox, so it does not depend on the host variant, and a
variant-keyed directory would strand the LLVM cache on every host
compiler update. The manifest already skips build/ when it copies the
sources. flatpak-builder keys its cache on the manifest's path, so the
manifest is never rewritten into a copy the way CI stamps the version
in, and a local build reports 0.0.0-dev. The only argument accepted is
--jobs=<N>: anything that ends the run without a build would leave the
smoke test checking the previous one.

`run install` installs the last export as branch `local` from a user
remote of its own and makes it current, leaving the published branch
installed. It does not build first: flatpak-builder never caches a
`type: dir` module, so every build reruns build.sh and the suite.
remote-add --if-not-exists is followed by remote-modify --url, because
the remote may exist and point at another checkout.

Both refuse to run from the Flatpak build of crafter-build, which has
no flatpak to drive.

The smoke test goes through `flatpak build` and not `<builder> --run`,
which is what CI uses. flatpak-builder 1.4.9 run as the
org.flatpak.Builder app exits 1 when the command it ran succeeded and 0
when it failed (`true` gives 1, `exit 3` gives 0), so the script
reported a good build as a failure.

Measured on the 56-core box with LLVM and the WASI sysroot cached: 260
seconds, the crafter-build module only, 29 tests passing in the sandbox.
A second `run install` after a rebuild moved the installed commit.

`clean` now also drops that state, about 2 GB and a cold LLVM build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-06 18:29:32 +02:00
commit 2f8f25fd39
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
2 changed files with 145 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

132
project.cpp
View file

@ -8,6 +8,134 @@ import Crafter.LintRules;
namespace fs = std::filesystem;
using namespace Crafter;
// The local Flatpak: `crafter-build run flatpak` builds it from this checkout
// and `crafter-build run install` installs that build. The development box
// compiles with its own clang, the shipped artifact with the LLVM bundled in
// the Flatpak, so this is how a change is tried against the real thing.
namespace {
struct FlatpakPaths {
fs::path manifest;
// flatpak-builder's downloads, ccache and per-module cache. The
// LLVM build lives here: about 2 GB, and 25 minutes to rebuild.
fs::path state;
// The OSTree repo the build is exported to; `install` reads it.
fs::path repo;
fs::path buildDir;
};
// Everything under <project>/build/flatpak, and a fixed name on purpose
// instead of a BuildDir(): the Flatpak is compiled by the toolchain
// inside the sandbox, so it does not depend on the host variant, and a
// variant-keyed directory would strand the LLVM cache on every host
// compiler update. The manifest's `skip: build` keeps it out of the
// sources the sandbox copies. flatpak-builder keys its cache on the
// manifest's absolute path, so the manifest is used where it is, never
// through a rewritten copy the way CI stamps the version in; a local
// build therefore reports version 0.0.0-dev.
FlatpakPaths FlatpakLayout(const fs::path& projectDir) {
const fs::path root = projectDir / "build" / "flatpak";
return {
.manifest = projectDir / "packaging" / "flatpak" / "net.catcrafts.CrafterBuild.yaml",
.state = root / "state",
.repo = root / "repo",
.buildDir = root / "build-dir",
};
}
// Both scripts drive the host's flatpak, and the sandbox crafter-build
// itself runs in as a Flatpak has none.
bool RefusedInSandbox(std::string_view script) {
if (!fs::exists("/.flatpak-info")) return false;
std::println(std::cerr, "run {0}: needs the host's flatpak, which the Flatpak build of crafter-build cannot reach. Use a native build: ./build.sh, then ./bin/crafter-build run {0}", script);
return true;
}
// flatpak-builder as a host package, else the org.flatpak.Builder app.
// Empty when neither is installed.
std::string FlatpakBuilder() {
if (RunCommandWithTimeout("flatpak-builder --version", std::chrono::seconds(10)).exitCode == 0) return "flatpak-builder";
if (RunCommandWithTimeout("flatpak info org.flatpak.Builder", std::chrono::seconds(10)).exitCode == 0) return "flatpak run org.flatpak.Builder";
return "";
}
// A repo path as the file:// URL of a remote. Percent-encoded, because
// `remote-modify --url=` stores what it is given as it is.
std::string FileUrl(const fs::path& path) {
std::string url = "file://";
for (const char c : path.string()) {
const bool plain = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || std::string_view("/-._~").contains(c);
if (plain) url.push_back(c);
else url += std::format("%{:02X}", static_cast<std::uint8_t>(c));
}
return url;
}
std::int32_t BuildFlatpak(ScriptContext& ctx) {
if (RefusedInSandbox("flatpak")) return 1;
// Nothing else is passed through: a flag such as --stop-at or
// --download-only would end the run without a build and leave the
// smoke test below running against the previous one.
std::string jobs;
for (const std::string& arg : ctx.args) {
const std::string_view count = arg.starts_with("--jobs=") ? std::string_view(arg).substr(7) : std::string_view();
if (count.empty() || !std::ranges::all_of(count, [](char c) { return c >= '0' && c <= '9'; })) {
std::println(std::cerr, "run flatpak: unknown argument '{}' (only --jobs=<N> is accepted)", arg);
return 1;
}
jobs = std::format(" --jobs={}", count);
}
const std::string builder = FlatpakBuilder();
if (builder.empty()) {
std::println(std::cerr, "run flatpak: neither flatpak-builder nor the org.flatpak.Builder app is installed (flatpak install flathub org.flatpak.Builder)");
return 1;
}
const FlatpakPaths paths = FlatpakLayout(ctx.projectDir);
const std::string buildDir = ShellQuote(paths.buildDir.string());
if (const std::int32_t code = ctx.Run(std::format("{}{} --user --force-clean --ccache --state-dir={} --repo={} --default-branch=local {} {}", builder, jobs, ShellQuote(paths.state.string()), ShellQuote(paths.repo.string()), buildDir, ShellQuote(paths.manifest.string()))); code != 0) return code;
// The smoke test CI runs on its build, but through `flatpak build`
// and not `<builder> --run`: the org.flatpak.Builder app (1.4.9)
// exits 1 when the command it ran succeeded and 0 when it failed,
// and --run also wants the state dir again or it drops a
// .flatpak-builder/ into the working directory.
return ctx.Run(std::format("flatpak build {} crafter-build --version", buildDir));
}
std::int32_t InstallFlatpak(ScriptContext& ctx) {
if (RefusedInSandbox("install")) return 1;
if (!ctx.args.empty()) {
std::println(std::cerr, "run install: takes no arguments (got '{}')", ctx.args.front());
return 1;
}
const FlatpakPaths paths = FlatpakLayout(ctx.projectDir);
// Deliberately not built from here: flatpak-builder never caches a
// `type: dir` module, so every build reruns build.sh and the whole
// suite inside the sandbox. Installing is the cheap half.
if (!fs::exists(paths.repo / "config")) {
std::println(std::cerr, "run install: nothing built yet, run: crafter-build run flatpak");
return 1;
}
const std::string url = ShellQuote(FileUrl(paths.repo));
// Its own branch from its own remote, next to whichever published
// branch is installed. The remote may already exist and point at
// another checkout's repo; --if-not-exists would keep that silently.
if (const std::int32_t code = ctx.Run(std::format("flatpak remote-add --user --if-not-exists --no-gpg-verify crafter-build-local {}", url)); code != 0) return code;
if (const std::int32_t code = ctx.Run(std::format("flatpak remote-modify --user --url={} crafter-build-local", url)); code != 0) return code;
if (const std::int32_t code = ctx.Run("flatpak install --user --noninteractive --or-update crafter-build-local net.catcrafts.CrafterBuild//local"); code != 0) return code;
// Only a first install becomes the branch `flatpak run` starts; an
// update leaves that choice where it was.
if (const std::int32_t code = ctx.Run("flatpak make-current --user net.catcrafts.CrafterBuild local"); code != 0) return code;
std::println(
R"(
`flatpak run net.catcrafts.CrafterBuild` now starts this build.
Back to a published branch, keeping this one installed:
flatpak make-current --user net.catcrafts.CrafterBuild <branch>
Remove it (do this before `crafter-build clean`, which deletes the repo the remote points at):
flatpak uninstall --user net.catcrafts.CrafterBuild//local
flatpak remote-delete --user crafter-build-local)");
return 0;
}
}
extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
std::vector<std::string> depArgs(args.begin(), args.end());
Configuration* math = GitProject({
@ -163,5 +291,9 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
// rules also auto-fix under `crafter-build format`.
LintRules::AddRules(cfg);
// See the top of this file.
cfg.scripts.push_back({"flatpak", BuildFlatpak});
cfg.scripts.push_back({"install", InstallFlatpak});
return cfg;
}