Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 3 Pull requests Projects Releases 4 Packages Wiki Activity Actions

feat(ci): build and publish the Flatpak

Browse source 
One job replaces the tarball pipeline. On a pull request it builds the
Flatpak with flatpak-builder, which runs the bootstrap, the self-host
build and the test suite inside the sandbox, and smoke-tests the result
with `crafter-build --version` and the hello-world example. On a push to
master it also bundles the build, promotes it into the live repo as the
`beta` branch and refreshes the rolling `latest` prerelease. On a tag
X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's
newest release is that version, and publishes a release whose notes are
that entry's description.

The job runs on the `flatpak` runner label: the privileged forgejo-runner
on the Hetzner box, with the Sdk install, the flatpak-builder state and
the repo's web root as volumes. The state volume is what keeps a normal
run short: flatpak-builder rebuilds the LLVM and WASI modules only when
their sources change. The live repo is canonical and never built into;
builds are promoted with build-commit-from, then build-update-repo signs
and prunes it and generates static deltas.

The musl job, the per-march tarballs, the toolchain packages the runner
used to install and the PKGBUILD go: the Flatpak is the one artifact.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-02 22:36:55 +02:00
commit ccfd607870
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
3 changed files with 152 additions and 225 deletions
Download patch file Download diff file Expand all files Collapse all files

326
.forgejo/workflows/ci.yaml
View file

@ -1,229 +1,185 @@
name: CI
name: Flatpak
# One job, three outcomes:
# pull request -> build the Flatpak and run the suite inside it
# push master -> also publish the result as the `beta` branch of the
# Catcrafts repo and refresh the rolling `latest` release
# tag X.Y.Z -> publish as `stable` and create release X.Y.Z
#
# The `flatpak` runner label is the privileged runner on the Hetzner box
# (bubblewrap needs it) with three volumes allowed: the Sdk install, the
# flatpak-builder state (ccache and the LLVM module, so only crafter-build
# itself rebuilds on a normal run) and the published repo's web root.
on:
pull_request:
branches: [master]
push:
branches: [master]
tags: ['[0-9]+.[0-9]+.[0-9]+']
workflow_dispatch:
env:
APP_ID: net.catcrafts.CrafterBuild
MANIFEST: packaging/flatpak/net.catcrafts.CrafterBuild.yaml
CI_MANIFEST: packaging/flatpak/ci-manifest.yaml
RUNTIME: org.freedesktop.Sdk//26.08
REPO_DIR: /srv/flatpak/repo
WEB_DIR: /srv/flatpak
jobs:
build-test-release:
runs-on: arch-latest
env:
# x86-64-v2 = SSE4.2 baseline; runs on the CI SBC (Intel N5105 / Tremont,
# no AVX) and on essentially every x86_64 CPU since ~2011. The bootstrap
# MUST stay v2 — anything higher would emit instructions the SBC can't
# execute and the bootstrap binary itself would crash before it could
# rebuild for higher levels. v3/v4 variants are produced from this v2
# bootstrap by re-running it with CRAFTER_BUILD_MARCH overridden.
CRAFTER_BUILD_MARCH: x86-64-v2
CRAFTER_BUILD_MTUNE: generic
flatpak:
runs-on: flatpak
container:
image: archlinux:latest
options: >-
-v crafter-flatpak-system:/var/lib/flatpak
-v crafter-flatpak-builder:/root/.cache/flatpak-builder
-v /var/www/flatpak:/srv/flatpak
steps:
- name: Install build dependencies
- name: Install tools
run: |
# The slim archlinux:latest image arrives without a populated
# pacman keyring AND without a local pacman master key, so:
# 1. --init generates the local signing key (needed when pacman
# itself rewrites the keyring during package upgrades)
# 2. --populate archlinux imports the upstream master keys so
# currently-shipping signatures verify
# After that we can refresh archlinux-keyring to pick up keys for
# packagers added after the image's snapshot, then -Syu the rest.
pacman-key --init
pacman-key --populate archlinux
pacman -Sy --noconfirm --needed archlinux-keyring
pacman -Syu --noconfirm --needed \
base-devel git zip tar jq \
clang llvm lld libc++ cmake \
mingw-w64-gcc \
wasi-libc wasi-libc++ wasi-libc++abi wasi-compiler-rt \
nodejs
# The container runs as root and the workspace may be owned by a
# different uid; tell git not to refuse operations on it.
pacman -Syu --noconfirm --needed flatpak flatpak-builder ccache git nodejs jq gnupg go-yq ostree
git config --global --add safe.directory '*'
- name: Sdk
# A no-op once the volume is warm; `update` follows point releases.
run: |
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
flatpak install -y --noninteractive flathub "$RUNTIME"
flatpak update -y --noninteractive
- name: Checkout
uses: actions/checkout@v4
with:
# Persist the auth token so the 'Update rolling latest tag' step
# below can push the tag back via the implicit GITHUB_TOKEN.
fetch-depth: 0
persist-credentials: true
- name: Cache glslang clone+build
# Cache covers crafter-build's external dep cache. The first run builds
# glslang; the cache makes subsequent runs reuse that build.
uses: actions/cache@v4
with:
path: |
~/.cache/crafter.build/external
key: glslang-${{ runner.os }}-v2
- name: Bootstrap (build.sh)
run: ./build.sh
# Full style gate: Report() findings (naming, enum-class, char*, …) AND
# would-reformat findings from the transform rules. A dirty tree means
# someone skipped `crafter-build format` — fail fast, before the
# (much slower) test suite.
# - name: Lint
# run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build ./bin/crafter-build lint
- name: Run tests
run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build ./bin/crafter-build test
# Everything below produces release artifacts and only runs on push to
# master. PRs stop after the test step above — the bootstrap binary is
# enough to validate the change; the per-march variant builds and the
# archive uploads are wasted work on a PR.
- name: Build Linux x86-64-v2 (matches bootstrap)
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build ./bin/crafter-build
- name: Build Linux x86-64-v3
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build CRAFTER_BUILD_MARCH=x86-64-v3 ./bin/crafter-build
- name: Build Linux x86-64-v4
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build CRAFTER_BUILD_MARCH=x86-64-v4 ./bin/crafter-build
- name: Package artifacts
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
- name: Version and channel
id: meta
run: |
set -eu
if [ "${GITHUB_REF_TYPE}" = tag ]; then
version="$GITHUB_REF_NAME"
branch=stable
# The metainfo's newest release is the release notes and must be
# the version being tagged.
top=$(grep -m1 -oP '<release version="\K[^"]+' packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml)
if [ "$top" != "$version" ]; then
echo "metainfo lists $top as the newest release, but the tag is $version" >&2
exit 1
fi
else
version="0.0.0-dev+$(git rev-parse --short HEAD)"
branch=beta
fi
echo "version=$version" >> "$GITHUB_OUTPUT"
echo "branch=$branch" >> "$GITHUB_OUTPUT"
echo "version $version, flatpak branch $branch"
- name: Build
# The manifest is rewritten beside the original so its relative
# source paths still resolve. The suite runs inside this build.
env:
VERSION: ${{ steps.meta.outputs.version }}
run: |
set -eu
yq '."build-options".env.CRAFTER_BUILD_VERSION = strenv(VERSION)' "$MANIFEST" > "$CI_MANIFEST"
grep -n 'CRAFTER_BUILD_VERSION' "$CI_MANIFEST"
flatpak-builder --force-clean --ccache --disable-rofiles-fuse \
--state-dir=/root/.cache/flatpak-builder \
--repo=repo --default-branch="${{ steps.meta.outputs.branch }}" \
build-dir "$CI_MANIFEST"
- name: Smoke test
run: |
flatpak-builder --run build-dir "$CI_MANIFEST" crafter-build --version
flatpak-builder --run build-dir "$CI_MANIFEST" sh -c 'cd examples/hello-world && crafter-build && ls bin/*/hello*'
# Everything below publishes and only runs on master and on tags.
- name: Bundle
if: github.event_name != 'pull_request'
run: |
set -eux
mkdir -p dist
flatpak build-bundle repo "dist/crafter-build-${{ steps.meta.outputs.version }}.flatpak" "$APP_ID" "${{ steps.meta.outputs.branch }}"
cp packaging/flatpak/net.catcrafts.CrafterBuild.flatpakref dist/
(cd dist && sha256sum * > SHA256SUMS)
ls -la dist
# 3 archives, one per march level. Each carries the executable, the
# static library and share/; libc++ comes from the host.
for march in x86-64-v2 x86-64-v3 x86-64-v4; do
# Short suffix (v2/v3/v4) for archive filenames; the upload-artifact
# steps below reference these short names.
short=${march##*-}
# VariantId is "<name>-<target>-<march>-<mtune>-<hash>"; CI pins
# mtune=generic so each (target, march) resolves to a single subdir
# but the trailing hash isn't known to the workflow, hence the glob.
stage_lin=$(mktemp -d)
mkdir -p "$stage_lin/bin" "$stage_lin/lib"
cp bin/crafter.build-exe-x86_64-pc-linux-gnu-$march-*/crafter-build "$stage_lin/bin/"
cp bin/crafter.build-lib-x86_64-pc-linux-gnu-$march-*/libcrafter-build.a "$stage_lin/lib/"
cp -r share "$stage_lin/"
tar czf "dist/crafter-build-linux-x86_64-$short.tar.gz" -C "$stage_lin" .
done
ls -la dist/
# Upload each archive as its own workflow artifact (one upload per file)
# so users browsing the PR / run page get small, individually-named
# downloads instead of one wrapper zip containing the lot.
- name: Upload Linux v2 artifact
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
- name: Upload bundle
if: github.event_name != 'pull_request'
uses: actions/upload-artifact@v3
with: { name: crafter-build-linux-x86_64-v2, path: dist/crafter-build-linux-x86_64-v2.tar.gz, if-no-files-found: error }
- name: Upload Linux v3 artifact
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
uses: actions/upload-artifact@v3
with: { name: crafter-build-linux-x86_64-v3, path: dist/crafter-build-linux-x86_64-v3.tar.gz, if-no-files-found: error }
- name: Upload Linux v4 artifact
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
uses: actions/upload-artifact@v3
with: { name: crafter-build-linux-x86_64-v4, path: dist/crafter-build-linux-x86_64-v4.tar.gz, if-no-files-found: error }
with:
name: crafter-build-${{ steps.meta.outputs.version }}
path: dist/*.flatpak
if-no-files-found: error
- name: Update rolling 'latest' tag
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
- name: Publish to the Catcrafts repo
if: github.event_name != 'pull_request'
env:
FLATPAK_GPG_PRIVATE_KEY: ${{ secrets.FLATPAK_GPG_PRIVATE_KEY }}
FLATPAK_GPG_KEY_ID: ${{ secrets.FLATPAK_GPG_KEY_ID }}
run: |
set -eu
export GNUPGHOME=$(mktemp -d)
printf '%s' "$FLATPAK_GPG_PRIVATE_KEY" | gpg --batch --quiet --import
# The live repo is the canonical copy; builds are promoted into it,
# never built into it.
[ -d "$REPO_DIR" ] || ostree init --repo="$REPO_DIR" --mode=archive
flatpak build-commit-from --src-repo=repo \
--gpg-sign="$FLATPAK_GPG_KEY_ID" --gpg-homedir="$GNUPGHOME" \
"$REPO_DIR" "app/$APP_ID/x86_64/${{ steps.meta.outputs.branch }}"
flatpak build-update-repo --gpg-sign="$FLATPAK_GPG_KEY_ID" --gpg-homedir="$GNUPGHOME" \
--generate-static-deltas --prune "$REPO_DIR"
# The repo description and the app ref carry the public key.
key=$(gpg --export "$FLATPAK_GPG_KEY_ID" | base64 --wrap=0)
sed "s|GPGKey=.*|GPGKey=$key|" packaging/flatpak/catcrafts.flatpakrepo > "$WEB_DIR/catcrafts.flatpakrepo"
sed "s|GPGKey=.*|GPGKey=$key|" packaging/flatpak/net.catcrafts.CrafterBuild.flatpakref > "$WEB_DIR/net.catcrafts.CrafterBuild.flatpakref"
- name: Rolling 'latest' release
if: github.event_name != 'pull_request' && github.ref_type != 'tag'
run: |
git config user.email "ci@catcrafts.net"
git config user.name "Crafter Build CI"
git tag -f latest
git push origin latest --force
- name: Publish rolling 'latest' release
# forgejo-release uploads each file in release-dir as a separate
# release asset, so the Releases page shows individually
# downloadable archives — pick the v-level matching your CPU.
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
if: github.event_name != 'pull_request' && github.ref_type != 'tag'
uses: https://code.forgejo.org/actions/forgejo-release@v2
with:
direction: upload
url: ${{ github.server_url }}
repo: ${{ github.repository }}
tag: latest
title: Latest main build
title: Latest master build
prerelease: true
override: true
release-dir: dist
token: ${{ secrets.GITHUB_TOKEN }}
# musl build for Alpine / postmarketOS hosts: the glibc launcher above cannot
# run there (musl has no glibc loader and gcompat lacks libgcc_s + the
# __isoc23_* symbols). Bootstrap natively on alpine:edge and ship the
# bootstrap output — the same thing the Arch PKGBUILD installs. Dynamic
# against musl libc++ (crafter-build dlopens project.so, so it cannot be
# fully static); consumers need `apk add clang lld libc++-dev llvm-runtimes`.
release-musl:
needs: build-test-release
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
runs-on: arch-latest
container:
image: alpine:edge
env:
CRAFTER_BUILD_MARCH: x86-64-v2
CRAFTER_BUILD_MTUNE: generic
steps:
- name: Install build dependencies
# nodejs: actions/checkout is a Node action. build-base: gcc's
# libgcc_s + crt objects, which clang's musl driver links against, plus
# make/binutils for the glslang cmake build (python3: its generators).
# clang-dev + llvm-dev: the libclang header Crafter.Build-Lint includes
# lives under llvm-config --includedir here, not /usr/include. llvm:
# llvm-ar for the LTO static archives the self-host/test builds write.
- name: Release notes
if: github.ref_type == 'tag'
run: |
apk add -q nodejs git curl jq tar clang clang-dev lld libc++-dev \
llvm-libunwind-dev llvm-runtimes llvm llvm-dev cmake build-base python3
git config --global --add safe.directory '*'
- name: Checkout
uses: actions/checkout@v4
- name: Bootstrap (build.sh)
run: ./build.sh
- name: Run tests
run: CRAFTER_BUILD_HOME=$PWD/share/crafter-build ./bin/crafter-build test
- name: Package artifact
run: |
set -eux
file bin/crafter-build || true
mkdir -p dist
stage=$(mktemp -d)
mkdir -p "$stage/bin" "$stage/lib"
cp bin/crafter-build "$stage/bin/"
cp lib/libcrafter-build.a "$stage/lib/"
cp -r share "$stage/"
tar czf dist/crafter-build-linux-x86_64-musl-v2.tar.gz -C "$stage" .
ls -la dist/
- name: Upload musl artifact
uses: actions/upload-artifact@v3
with: { name: crafter-build-linux-x86_64-musl-v2, path: dist/crafter-build-linux-x86_64-musl-v2.tar.gz, if-no-files-found: error }
# Add the asset to the rolling 'latest' release the first job just
# (re)published, via the API rather than forgejo-release: that action's
# override mode would drop the other job's assets.
- name: Attach to rolling 'latest' release
env:
TOKEN: ${{ secrets.GITHUB_TOKEN }}
API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
run: |
set -eu
asset=crafter-build-linux-x86_64-musl-v2.tar.gz
rel=$(curl -fsS -H "Authorization: token $TOKEN" "$API/releases/tags/latest")
id=$(printf '%s' "$rel" | jq -r .id)
for old in $(printf '%s' "$rel" | jq -r --arg n "$asset" '.assets[] | select(.name==$n) | .id'); do
curl -fsS -X DELETE -H "Authorization: token $TOKEN" "$API/releases/$id/assets/$old"
done
curl -fsS -H "Authorization: token $TOKEN" -F "attachment=@dist/$asset" \
"$API/releases/$id/assets?name=$asset" > /dev/null
echo "attached $asset to release $id"
# The newest <release> block's description, as plain text.
python3 - > release-notes.md <<'EOF'
import re, html
text = open("packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml").read()
block = re.search(r"<release [^>]*>(.*?)</release>", text, re.S).group(1)
paras = re.findall(r"<p>(.*?)</p>", block, re.S)
print("\n\n".join(html.unescape(" ".join(p.split())) for p in paras))
print("\nInstall:\n\n flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo\n flatpak install catcrafts net.catcrafts.CrafterBuild\n")
EOF
- name: Publish release
if: github.ref_type == 'tag'
uses: https://code.forgejo.org/actions/forgejo-release@v2
with:
direction: upload
url: ${{ github.server_url }}
repo: ${{ github.repository }}
tag: ${{ github.ref_name }}
title: crafter-build ${{ github.ref_name }}
release-notes: release-notes.md
release-dir: dist
token: ${{ secrets.GITHUB_TOKEN }}