Catcrafts/Crafter.Review
Watch
2
0
Fork
You've already forked Crafter.Review
 0
Code Issues 7 Pull requests Projects Releases 3 Packages Wiki Activity Actions
Crafter.Review/tests/OpenCode/main.cpp
Jorijn van der Graaf e215ad986e
 feat(seats): the env file's settings count, and a machine's own seats file 
ApiConfig and PoolConfig read the seat order, the budgets and the
concurrency from the process environment alone, so a `.env` that said
REVIEW_GATES_SEAT_ORDER=oauth-first ordered nothing: the Python driver
reads those lines from the file under the process environment
(config_env, 2026-10-17: a setting lives beside the seats it orders).
SeatPool::SettingsOf does the same - the file's placement settings
under the process environment, in a map handed to the placement readers
alone (ApiConfig, PoolConfig, Seats::ConfigFromEnv) - and leaves the
process environment untouched, so no other reader sees a line of the
file: a path's or a binary's line reaches nothing, and two drivers
reading one .env keep one board and one ledger. The placement settings
are one list, SeatPool::PlacementSettings - the fifteen names those
readers read - for the map, the run's log and what makes a working
directory's .env a seats file; a REVIEW_GATES_* line of any other name
stays out of the map and is named, by the run's log and on `seats`'s
stderr. One dotenv grammar serves the tokens and the settings
(Seats::DotenvLines).

`run` and `seats` resolve the file from their own --env-file
(SeatPool::ResolveSeatsFile): the --env-file, else $SEATS_ENV_FILE, else
the working directory's .env - the operator's, as the Python driver and
`seats` read it, not the repository under review's - when it is a file
holding a seat or a placement setting, each with a value, or one that
cannot be read as UTF-8 text (its error is then the command's) - a
readable application .env, its Vite or Supabase lines, one of blank
placeholder lines or one that is no file is passed over, and the run
says so - else the machine's own
$XDG_CONFIG_HOME/crafter-review/env, else ~/.config/crafter-review/env:
one file for every repository on a machine, a repository's seats .env
winning over it. Each file is read once and the settings and the tokens
are taken from that read (SeatsFile::Text), so a file that can be read
only once - a pipe, `--env-file <(gpg -d seats.env.gpg)`, a FIFO - gives
its settings and its seats alike, and a failed read raises its own error
where the tokens are laid over the environment: SeatPool::WithFileTokens,
for `run` and `seats` alike, and LoadSeats takes the merged environment
and no longer reads a file by path. An empty `seats --env-file` means no file
now, as it did for `run`, and the `seats` report names the side that
runs first (it said "api seats run first" whatever the order).

Runs, Commands and KernelModes, the tests that run `run` and `seats`,
point XDG_CONFIG_HOME, and HOST_XDG_CONFIG_HOME for the Flatpak, at a
path of their own that nothing creates, so a global file stays out of
them; the grammar, the settings and their precedence, the lookup and the
one read have their tests, a run with `--env` (a prefix) and with a
repeated --env-file, and a `seats` with `--env`, take tokens and
settings from the one file, an application's .env and a venv are passed
over, a FIFO is read once, no line of the file reaches `stop`, every
listed placement setting moves a reader, another REVIEW_GATES_* line is
named and a .env of blank lines passed over, and the
README, the environment table and the procedure say the new rule.
2026-11-02 00:30:52 +01:00

232 lines
16 KiB
C++
Raw Blame History

// SPDX-License-Identifier: AGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// The OpenCode harness, against the goldens (tests/Goldens/fixture/
// pipeline.json, under `opencode`): the tool source, the permission rules,
// the agent config, the commands, the environment, the home, the event
// parsing, the classification, and a warm-up against a fake opencode. The
// test binary is its own supervisor (`__supervise`), as the driver is.
#include <signal.h>
#include <sys/stat.h>
import std;
import Crafter.Review;
namespace fs = std::filesystem;
using namespace Crafter::Review;
namespace {
std::int32_t Failures = 0;
void Check(std::string_view name, bool cond, std::string_view detail = {}) {
std::println("{}{}{}", cond ? " ok " : " FAIL ", name, !cond && !detail.empty() ? std::format(" ({})", detail) : std::string());
if (!cond) ++Failures;
}
void Write(const fs::path& path, std::string_view text) {
fs::create_directories(path.parent_path());
std::ofstream(path, std::ios::binary) << text;
}
std::string Read(const fs::path& path) {
return Platform::ReadFile(path).value_or("");
}
std::vector<std::string> Strings(const Json& array) {
std::vector<std::string> out;
for (const Json& item : array) out.push_back(item.get<std::string>());
return out;
}
std::vector<std::string> Names(const fs::path& dir) {
std::vector<std::string> names;
for (const fs::directory_entry& entry : fs::directory_iterator(dir)) names.push_back(entry.path().filename().string());
std::ranges::sort(names);
return names;
}
// What the generator's temp dir is in the goldens.
std::string Tidy(std::string text, const fs::path& td) {
return Dyn::Replace(text, td.string(), "<td>");
}
std::int32_t Checks() {
std::println("opencode harness + OpenRouter seats: loader, environments, permissions, event parsing, the tool schema");
Platform::SetMode(Platform::Mode::Native);
// The directory modes below are the generator's, made under 022.
umask(022);
const fs::path td = fs::temp_directory_path() / std::format("crafter-review-opencode-{}", Platform::PublishedPid());
fs::remove_all(td);
fs::create_directories(td);
const fs::path envFile = td / ".env";
Write(envFile, "CLAUDE_CODE_OAUTH_TOKEN1=tok\nOPENROUTER_KEY=sk-or-one\n");
// The seats file as run and seats take it: one read, its tokens over
// an environment (an empty one here).
const std::map<std::string, std::string> fileEnv = SeatPool::WithFileTokens({}, SeatPool::ResolveSeatsFile(envFile.string(), td));
auto ids = [](const std::vector<Json>& seats) {
std::vector<std::string> out;
for (const Json& s : seats) out.push_back(s.at("id").get<std::string>());
return out;
};
Check("OPENROUTER_KEY is a seat only on request",
ids(SeatPool::LoadSeats(fileEnv, "")) == std::vector<std::string>{"1"} && ids(SeatPool::LoadSeats(fileEnv, "openrouter")) == std::vector<std::string>{"or1"});
try {
SeatPool::LoadSeats(fileEnv, "or1,1");
Check("an OpenRouter seat never shares a pool with Anthropic seats", false);
} catch (const std::invalid_argument& e) {
Check("an OpenRouter seat never shares a pool with Anthropic seats", std::string_view(e.what()).contains("cannot share"), e.what());
}
const std::vector<std::string> allow = {"Read", "Glob", "Grep", "Bash(cat:*)", "Bash(git -C . diff:*)", "Bash(python3 -:*)"};
const Json p = OpenCode::Permissions(allow);
Check("allowlist -> permission rules", p.at("read") == "allow" && p.at("StructuredOutput") == "allow" && p.at("bash").at("*") == "deny" && p.at("bash").at("git -C . diff*") == "allow");
const std::vector<Json> ev = {
{{"type", "step_start"}, {"sessionID", "ses_1"}, {"part", Json::object()}},
{{"type", "tool_use"},
{"part",
{{"type", "tool"},
{"tool", "StructuredOutput"},
{"state", {{"status", "completed"}, {"input", {{"verdict", "SEND-READY"}, {"findings", Json::array()}, {"notes", Json::array()}, {"coverage", {{{"item", "u"}, {"verdict", "clear"}, {"basis", "b"}}}}}}}}}}},
{{"type", "step_finish"}, {"part", {{"type", "step-finish"}, {"reason", "stop"}, {"tokens", {{"input", 20}, {"output", 4}, {"reasoning", 1}, {"cache", {{"read", 150}, {"write", 0}}}}}, {"cost", 0.0005}}}},
};
std::vector<std::string> lines;
for (const Json& e : ev) lines.push_back(DumpSpaced(e));
const Json ps = OpenCode::ParseEvents(Dyn::Join(lines, "\n"));
Check("events -> the structured output, usage, classify ok",
ps.at("structured").at("coverage").at(0).at("item") == "u" && ps.at("usage").at("output") == 5 && OpenCode::Classify(ps, 0, "") == std::pair<std::string, std::string>{"ok", ""});
const std::string src = OpenCode::ToolSource();
Check("the tool schema carries the note kinds, the coverage vocabulary and the verdicts of the schema; no persona field",
std::ranges::all_of(std::array<std::string_view, 5>{"reserve-answer", "unverified", "FIX FIRST", "refutation-verified", "n/a"},
[&](std::string_view k) { return src.contains(DumpSpaced(Json(std::string(k)))); })
&& !src.contains("notMine") && !src.contains("wouldComment"));
const fs::path repo = td / "repo";
fs::create_directories(repo / "src");
fs::create_directories(repo / ".git");
fs::create_directories(repo / "node_modules");
Write(repo / "package.json", "{}");
const std::vector<std::string> one = {"openrouter/x/y"};
const fs::path home = OpenCode::PrepareHome(td / "sc", repo, one);
Check("prepare_home: the view links every top-level entry but .git and node_modules",
Names(home / "view") == std::vector<std::string>{"package.json", "src"} && fs::is_symlink(fs::symlink_status(home / "view" / "src")));
// ---- against the goldens ---------------------------------------------------------------
const Json g = Dyn::LoadJsonFile(fs::current_path() / "tests" / "Goldens" / "fixture" / "pipeline.json").at("opencode");
Check("goldens: the tool source, byte for byte", src == g.at("toolSource").get<std::string>());
Check("goldens: the provider routing", DumpFile(OpenCode::Routing()) == DumpFile(g.at("routing")));
std::vector<std::string> missed;
for (const Json& entry : g.at("modelId")) {
const auto [prov, id] = OpenCode::ModelId(entry.at(0).get<std::string>());
if (Json::array({prov, id}) != entry.at(1)) missed.push_back(entry.at(0).get<std::string>());
}
Check("goldens: model ids", missed.empty(), Dyn::Join(missed, ", "));
missed.clear();
std::size_t index = 0;
for (const Json& entry : g.at("permissions")) {
const std::vector<std::string> list = Strings(entry.at(0));
if (DumpFile(OpenCode::Permissions(list)) != DumpFile(entry.at(1))) missed.push_back(DumpCompact(entry.at(0)));
const std::map<std::string, std::string> child = OpenCode::ChildEnv({{"PATH", "/bin"}}, "/abs/system é.md", "/abs/so.json", list, td / "data" / std::to_string(index));
if (child.at("OPENCODE_CONFIG_CONTENT") != g.at("agentConfig").at(index).at(1).get<std::string>()) missed.push_back(std::format("agent config {}", std::to_string(index)));
if (child.at("REVIEW_GATES_STRUCTURED_OUT") != "/abs/so.json" || child.at("XDG_DATA_HOME") != (td / "data" / std::to_string(index)).string()) missed.push_back(std::format("child env {}", std::to_string(index)));
++index;
}
Check("goldens: permission rules and the inline agent config, key for key", missed.empty(), Dyn::Join(missed, "; "));
missed.clear();
for (const Json& entry : g.at("cmd")) {
const Json& a = entry.at(0);
const std::vector<std::string> got = OpenCode::Cmd(a.at(0).get<std::string>(), a.at(1).get<std::string>(), a.at(2).get<std::string>(), a.at(3).get<std::string>(),
a.at(4).is_null() ? std::string() : a.at(4).get<std::string>(), a.at(5).get<std::string>());
if (got != Strings(entry.at(1))) missed.push_back(DumpCompact(a));
}
Check("goldens: the opencode command lines", missed.empty(), Dyn::Join(missed, "; "));
std::map<std::string, std::string> baseIn;
for (auto it = g.at("baseEnv").at(0).begin(); it != g.at("baseEnv").at(0).end(); ++it) baseIn[it.key()] = it.value().get<std::string>();
std::map<std::string, std::string> baseWant;
for (auto it = g.at("baseEnv").at(1).begin(); it != g.at("baseEnv").at(1).end(); ++it) baseWant[it.key()] = it.value().get<std::string>();
Check("goldens: the shared environment - its own XDG home and key, no Anthropic or OpenCode variable", OpenCode::BaseEnv("/h", "sk-or-key", baseIn) == baseWant);
// The generator's temp dir, which the goldens call <td>.
const fs::path gd = td / "g";
const fs::path repo2 = gd / "repo";
for (const std::string_view d : {".git", "node_modules", "src", "b", "a"}) fs::create_directories(repo2 / d);
for (const std::string_view name : {"package.json", ".env", "Z"}) Write(repo2 / name, "{}");
const Json& h = g.at("home");
const fs::path home2 = OpenCode::PrepareHome(gd / "sc", repo2, Strings(h.at("models")));
const fs::path config = home2 / "xdg" / "opencode" / "opencode.json";
const std::string firstConfig = Read(config);
OpenCode::PrepareHome(gd / "sc", repo2, one);
Check("goldens: prepare_home's config, byte for byte, and rewritten by the next call", firstConfig == h.at("firstConfig").get<std::string>() && Read(config) == h.at("config").get<std::string>());
std::vector<std::string> view;
for (const std::string& name : Names(home2 / "view")) view.push_back(std::format("{} -> {}", name, Tidy(fs::read_symlink(home2 / "view" / name).string(), gd)));
std::vector<std::string> viewWant;
for (const Json& entry : h.at("view")) viewWant.push_back(std::format("{} -> {}", entry.at(0).get<std::string>(), entry.at(1).get<std::string>()));
Check("goldens: the view's links, sorted, dot files included, .git and node_modules left out; idempotent", view == viewWant && Tidy(home2.string(), gd) == h.at("home").get<std::string>(),
Dyn::Join(view, "; "));
missed.clear();
for (auto it = h.at("modes").begin(); it != h.at("modes").end(); ++it) {
const auto mode = static_cast<std::uint32_t>(fs::status(home2 / it.key()).permissions()) & 0777;
if (std::format("0o{:o}", mode) != it.value().get<std::string>()) missed.push_back(std::format("{} 0o{:o}", it.key(), mode));
}
Check("goldens: os.makedirs' modes - the last directory 0700, its parents as mkdir makes them", missed.empty() && Read(home2 / "xdg" / "opencode" / "tools" / "StructuredOutput.ts") == src,
Dyn::Join(missed, ", "));
missed.clear();
for (const Json& entry : g.at("events")) {
if (DumpFile(OpenCode::ParseEvents(entry.at(0).get<std::string>())) != DumpFile(entry.at(1))) missed.push_back(Dyn::Repr(Dyn::Slice(entry.at(0).get<std::string>(), 0, 60)));
}
Check("goldens: the event stream - steps, tokens, tool calls and errors, the session, the first turn", missed.empty(), Dyn::Join(missed, "; "));
missed.clear();
for (const Json& entry : g.at("classify")) {
const Json& c = entry.at(0);
const auto [status, reason] = OpenCode::Classify(c.at(0), c.at(1).get<std::int64_t>(), c.at(2).get<std::string>(), c.at(3).get<bool>());
if (Json::array({status, reason}) != entry.at(1)) missed.push_back(std::format("{} -> {} {}", Dyn::Repr(Dyn::Slice(c.at(2).get<std::string>(), 0, 40)), status, Dyn::Repr(Dyn::Slice(reason, 0, 60))));
}
Check("goldens: classify - the time budget, then credits, auth and limits as the regexes read them (re.I, \\b), then why nothing came back", missed.empty(), Dyn::Join(missed, "; "));
const Json& w = g.at("warm");
const fs::path fake = td / "opencode";
Write(fake, g.at("fake").get<std::string>());
fs::permissions(fake, fs::perms::owner_all | fs::perms::group_read | fs::perms::group_exec | fs::perms::others_read | fs::perms::others_exec);
const std::map<std::string, std::string> env = OpenCode::BaseEnv(home2, "sk-or", std::map<std::string, std::string>{{"PATH", "/usr/bin:/bin"}});
std::vector<std::string> log;
const auto warmed = OpenCode::Warm(fake.string(), home2, env, "openrouter/x/y", [&](const std::string& line) { log.push_back(line); });
const std::string args = Tidy(Read(home2 / "data" / "warm" / "args"), gd);
Check("goldens: a warm-up that fails - its log line, the tail of its stderr, the command it ran",
warmed == std::pair<bool, std::string>{w.at("result").at(0).get<bool>(), w.at("result").at(1).get<std::string>()} && log == Strings(w.at("log")) && args == w.at("args").get<std::string>(),
std::format("{} | {}", Dyn::Join(log, " | "), Dyn::Repr(args)));
const fs::path slow = td / "slow-opencode";
Write(slow, "#!/bin/sh\necho $$ > \"$XDG_DATA_HOME/pid\"\nexec sleep 30\n");
fs::permissions(slow, fs::perms::owner_all);
log.clear();
const auto timed = OpenCode::Warm(slow.string(), home2, env, "openrouter/x/y", [&](const std::string& line) { log.push_back(line); }, 1);
const std::optional<std::int64_t> sleeper = Dyn::ParseInt(Dyn::Strip(Read(home2 / "data" / "warm" / "pid")));
Check("goldens: a warm-up out of time says so and logs nothing",
timed == std::pair<bool, std::string>{w.at("timedOut").at(0).get<bool>(), w.at("timedOut").at(1).get<std::string>()} && log.empty() && w.at("timedLog").empty());
Check("...and leaves no process behind: the supervisor takes it down", sleeper && !Platform::ProcStart(*sleeper));
// ---- beyond the goldens -----------------------------------------------------------------
const fs::path so = td / "so.json";
Write(so, R"({"verdict": "SEND-READY", "findings": [], "notes": [], "coverage": []})");
const fs::path bad = td / "bad.json";
Write(bad, R"({"verdict": "SEND-READY", "findings": "none"})");
const fs::path garbage = td / "garbage.json";
Write(garbage, "{");
const std::string noTool = DumpSpaced({{"type", "step_finish"}, {"part", {{"reason", "stop"}}}});
Check("the structured-output file stands in when the stream carries no call: one with a findings list, nothing else",
OpenCode::ParseEvents(noTool, so).at("structured").at("verdict") == "SEND-READY" && OpenCode::ParseEvents(noTool, bad).at("structured").is_null()
&& OpenCode::ParseEvents(noTool, garbage).at("structured").is_null() && OpenCode::ParseEvents(noTool, td / "missing.json").at("structured").is_null()
&& OpenCode::ParseEvents(noTool).at("structured").is_null());
fs::remove_all(td);
return Failures == 0 ? 0 : 1;
}
}
int main(int argc, char** argv) {
const std::vector<std::string_view> args(argv, argv + argc);
if (args.size() > 1 && args[1] == Process::SuperviseVerb) return Process::SuperviseMain(std::span(args).subspan(2));
try {
return Checks();
} catch (const std::exception& error) {
std::println(std::cerr, "opencode: {}", error.what());
return 1;
}
}
Reference in a new issue View git blame Copy permalink