Jorijn van der Graaf
ceb87f70ef
The ledger reset every API seat's spend on one REVIEW_GATES_API_CYCLE_DAY and cleared every credit-out, an OpenRouter key's too, at that reset. Each API seat's cycle starts on its ANTHROPIC_API_KEY<n>_GRANT_DAY now (1..31, a day past a month's end its last), else the cycle day, also 1..31. A credit-out carries across the rollover, and the run's probe tries a key held out on credit again, as `seats` does an API key: an ok lifts the mark - an API key's is new credits, its cycle restarting from the grant day, or from today when that day is over a week back; an OpenRouter key keeps its cycle's spend, and a key that answers with its cycle's budget spent stays out on that budget. A probe that finds a key out of credit records it, unless the ledger already has it out. Both are decided in one locked update on the file's entry, a mark another driver made since the probe started standing, and the ledger's stamps carry their UTC offset so marks are ordered by instant. `seats` shows the probe's reading beside the driver's verdict (the `probe` and `driver` columns replace `state` and `ledger: out`), each seat's grant day and when its next credits are due, and notes on what the probe changed in the ledger or why the driver's verdict stands; the run logs the days.
341 lines
21 KiB
C++
341 lines
21 KiB
C++
// SPDX-License-Identifier: AGPL-3.0-only
|
|
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
|
|
|
export module Crafter.Review:SeatPool;
|
|
import std;
|
|
import :Json;
|
|
|
|
// The seat pool: which credential each lens process runs on. OAuth seats
|
|
// (CLAUDE_CODE_OAUTH_TOKEN1..6), API-credit seats (ANTHROPIC_API_KEY1..6,
|
|
// real money, a ledger and a budget), OpenRouter seats
|
|
// (OPENROUTER_KEY[1..6], never in the default pool), and `local`, this
|
|
// machine's login. Placement: API seats first (their credit expires monthly),
|
|
// then OAuth seats by projected utilisation; a process that fails on a seat's
|
|
// limit, credit or auth is placed again; the pool aborts only when every seat
|
|
// is out. Secrets never reach a log, a dump or a child's environment except
|
|
// the one the CLI reads.
|
|
//
|
|
// Several drivers at once (2026-10-31): every driver on this machine
|
|
// registers on the board (~/.cache/review-gates/seat-board.json) what it has
|
|
// in flight on each seat, and folds the board into its seats and admission
|
|
// before every launch; the pick is registered under the same lock. Every
|
|
// driver on the machine shares the file, its shape and its lock. A seat is a
|
|
// JSON object: the dump records it (without the secret).
|
|
//
|
|
// Liveness (README, AGENTS.md): a driver is alive while its lifetime lock
|
|
// (<board dir>/drivers/<pid>-<start>.lock) is held; without one, natively
|
|
// the pid and start time decide, and in the Flatpak the driver is taken for
|
|
// alive, because pids of another namespace say nothing here.
|
|
export namespace Crafter::Review::SeatPool {
|
|
// A job is re-placed this often after a limit, credit or auth failure.
|
|
constexpr std::int32_t MaxAttempts = 4;
|
|
// An API seat's pause after a 429.
|
|
constexpr double ApiBackoffS = 90;
|
|
constexpr double OauthFallbackPauseS = 300;
|
|
// An OAuth seat's pause after a 'concurrents' rejection: too many requests
|
|
// at once on the account, cleared as the ones in flight answer. Not the
|
|
// window's reset (2026-10-31: a seat sat two hours at 10 % on one).
|
|
constexpr double ConcurrentBackoffS = 20;
|
|
// The passes one OAuth account (or the local login) takes at once, every
|
|
// driver's counted. Until 2026-10-31 the 5-hour projection was the only
|
|
// limit, and an account without a 5-hour limit has none: on the record,
|
|
// every run with 149 or more in flight on one account drew usage-limit
|
|
// rejections, none of the 119 with 139 or fewer.
|
|
// REVIEW_GATES_OAUTH_CONCURRENCY overrides.
|
|
constexpr std::int64_t OauthConcurrency = 128;
|
|
// What one running pass is projected to cost of an OAuth seat's windows,
|
|
// in percentage points, by the run's reasoning effort (every lens of a run
|
|
// shares --effort): the pooled means of 5438 Opus 5.5 passes on 141
|
|
// seat-runs (2026-10-12..15), rounded up - max 0.44 / 0.118, xhigh
|
|
// 0.12 / 0.033. The 85 % ceiling and the cutoff's last percent absorb the
|
|
// spread. An effort not in the table projects at the max figures;
|
|
// REVIEW_GATES_LENS_COST_PCT and REVIEW_GATES_LENS_COST_7D_PCT override
|
|
// every effort.
|
|
constexpr double LensCostPct(std::string_view effort) { return effort == "xhigh" ? 0.125 : 0.5; }
|
|
constexpr double LensCost7dPct(std::string_view effort) { return effort == "xhigh" ? 0.04 : 0.125; }
|
|
// The kinds whose processes cost money the ledger counts.
|
|
constexpr std::array<std::string_view, 2> Paid = {"api", "openrouter"};
|
|
// The kinds whose out/invalid verdict crosses drivers on the board.
|
|
constexpr std::array<std::string_view, 2> StateShared = {"oauth", "local"};
|
|
constexpr std::string_view BoardFile = "seat-board.json";
|
|
// An in-flight entry outlives its pass's time budget x1.5 by this much
|
|
// before the sweep takes it.
|
|
constexpr double InflightGraceS = 300;
|
|
constexpr std::string_view OpenRouterBaseUrl = "https://openrouter.ai/api";
|
|
constexpr std::string_view OpenRouterModelsUrl = "https://openrouter.ai/api/v1/models";
|
|
constexpr std::string_view OpenRouterKeyUrl = "https://openrouter.ai/api/v1/key";
|
|
extern const std::string_view ResumePrompt;
|
|
|
|
bool IsPaid(const Json& seat);
|
|
// The API-seat knobs.
|
|
Json ApiConfig(const std::map<std::string, std::string>& env);
|
|
// The seats file of `run` or `seats` (2026-11-01): the --env-file given
|
|
// (an empty one: none), else $SEATS_ENV_FILE, else <cwd>/.env - the
|
|
// working directory's, the operator's, not the repository under
|
|
// review's - when it is a file holding a seat or a placement setting,
|
|
// each with a value (an application's .env, one of blank lines, or one
|
|
// that is no file, is passed over and recorded so), else the machine's
|
|
// own (GlobalEnvFile); none: this
|
|
// machine's own login. Each file is read once, and the command takes
|
|
// its settings and its tokens from that read - the tokens through
|
|
// Text() - so a file that can be read only once (a pipe, `--env-file
|
|
// <(gpg -d seats.env.gpg)`, a FIFO) gives its settings and its seats
|
|
// alike (the self-review's round 10: read twice, it lost its seats or
|
|
// hung).
|
|
struct SeatsFile {
|
|
std::optional<std::filesystem::path> path;
|
|
std::optional<std::filesystem::path> passedOver;
|
|
std::string text;
|
|
std::exception_ptr error;
|
|
// The text as read; the read's own error, raised again, when the
|
|
// read failed (a file missing or not UTF-8 ends the command).
|
|
const std::string& Text() const;
|
|
};
|
|
// The lookup and its one read.
|
|
SeatsFile ResolveSeatsFile(const std::optional<std::string>& given, const std::filesystem::path& cwd);
|
|
// The settings a seats file may carry: every name ApiConfig, PoolConfig
|
|
// and Seats::ConfigFromEnv read (2026-11-01, the self-review's round 19:
|
|
// one list for the map they take, its log and what makes a working
|
|
// directory's .env a seats file).
|
|
constexpr std::array<std::string_view, 15> PlacementSettings = {
|
|
"REVIEW_GATES_API_BUDGET_USD", "REVIEW_GATES_API_LENS_COST_USD", "REVIEW_GATES_API_CONCURRENCY", "REVIEW_GATES_API_CYCLE_DAY",
|
|
"REVIEW_GATES_SEAT_ORDER", "REVIEW_GATES_OPENROUTER_BUDGET_USD", "REVIEW_GATES_OPENROUTER_LENS_COST_USD",
|
|
"REVIEW_GATES_OPENROUTER_CONCURRENCY", "REVIEW_GATES_LENS_COST_PCT", "REVIEW_GATES_LENS_COST_7D_PCT",
|
|
"REVIEW_GATES_OAUTH_CONCURRENCY", "REVIEW_GATES_SEAT_MAX_UTIL", "REVIEW_GATES_SEAT_WAIT_MIN", "REVIEW_GATES_SEAT_HANDOVER_PCT",
|
|
"REVIEW_GATES_SEAT_POLICY"};
|
|
// The settings the placement readers take (ApiConfig, PoolConfig,
|
|
// Seats::ConfigFromEnv): the process environment, and the file's
|
|
// PlacementSettings lines where it has no such variable - the Python
|
|
// driver's config_env (2026-10-17: a setting lives beside the seats it
|
|
// orders). A map for those readers alone: the process environment is
|
|
// not touched, so no other reader sees a line of the file, and a path's
|
|
// or a binary's line reaches nothing (the self-review's round 11). The
|
|
// names the file supplied are `fromFile`; its REVIEW_GATES_* lines of
|
|
// any other name, which only the process environment sets, are
|
|
// `ignored` - named by the run and `seats`, never taken in silence.
|
|
struct Settings {
|
|
std::map<std::string, std::string> env;
|
|
std::vector<std::string> fromFile;
|
|
std::vector<std::string> ignored;
|
|
};
|
|
Settings SettingsOf(const SeatsFile& file);
|
|
// env with the seats file's seat lines over it, from the file's one read
|
|
// (Text(): a read that failed raises its error here) - the one place a
|
|
// command lays the file's tokens over an environment (the self-review's
|
|
// round 19: run and seats had each done it by hand, and LoadSeats read
|
|
// the file a second time by path).
|
|
std::map<std::string, std::string> WithFileTokens(std::map<std::string, std::string> env, const SeatsFile& file);
|
|
// The machine's own seats file, read when a run or `seats` has no
|
|
// --env-file, no $SEATS_ENV_FILE and no seats ./.env: $XDG_CONFIG_HOME/crafter-review/env,
|
|
// else ~/.config/crafter-review/env - when it exists (2026-11-01, one
|
|
// file for every repository on a machine; a repository's own .env wins).
|
|
std::optional<std::filesystem::path> GlobalEnvFile();
|
|
// The other drivers' load on a seat as the last board fold left it.
|
|
Json Foreign(const Json& seat);
|
|
std::filesystem::path LedgerPath(const std::optional<std::filesystem::path>& scratchRoot = std::nullopt);
|
|
// The most recent date with that day of the month, as YYYY-MM-DD; a day
|
|
// past a month's end is its last day.
|
|
std::string CycleStart(double now, std::int32_t day);
|
|
// When a seat's next credits are due, as YYYY-MM-DD: the grant day after
|
|
// the current cycle's, or the current cycle's own when the seat went out
|
|
// on credit before it (outSince) and is still out - due today, or overdue:
|
|
// the grant day is wrong.
|
|
std::string NextGrant(double now, std::int32_t day, const Json& outSince = nullptr);
|
|
// The driver's verdict on a paid seat from its ledger entry alone: out
|
|
// with the reason, or ok with none.
|
|
std::pair<bool, std::string> LedgerVerdict(const Json& entry, double budget);
|
|
// Why that verdict holds a seat out: "credit" (the key answered the
|
|
// credit error), "budget", or null when it does not.
|
|
Json LedgerOut(const Json& entry, double budget);
|
|
|
|
// What the driver spent on each API seat this cycle, and whether the key
|
|
// answered the credit error; one file for every driver, each write a delta
|
|
// applied under its lock. A seat's cycle starts on its grant day (days,
|
|
// else the config's cycleDay).
|
|
class Ledger {
|
|
public:
|
|
Ledger(std::filesystem::path path, Json apiConfig, Json data = Json::object(), std::map<std::string, std::int32_t> days = {});
|
|
static Ledger Load(const std::filesystem::path& path, const Json& apiConfig, std::map<std::string, std::int32_t> days = {});
|
|
std::int32_t Day(const std::string& seatId) const;
|
|
void SetDay(const std::string& seatId, std::int32_t day);
|
|
// The seat's entry of the current cycle. A new cycle starts the spend
|
|
// over; a credit-out carries over it - only a probe that sees the key
|
|
// answer again lifts it (Reconcile) - with the time the key was last
|
|
// seen answering again. This process's copy: only an Update
|
|
// re-reads the file.
|
|
Json& Entry(const std::string& seatId, std::optional<double> now = std::nullopt);
|
|
Json Add(const std::string& seatId, double usd, std::optional<double> now = std::nullopt);
|
|
Json MarkOut(const std::string& seatId, std::string_view reason, std::optional<double> now = std::nullopt);
|
|
// The probe's reading of an API key (probedAt: when the probe
|
|
// started), and the ledger follows it, decided under the lock on the
|
|
// entry as the file has it: "ok" on a key held out on credit since
|
|
// before the probe started lifts the mark - an API key's is new
|
|
// credits, GrantEntry's entry, "new-grant"; an OpenRouter key's keeps
|
|
// its cycle's spend, "credit-back" -; the credit error on a key the file
|
|
// has as fine, no grant seen since the probe started, is recorded,
|
|
// "credit-gone". Anything else writes nothing, "": a mark another
|
|
// driver made after the probe started may postdate its reading, and
|
|
// a grant another driver already recorded keeps its spend.
|
|
std::string Reconcile(const std::string& seatId, const Json& state, double probedAt, std::optional<double> now = std::nullopt);
|
|
void Save();
|
|
const Json& Data() const;
|
|
const std::filesystem::path& Path() const;
|
|
// The config the ledger was loaded with (ApiConfig's).
|
|
const Json& ApiConfig() const;
|
|
|
|
private:
|
|
std::filesystem::path path;
|
|
Json apiConfig;
|
|
Json data;
|
|
std::map<std::string, std::int32_t> days;
|
|
// change returns whether to write the file.
|
|
Json Update(const std::string& seatId, const std::function<bool(Json&)>& change, std::optional<double> now);
|
|
// A grant's entry: the cycle from the seat's grant day when that is
|
|
// within the week, else from today; no spend, no out mark.
|
|
Json GrantEntry(const std::string& seatId, double at) const;
|
|
};
|
|
|
|
Json NewSeat(const std::string& id, const std::string& kind, const std::string& secret);
|
|
// The seats a run may use. spec: auto (every Anthropic seat of env - the
|
|
// seats file's over the process environment's, WithFileTokens - else
|
|
// local), local | api | oauth | openrouter,
|
|
// or a list such as api1,1,2,local. std::invalid_argument with a
|
|
// ValueError message for a seat that is not configured, or OpenRouter
|
|
// mixed with the rest. An API seat gets its grantDay
|
|
// (ANTHROPIC_API_KEY<n>_GRANT_DAY of env, else the cycleDay) into the
|
|
// seat and the ledger; a seat the ledger holds out is out, ledgerOut
|
|
// "credit" or "budget".
|
|
std::vector<Json> LoadSeats(const std::map<std::string, std::string>& env, std::string_view spec, Ledger* ledger = nullptr, const Json& apiConfig = nullptr,
|
|
std::optional<double> now = std::nullopt);
|
|
using ProbeFunction = std::function<Json(std::int32_t seat, const std::string& token, const std::string& model, double timeout, const std::string& claudeBin, bool local, bool api)>;
|
|
// An OpenRouter key's probe, ProbeOpenRouter's shape.
|
|
using OpenRouterProbeFunction = std::function<Json(const std::string& secret, double timeout)>;
|
|
// One probe per seat before the run; a seat already out is skipped,
|
|
// except a paid key the ledger holds out on credit: probed again
|
|
// (Reconcile). A probed key that answers with its cycle's budget spent
|
|
// (an OpenRouter key's lift keeps its spend) is out on that budget. The
|
|
// OpenRouter price catalogue is fetched when there are models to price.
|
|
void Probe(std::vector<Json>& seats, const std::string& claudeBin, double timeout, const std::vector<std::string>& models,
|
|
const std::function<void(const std::string&)>& log, const ProbeFunction& probe = {}, Ledger* ledger = nullptr, std::optional<double> now = std::nullopt,
|
|
const OpenRouterProbeFunction& probeKey = {});
|
|
void ApplyProbe(Json& seat, const Json& result);
|
|
std::string Describe(const Json& seat);
|
|
|
|
// The OAuth placement knobs, with the pool's own per-pass projections at
|
|
// the run's effort (LensCostPct, LensCost7dPct; the env overrides them)
|
|
// and the per-account cap.
|
|
Json PoolConfig(const std::map<std::string, std::string>& env, std::string_view effort = {});
|
|
double OauthProjected(const Json& seat, const Json& config);
|
|
// A window whose reset has passed counts as empty until the next reading.
|
|
std::vector<std::string> ExpireWindows(Json& seat, double now);
|
|
bool Eligible(Json& seat, double now, const Json& config, const Json& apiConfig);
|
|
// Every seat with room for a launch now, best first (pointers into seats).
|
|
std::vector<Json*> Ranked(std::vector<Json>& seats, double now, const Json& config, const Json& apiConfig);
|
|
Json* Pick(std::vector<Json>& seats, double now, const Json& config, const Json& apiConfig);
|
|
|
|
std::filesystem::path SeatDir(const std::filesystem::path& scratch, std::string_view seatId);
|
|
// A pass interrupted by its seat's limit continues on another seat once
|
|
// its transcript is copied into that seat's config dir.
|
|
std::optional<std::filesystem::path> CarryTranscript(const std::filesystem::path& scratch, std::string_view fromSeat, std::string_view toSeat,
|
|
std::string_view sessionId, const std::function<void(const std::string&)>& log = {});
|
|
bool AllOut(const std::vector<Json>& seats);
|
|
std::optional<double> EarliestResume(const std::vector<Json>& seats, double now, std::optional<double> recheck = std::nullopt);
|
|
// A finished process onto its seat; true when the job must be placed again
|
|
// (the seat failed it, not the lens).
|
|
bool Settle(Json& seat, Json& record, double now, const Json& config, const Json& apiConfig, Ledger* ledger, const std::function<void(const std::string&)>& log);
|
|
double RetryAfter(std::string_view text, double fallback);
|
|
|
|
std::filesystem::path WriteCredentials(const std::filesystem::path& configDir, std::string_view token);
|
|
// The environment of a lens process on this seat.
|
|
std::map<std::string, std::string> ChildEnv(const Json& seat, const std::map<std::string, std::string>& base, const std::filesystem::path& scratch);
|
|
|
|
std::string ModelKey(std::string_view model);
|
|
// {catalogue id: prices or null} of the run's models.
|
|
Json FetchOpenRouterPrices(const std::vector<std::string>& models, const std::function<void(const std::string&)>& log = {}, double timeout = 20);
|
|
double OpenRouterCost(std::string_view model, const Json& usage);
|
|
Json ProbeOpenRouter(const std::string& secret, double timeout = 15);
|
|
|
|
// ---- several drivers at once: the board --------------------------------------
|
|
std::filesystem::path BoardPath();
|
|
// `<kind>:<12 hex of the secret's sha256>`, `local` for the login.
|
|
std::string SeatKey(std::string_view kind, std::string_view secret);
|
|
// This driver as the board names it: `<pid>:<start>`, the sandbox
|
|
// instance's in the Flatpak (Platform::Self).
|
|
std::string MyKey();
|
|
// Is the driver `key` alive? Its lifetime lock decides when it has one
|
|
// (a free one is taken away when `reap`); else natively its pid and start
|
|
// time, and in the sandbox yes.
|
|
bool DriverAlive(std::string_view key, const std::filesystem::path& boardDir, bool reap = true);
|
|
// One live driver per scratch dir: a `.driver` marker, taken over when its
|
|
// driver is dead.
|
|
bool ClaimScratch(const std::filesystem::path& path);
|
|
// Fold `other`'s readings into `into`, in place; the windows that changed.
|
|
std::vector<std::string> MergeWindows(Json& into, const Json& other, double now);
|
|
|
|
// The other drivers' passes in flight on this machine.
|
|
struct Others {
|
|
std::int64_t count = 0;
|
|
double mb = 0;
|
|
std::set<std::string> stacks;
|
|
std::int64_t unmeasured = 0;
|
|
};
|
|
|
|
class Board {
|
|
public:
|
|
explicit Board(std::filesystem::path path, std::string me = {}, std::optional<std::string> boot = std::nullopt,
|
|
std::function<void(const std::string&)> log = {});
|
|
// Is the driver `key` on the board alive: this driver; else not when
|
|
// it joined under another boot; else DriverAlive.
|
|
bool Alive(std::string_view key, const Json& drivers, bool reap = true) const;
|
|
// Drop what no longer stands: dead drivers with their in-flight
|
|
// entries, entries past their deadline, verdicts past their expiry
|
|
// or whose driver is gone, seat records with nothing live. (changed,
|
|
// the lines that say what went).
|
|
std::pair<bool, std::vector<std::string>> Sweep(Json& data, double now, bool reap = true) const;
|
|
void Join(const Json& info, double now);
|
|
// Everything of this driver off the board; idempotent, never throws.
|
|
void Leave();
|
|
void Publish(const std::vector<Json>& seats, double now, bool probed = false);
|
|
std::vector<std::string> Fold(std::vector<Json>& seats, double now);
|
|
static std::int64_t TotalInflight(const Json& data);
|
|
// The memory a gate's budget starts from, and how it was had.
|
|
std::pair<double, std::string> MemoryBase(std::int64_t free, double now);
|
|
using Chooser = std::function<std::optional<std::pair<Json*, Json*>>(const std::vector<Json*>& ranked, const Others& others)>;
|
|
struct Placed {
|
|
std::vector<Json*> ranked;
|
|
std::optional<std::pair<Json*, Json*>> picked;
|
|
std::int64_t full = 0;
|
|
};
|
|
// One locked section: the board folded, the seats ranked, the
|
|
// chooser's pick registered before the lock drops.
|
|
Placed Place(std::vector<Json>& seats, double now, const Json& config, const Json& apiConfig, const Chooser& chooser, std::optional<std::int64_t> cap = std::nullopt);
|
|
void Finish(const Json& seat, std::string_view label, double now);
|
|
// The board as it stands, swept in memory, nothing written.
|
|
Json View(double now) const;
|
|
std::vector<std::string> OthersLines(const std::vector<Json>& seats, double now, const Json& config, const Json& apiConfig, std::optional<std::int64_t> cap = std::nullopt) const;
|
|
const Others& OthersNow() const;
|
|
const std::string& Me() const;
|
|
const std::filesystem::path& Path() const;
|
|
const Json& Data() const;
|
|
|
|
private:
|
|
std::filesystem::path path;
|
|
std::string me;
|
|
std::string boot;
|
|
std::function<void(const std::string&)> log;
|
|
Json data;
|
|
Others others;
|
|
bool warned = false;
|
|
void Warn(const std::string& message);
|
|
Json Update(const std::function<bool(Json&)>& change);
|
|
Json& Record(Json& data, const Json& seat) const;
|
|
static bool StateHolds(const Json& state, const Json& data, double now);
|
|
Others OthersOf(const Json& data) const;
|
|
std::pair<bool, std::vector<std::string>> FoldInto(std::vector<Json>& seats, double now, Json& data);
|
|
bool PublishSeat(Json& data, const Json& seat, double now, bool probed) const;
|
|
};
|
|
|
|
// The seats as the dump records them: no secrets.
|
|
Json Summary(const std::vector<Json>& seats);
|
|
std::string Table(const std::vector<Json>& seats, std::optional<double> now = std::nullopt);
|
|
}
|