Catcrafts/Crafter.Review
Watch
2
0
Fork
You've already forked Crafter.Review
 0
Code Issues 7 Pull requests Projects Releases 3 Packages Wiki Activity Actions

feat(pipeline): the pure pipeline, checked against its goldens

Browse source 
The modules that turn a frozen range into passes and passes into a
verdict: :Schema, :Tools (the DENY, git and base lists, a leaf, so the
config and the pool both read them), :Plan, :Artifact, :Inventory,
:Dispositions, :State, :Normalize, :Stream, :Prompt, :Config, and the URL
half of :PgWire. Units, jobs, results, the config and the claude stream
stay JSON from end to end, so every key keeps its place; :Text gives JSON
values truthiness, get, ==, membership and Str(), and throws Dyn::Error
under a name for each kind of unexpected data.

Exactness where the bytes are seen:
- the stream classifiers are hand-written scanners with case-insensitive
  folds (the Kelvin sign is a k), Unicode \b and \s, and `.` stopping at
  \n; the SQL call detection takes `<exe> sql` with any path before it;
- config problems carry the JSON scanner's and the UTF-8 decoder's own
  texts, positions included;
- a float total is Neumaier's compensated sum, and the build passes
  -ffp-contract=off: -march=native would fuse a*b+c;
- `crafter-review inventory` and `config` print what the goldens record,
  byte for byte (the fixture and 14 configs, broken ones included); a
  message names this binary as the command to run.

tests/Goldens: the recorded answers over the fixtures next to them (the
commits fixture, a rich versioned series with CRLF, a folded subject,
Markdown headings in and out of fences, the code-key rewrites); the suite
compares 2304 results as DumpFile writes them, so a key out of order fails
too. All agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-31 06:48:55 +01:00
commit 741e3a656d
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
70 changed files with 30565 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

293
implementations/Crafter.Review-Inventory.cpp Normal file
View file

@ -0,0 +1,293 @@
// SPDX-License-Identifier: AGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
module Crafter.Review:Inventory_impl;
import :Inventory;
import :Json;
import :Text;
import :Artifact;
import :Plan;
import std;
namespace fs = std::filesystem;
namespace Crafter::Review::Inventory {
namespace {
constexpr std::string_view Usage = R"(Partition a frozen commit range (a `git format-patch` export) into review units for the coverage contract.
The inventory is MECHANICAL: no agent decides what the units are. Every non-blank line of the artifact belongs to
exactly one unit, and the tool fails if that invariant does not hold. Fixed slots are added so that what is NOT in
the text still has a row in the coverage matrix.
crafter-review inventory commits <dir-with-*.patch> > gate-state/inventory-<hash8>.json
Units (ids):
pK:subject commit K's subject
pK:msg commit K's message body (before the trailers) - only when non-empty
pK:trailers commit K's trailer block - only when non-empty
pK:diffstat commit K's diffstat
pK:<file>#hdr the diff header of <file> in commit K (mode, rename, blobs)
pK:<file>#H hunk H of <file> in commit K; a hunk of a documentation file (DOC_FILE) that holds more than one
heading-delimited part is cut into #H.1, #H.2, ... (doc_parts)
pK:outside fixed slot: the in-repository users/callers of what commit K changes (outside its diff)
pK:absent fixed slot: what a reviewer expects in commit K and does not find
whole:* fixed slots (WHOLE_COMMITS)
)";
// re.match(r'\s{0,3}(```|~~~)', body): a fence, indented by three
// whitespace characters at most.
bool OpensFence(std::string_view body) {
std::size_t at = 0;
for (std::int32_t i = 0; i < 3 && at < body.size(); ++i) {
const auto [c, used] = Dyn::CodePointAt(body, at);
if (!Dyn::IsSpace(c)) break;
at += used;
}
const std::string_view rest = body.substr(at);
return rest.starts_with("```") || rest.starts_with("~~~");
}
// re.match(r'#{1,6}\s', body): an ATX heading.
bool IsHeading(std::string_view body) {
std::size_t hashes = 0;
while (hashes < body.size() && body[hashes] == '#') ++hashes;
if (hashes == 0 || hashes > 6 || hashes >= body.size()) return false;
return Dyn::IsSpace(Dyn::CodePointAt(body, hashes).first);
}
// re.match(r'^[A-Z][A-Za-z-]+: ', line): a trailer such as
// `Signed-off-by: `.
bool IsTrailer(std::string_view line) {
if (line.empty() || line[0] < 'A' || line[0] > 'Z') return false;
std::size_t at = 1;
while (at < line.size() && ((line[at] >= 'A' && line[at] <= 'Z') || (line[at] >= 'a' && line[at] <= 'z') || line[at] == '-')) ++at;
return at > 1 && line.substr(at).starts_with(": ");
}
bool AnyNonBlank(std::span<const std::string> lines, std::int64_t from, std::int64_t to) {
const auto [a, b] = Dyn::SliceBounds(lines.size(), from, to);
return std::any_of(lines.begin() + static_cast<std::ptrdiff_t>(a), lines.begin() + static_cast<std::ptrdiff_t>(b),
[](const std::string& line) { return !Dyn::Strip(line).empty(); });
}
std::span<const std::string> Lines(std::span<const std::string> lines, std::int64_t from, std::int64_t to) {
const auto [a, b] = Dyn::SliceBounds(lines.size(), from, to);
return lines.subspan(a, b - a);
}
Json Unit(std::string id, std::int64_t first, std::int64_t last, std::string excerpt) {
return {{"id", std::move(id)}, {"lines", {first, last}}, {"excerpt", std::move(excerpt)}};
}
std::string ReprInts(std::span<const std::int64_t> values) {
std::string out = "[";
for (std::size_t i = 0; i < values.size(); ++i) out += std::format("{}{}", i ? ", " : "", values[i]);
return std::format("{}]", out);
}
}
std::vector<std::pair<std::size_t, std::size_t>> DocParts(std::span<const std::string> lines, std::size_t a, std::size_t b) {
std::vector<std::pair<std::size_t, std::size_t>> parts;
std::size_t start = a;
bool hasBody = false;
bool fenced = false;
for (std::size_t i = a + 1; i < b; ++i) {
const std::string& line = lines[i];
// l[:1] in (' ', '+') and l[1:]: the marker and the line after it.
const bool isNew = line.starts_with(' ') || line.starts_with('+');
const std::string_view body = line.empty() ? std::string_view() : std::string_view(line).substr(Dyn::CodePointAt(line, 0).second);
if (isNew && OpensFence(body)) fenced = !fenced;
if (isNew && !fenced && IsHeading(body)) {
if (hasBody) {
parts.emplace_back(start, i);
start = i;
hasBody = false;
}
} else if (!Dyn::Strip(line).empty() && !Dyn::Strip(body).empty()) {
hasBody = true;
}
}
parts.emplace_back(start, b);
return parts;
}
std::string Excerpt(std::span<const std::string> lines) {
for (const std::string& line : lines) {
const std::string stripped = Dyn::Strip(line);
if (!stripped.empty()) return Dyn::Slice(stripped, 0, 90);
}
return "";
}
Series PartitionSeries(const fs::path& dir, bool splitDocs) {
Series series;
series.units = Json::array();
Json& units = series.units;
std::int64_t offset = 0;
for (const std::string& name : Artifact::PatchFiles(dir)) {
const std::string text = Dyn::ReadText(dir / name);
const std::vector<std::string> lines = Dyn::Split(text, "\n");
const auto count = static_cast<std::int64_t>(lines.size());
const std::optional<std::int64_t> number = Artifact::PatchNumber(name);
const std::string pid = std::format("p{}", number ? *number : -1);
const std::int64_t base = offset;
std::optional<std::int64_t> subject;
for (std::int64_t i = 0; i < count; ++i) {
if (lines[static_cast<std::size_t>(i)].starts_with("Subject:")) {
subject = i;
break;
}
}
std::int64_t body0 = 1;
if (subject) {
for (std::int64_t i = *subject + 1; i < count; ++i) {
if (Dyn::Strip(lines[static_cast<std::size_t>(i)]).empty()) {
body0 = i + 1;
break;
}
}
}
// >= : a body-less commit's first body line IS the '---'.
std::int64_t diff0 = count;
for (std::int64_t i = body0; i < count; ++i) {
if (lines[static_cast<std::size_t>(i)].starts_with("---")) {
diff0 = i;
break;
}
}
units.push_back(Unit(std::format("{}:subject", pid), base + 1, base + body0,
subject ? Dyn::Slice(Dyn::Strip(lines[static_cast<std::size_t>(*subject)]), 0, 90) : std::string()));
std::int64_t t = diff0;
while (t - 1 > body0 && (IsTrailer(lines[static_cast<std::size_t>(t - 1)]) || Dyn::Strip(lines[static_cast<std::size_t>(t - 1)]).empty())) --t;
if (AnyNonBlank(lines, body0, t)) {
units.push_back(Unit(std::format("{}:msg", pid), base + body0 + 1, base + t, Excerpt(Lines(lines, body0, t))));
}
if (AnyNonBlank(lines, t, diff0)) {
units.push_back(Unit(std::format("{}:trailers", pid), base + t + 1, base + diff0, Excerpt(Lines(lines, t, diff0))));
}
const std::int64_t statStart = diff0;
std::int64_t firstDiff = count;
for (std::int64_t i = diff0 + 1; i < count; ++i) {
if (lines[static_cast<std::size_t>(i)].starts_with("diff --git")) {
firstDiff = i;
break;
}
}
units.push_back(Unit(std::format("{}:diffstat", pid), base + statStart + 1, base + firstDiff, Excerpt(Lines(lines, statStart, firstDiff))));
std::string currentFile;
std::int64_t hunk = 0;
std::optional<std::int64_t> hunkStart;
auto closeHunk = [&](std::int64_t upto) {
if (!hunkStart) return;
std::vector<std::pair<std::size_t, std::size_t>> parts;
if (splitDocs && Plan::IsDocPath(currentFile)) {
parts = DocParts(lines, static_cast<std::size_t>(*hunkStart), static_cast<std::size_t>(upto));
} else {
parts = {{static_cast<std::size_t>(*hunkStart), static_cast<std::size_t>(upto)}};
}
for (std::size_t n = 0; n < parts.size(); ++n) {
const auto [s, e] = parts[n];
const std::string suffix = parts.size() > 1 ? std::format(".{}", n + 1) : std::string();
units.push_back(Unit(std::format("{}:{}#{}{}", pid, currentFile, hunk, suffix), base + static_cast<std::int64_t>(s) + 1,
base + static_cast<std::int64_t>(e), Excerpt(Lines(lines, static_cast<std::int64_t>(s), static_cast<std::int64_t>(s) + 1))));
}
};
std::int64_t i = firstDiff;
while (i < count) {
const std::string& line = lines[static_cast<std::size_t>(i)];
if (line.starts_with("diff --git")) {
closeHunk(i);
hunkStart.reset();
currentFile = Dyn::Strip(Dyn::Split(line, " b/").back());
hunk = 0;
const std::int64_t header = i;
std::int64_t j = i + 1;
while (j < count && !lines[static_cast<std::size_t>(j)].starts_with("@@") && !lines[static_cast<std::size_t>(j)].starts_with("diff --git")) ++j;
units.push_back(Unit(std::format("{}:{}#hdr", pid, currentFile), base + header + 1, base + j, Dyn::Slice(Dyn::Strip(line), 0, 90)));
i = j;
continue;
}
if (line.starts_with("@@")) {
closeHunk(i);
++hunk;
hunkStart = i;
}
++i;
}
closeHunk(count);
units.push_back(Unit(std::format("{}:outside", pid), 0, 0, "FIXED SLOT: in-repository users/callers of what this commit changes, outside its diff"));
units.push_back(Unit(std::format("{}:absent", pid), 0, 0, "FIXED SLOT: what a reviewer expects in this commit and does not find"));
series.lines.insert(series.lines.end(), lines.begin(), lines.end());
offset += count;
}
return series;
}
Built Build(std::string_view mode, std::string_view artifact) {
if (mode != "commits") throw InvariantError(std::format("inventory: mode {} is not supported here (commits only)", Dyn::Repr(mode)));
Series series = PartitionSeries(fs::path(artifact), true);
Json& units = series.units;
const std::string h8 = Artifact::ArtifactHash("commits", fs::path(artifact)).substr(0, 8);
std::map<std::int64_t, std::string> owner;
for (const Json& unit : units) {
const std::int64_t a = Dyn::At(unit, "lines")[0].get<std::int64_t>();
const std::int64_t b = Dyn::At(unit, "lines")[1].get<std::int64_t>();
const std::string& id = Dyn::At(unit, "id").get_ref<const std::string&>();
for (std::int64_t line = a; line <= b; ++line) {
if (line == 0) continue;
if (const auto held = owner.find(line); held != owner.end()) {
throw InvariantError(std::format("INVARIANT: line {} in both {} and {}", line, held->second, id));
}
owner[line] = id;
}
}
std::vector<std::int64_t> uncovered;
for (std::size_t i = 0; i < series.lines.size(); ++i) {
const auto line = static_cast<std::int64_t>(i) + 1;
if (!Dyn::Strip(series.lines[i]).empty() && !owner.contains(line)) uncovered.push_back(line);
}
if (!uncovered.empty()) {
const std::span<const std::int64_t> first(uncovered.data(), std::min<std::size_t>(uncovered.size(), 10));
throw InvariantError(std::format("INVARIANT: {} non-blank lines in no unit, first: {}", uncovered.size(), ReprInts(first)));
}
for (const WholeSlot& slot : WholeCommits) units.push_back(Unit(std::string(slot.id), 0, 0, std::format("FIXED SLOT: {}", slot.description)));
std::map<std::string, std::int32_t> seen;
for (const Json& unit : units) ++seen[Dyn::At(unit, "id").get<std::string>()];
Json duplicates = Json::array();
for (const auto& [id, times] : seen) {
if (times > 1) duplicates.push_back(id);
}
if (!duplicates.empty()) throw InvariantError(std::format("INVARIANT: duplicate ids {}", Dyn::ReprValue(duplicates)));
const std::size_t total = units.size();
Built built;
built.out = {
{"mode", mode},
{"artifact", artifact},
{"hash8", h8},
{"units", std::move(units)},
{"invariant", "every non-blank line belongs to exactly one unit; fixed slots have lines [0,0]"},
{"count", total},
};
built.lines = std::move(series.lines);
return built;
}
std::int32_t Main(std::span<const std::string_view> args) {
if (args.size() != 2 || args[0] != "commits") {
std::print(std::cerr, "{}", Usage);
return 1;
}
try {
const Built built = Build(args[0], args[1]);
std::println("{}", DumpFile(built.out));
return 0;
} catch (const InvariantError& error) {
std::println(std::cerr, "{}", error.what());
return 1;
} catch (const std::exception& error) {
std::println(std::cerr, "inventory: {}", error.what());
return 1;
}
}
}