Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

refactor(lint): derive CommentStripped from tokens, drop the char scanner

Browse source 
StripComments hand-scanned five states over raw characters and could not see
a raw string literal, which was documented as a v1 limitation. It was worse
than "does not recognise them": an odd number of quotes inside a raw string
desynchronised the scanner for the rest of the file. Simulated on the new
fixture, the old output was

    auto banner = R"         "hi)" \n                 \n ...  "MARKER text"

— the code after the raw string blanked away, and the *contents* of a later
string literal left standing as if it were code. Every rule reading
CommentStripped() saw that. There are 33 raw strings across 6 files here,
including the two largest.

The replacement projects the token stream onto a copy of the buffer, blanking
comment tokens whole and the bodies of string/character literals. Editing a
copy in place makes the length-preserving contract structural rather than
something each branch has to remember, and the token extents make raw
strings, escapes, encoding prefixes and '"' all fall out for free. A raw
string reduces to R"…" so rules that bracket a literal by counting quotes
keep working. Digit separators (1'000) are excluded by requiring the text
before the quote to be an encoding prefix.

The public contract is unchanged, so no rule needed editing, and `lint` over
this repo produces byte-identical output to the previous implementation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-07-27 02:59:11 +02:00
commit 04bc58b2fa
3 changed files with 81 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

34
tests/Lint/main.cpp
View file

@ -504,8 +504,38 @@ int main() {
ctx.SetContent("int replaced;\n");
std::span<const LintToken> after = ctx.Tokens();
Check(!after.empty() && ctx.TokenText(after.front()) == "int", "tokens: re-lexed after SetContent");
Check(std::ranges::none_of(after, [&](const LintToken& t) { return ctx.TokenText(t) == "HiddenBranch"; }),
"tokens: stale tokens are dropped after SetContent");
Check(std::ranges::none_of(after, [&](const LintToken& t) { return ctx.TokenText(t) == "HiddenBranch"; }), "tokens: stale tokens are dropped after SetContent");
});
RunLint(cfg, Mode(LintMode::Report));
}
// CommentStripped over a raw string holding an ODD number of quotes. The
// character-scanning version treated R"( as an ordinary string open, so the
// quote inside the body closed it early and every following line was
// swallowed as literal text — code after the raw string vanished from the
// stripped view. Lexing gets the extent right.
{
constexpr std::string_view Source =
"auto banner = R\"(he said \"hi)\";\n" // 1: one quote inside the body
"int afterRaw = 2;\n" // 2: must survive as code
"// MARKER comment\n" // 3
"auto plain = \"MARKER text\";\n"; // 4
Scratch s("strip-rawstring");
s.Write("f", Source);
Configuration cfg = s.Config({"f"});
cfg.AddLintRule("strip", [](LintContext& ctx) {
const std::string& code = ctx.CommentStripped();
Check(code.size() == ctx.content.size(), "strip: byte length preserved");
Check(std::ranges::count(code, '\n') == std::ranges::count(ctx.content, '\n'), "strip: newlines preserved");
Check(code.contains("afterRaw"), "strip: code after an odd-quoted raw string survives");
Check(!code.contains("he said"), "strip: raw string body is blanked");
Check(!code.contains("MARKER"), "strip: comment and literal bodies are blanked");
Check(code.contains("auto plain ="), "strip: code around a literal survives");
// The raw string collapses to R"…" — exactly two quotes, so rules
// that bracket a literal by counting quotes still work.
std::string_view line1 = std::string_view(code).substr(0, code.find('\n'));
Check(std::ranges::count(line1, '"') == 2, "strip: raw string leaves exactly two quotes");
});
RunLint(cfg, Mode(LintMode::Report));
}