Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

refactor(lint): derive CommentStripped from tokens, drop the char scanner

Browse source 
StripComments hand-scanned five states over raw characters and could not see
a raw string literal, which was documented as a v1 limitation. It was worse
than "does not recognise them": an odd number of quotes inside a raw string
desynchronised the scanner for the rest of the file. Simulated on the new
fixture, the old output was

    auto banner = R"         "hi)" \n                 \n ...  "MARKER text"

— the code after the raw string blanked away, and the *contents* of a later
string literal left standing as if it were code. Every rule reading
CommentStripped() saw that. There are 33 raw strings across 6 files here,
including the two largest.

The replacement projects the token stream onto a copy of the buffer, blanking
comment tokens whole and the bodies of string/character literals. Editing a
copy in place makes the length-preserving contract structural rather than
something each branch has to remember, and the token extents make raw
strings, escapes, encoding prefixes and '"' all fall out for free. A raw
string reduces to R"…" so rules that bracket a literal by counting quotes
keep working. Digit separators (1'000) are excluded by requiring the text
before the quote to be an encoding prefix.

The public contract is unchanged, so no rule needed editing, and `lint` over
this repo produces byte-identical output to the previous implementation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-07-27 02:59:11 +02:00
commit 04bc58b2fa
3 changed files with 81 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

105
implementations/Crafter.Build-Lint.cpp
View file

@ -215,72 +215,45 @@ namespace {
return tokens;
}
// Blank //-comments, /*...*/ comments and string/char literal bodies to
// spaces while copying '\n' through, so byte offsets and line numbers in
// the result match the original text. Raw string literals are not
// recognized (documented v1 limitation) — their bodies pass through as
// ordinary string content until the first '"'.
std::string StripComments(std::string_view src) {
enum class State { Code, LineComment, BlockComment, String, Char };
std::string out;
out.reserve(src.size());
State state = State::Code;
for (std::size_t i = 0; i < src.size(); ++i) {
char c = src[i];
char next = i + 1 < src.size() ? src[i + 1] : '\0';
switch (state) {
case State::Code:
if (c == '/' && next == '/') {
state = State::LineComment;
out += " ";
++i;
} else if (c == '/' && next == '*') {
state = State::BlockComment;
out += " ";
++i;
} else if (c == '"') {
state = State::String;
out += c; // keep the delimiter so quoting stays visible
} else if (c == '\'') {
state = State::Char;
out += c;
} else {
out += c;
}
break;
case State::LineComment:
if (c == '\n') {
state = State::Code;
out += c;
} else {
out += ' ';
}
break;
case State::BlockComment:
if (c == '*' && next == '/') {
state = State::Code;
out += " ";
++i;
} else {
out += c == '\n' ? '\n' : ' ';
}
break;
case State::String:
case State::Char: {
char delim = state == State::String ? '"' : '\'';
if (c == '\\' && next != '\0') {
out += " ";
++i;
if (next == '\n') out.back() = '\n';
} else if (c == delim) {
state = State::Code;
out += c;
} else {
out += c == '\n' ? '\n' : ' ';
}
break;
}
// Blank comments and the bodies of string/character literals to spaces,
// copying '\n' through so byte offsets and line numbers in the result
// match the original text exactly.
//
// Derived from the token stream rather than scanned character by
// character, which is what makes raw strings, escapes, encoding prefixes
// and a literal like '"' come out right. Editing a copy of the buffer in
// place — rather than appending to a fresh string — makes the
// length-preserving property structural instead of something every branch
// has to remember.
std::string StripLiterals(const std::string& content, std::span<const LintToken> tokens) {
std::string out = content;
auto blank = [&out](std::size_t from, std::size_t to) {
for (std::size_t i = from; i < to && i < out.size(); ++i) {
if (out[i] != '\n') out[i] = ' ';
}
};
for (const LintToken& token : tokens) {
std::size_t begin = token.offset;
std::size_t end = token.offset + token.length;
if (token.kind == LintTokenKind::Comment) {
blank(begin, end);
continue;
}
if (token.kind != LintTokenKind::Literal || token.length < 2) continue;
std::string_view text(content.data() + begin, token.length);
// Numeric literals are code and stay; only string and character
// literals have a body to hide. A digit separator makes 1'000 look
// quote-ish, so require everything before the quote to be an
// encoding prefix (L, u, U, u8, R and their combinations).
std::size_t quote = text.find_first_of("\"'");
if (quote == std::string_view::npos) continue;
std::string_view prefix = text.substr(0, quote);
if (!std::ranges::all_of(prefix, [](char c) { return c == 'L' || c == 'u' || c == 'U' || c == '8' || c == 'R'; })) continue;
// Keep the opening quote and the closing one, blank everything
// between. For a raw string that also blanks the R"delim( and
// )delim" scaffolding, leaving exactly two quotes — which is what
// rules counting quotes to find a literal's extent rely on.
blank(begin + quote + 1, end - 1);
}
return out;
}
@ -362,7 +335,7 @@ std::string_view LintContext::Line(std::size_t n) const {
const std::string& LintContext::CommentStripped() {
if (!commentStrippedCache) {
commentStrippedCache = StripComments(content);
commentStrippedCache = StripLiterals(content, Tokens());
}
return *commentStrippedCache;
}