StripComments hand-scanned five states over raw characters and could not see
a raw string literal, which was documented as a v1 limitation. It was worse
than "does not recognise them": an odd number of quotes inside a raw string
desynchronised the scanner for the rest of the file. Simulated on the new
fixture, the old output was
auto banner = R" "hi)" \n \n ... "MARKER text"
— the code after the raw string blanked away, and the *contents* of a later
string literal left standing as if it were code. Every rule reading
CommentStripped() saw that. There are 33 raw strings across 6 files here,
including the two largest.
The replacement projects the token stream onto a copy of the buffer, blanking
comment tokens whole and the bodies of string/character literals. Editing a
copy in place makes the length-preserving contract structural rather than
something each branch has to remember, and the token extents make raw
strings, escapes, encoding prefixes and '"' all fall out for free. A raw
string reduces to R"…" so rules that bracket a literal by counting quotes
keep working. Digit separators (1'000) are excluded by requiring the text
before the quote to be an encoding prefix.
The public contract is unchanged, so no rule needed editing, and `lint` over
this repo produces byte-identical output to the previous implementation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Jorijn van der Graaf
04bc58b2fa