Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

Merge branch 'lint-ast': libclang token layer for the linter

Browse source 
Replaces the linter's hand-rolled character scanning with clang's lexer.

StripComments could not see raw string literals, which was documented as a
v1 limitation but was worse than that: an odd number of quotes inside a raw
string desynchronised it for the rest of the file, hiding real code and
leaving the contents of later literals standing as if they were code. There
are 33 raw strings across 6 files here, including the two largest.

The reflow guards had the same class of bug from the other direction. They
were substring probes, so Line(n).contains("//") fired on // inside a string
literal and contains("R\"") fired on the characters R" inside a literal while
missing raw strings opened on an earlier line. Both misfire on this repo:
"MARKER" ends in R". They are now LineHasComment() and
LineHasMultiLineToken(), and two wrapped call sites in tests/Lint that had
been silently unjoinable join as a result.

libclang is dlopen'd rather than linked, so the mingw and MSVC cross-builds
keep linking and the release tarballs stay self-contained. Tokens need no
PCMs, no build flags and no prior build, and they cover preprocessor branches
that are inactive for the host — which is why the layout rules belong on
tokens rather than on an AST that would only ever see one platform's slice.

No rule behaviour was intended to change beyond the two guard fixes; lint
over this repo produced byte-identical output across the CommentStripped
swap.
This commit is contained in:
Jorijn van der Graaf 2026-07-30 22:57:10 +02:00
commit 5fa9c8a816
6 changed files with 553 additions and 88 deletions
Download patch file Download diff file Expand all files Collapse all files

70
interfaces/Crafter.Build-Clang.cppm
View file

@ -124,6 +124,31 @@ export namespace Crafter {
std::unordered_map<std::size_t, std::unordered_set<std::string>> lineRules;
};
// Lexical class of a LintToken, mirroring clang's token kinds one-to-one.
enum class LintTokenKind {
Punctuation,
Keyword,
Identifier,
Literal, // string, raw string, character, integer, floating literal
Comment, // // ... or /* ... */
};
// One token from LintContext::Tokens(). `offset`/`length` are byte offsets
// into LintContext::content and stay valid until the next SetContent.
//
// A raw string literal or a block comment is ONE token and may span lines,
// which is exactly what the hand-rolled scanners could not represent. The
// stream also covers text inside preprocessor branches that are inactive
// for the host — clang_tokenize lexes, it does not evaluate #if — so token
// rules see every platform's code, not just the one being built.
struct LintToken {
LintTokenKind kind = LintTokenKind::Punctuation;
std::size_t offset = 0;
std::size_t length = 0;
std::size_t line = 0; // 1-based, of the token's first byte
std::size_t column = 0; // 1-based, of the token's first byte
};
// Per-file view handed to each LintRule's check callback. Every member
// function is out-of-line and CRAFTER_API (defined in Crafter.Build:Lint's
// implementation unit) because rule lambdas execute from the user's
@ -136,10 +161,16 @@ export namespace Crafter {
CRAFTER_API std::string Extension() const; // ".cppm", ".cpp", ".h", ...
CRAFTER_API std::string_view Line(std::size_t n) const; // 1-based; empty if out of range
// `content` with //-comments, /*...*/ comments and string/char literal
// bodies blanked to spaces, newlines preserved — offsets and line
// numbers stay valid. Built on first call, cached per file. Raw string
// literals are not recognized (v1 limitation).
// `content` with comments and string/char literal bodies blanked to
// spaces, newlines preserved — offsets and line numbers stay valid.
// Built from Tokens() on first call, cached per file, so raw strings,
// escapes, encoding prefixes and literals like '"' all come out right.
// A raw string is reduced to R"…" with the body and the delimiter
// scaffolding blanked, leaving exactly two quote characters.
//
// Convenient for a quick scan, but Tokens() is the better tool for
// anything structural: this view cannot tell an identifier from a
// keyword, and it has already thrown away where the literals were.
CRAFTER_API const std::string& CommentStripped();
// Record a finding at `line` (1-based; pass 0 for a whole-file finding).
CRAFTER_API void Report(std::size_t line, std::string message);
@ -165,11 +196,42 @@ export namespace Crafter {
// re-parsed after SetContent.
CRAFTER_API bool Suppressed(std::string_view rule, std::size_t line);
// The file lexed by clang, in source order. Built on first call, cached
// per file, re-lexed after SetContent. Empty for extensions that are
// not C or C++ (shaders): lexing GLSL as C++ would produce nonsense.
//
// Prefer this to scanning characters. It is the only view that gets
// raw strings, line splices, digraphs and nested quoting right, and
// offsets index straight into `content`, so a transform can find in
// the token stream and edit in place.
CRAFTER_API std::span<const LintToken> Tokens();
// The token's own bytes: content.substr(tok.offset, tok.length).
CRAFTER_API std::string_view TokenText(const LintToken& token) const;
// Tokens whose FIRST byte is on `line` (1-based). A token that starts
// earlier and spans into `line` — a raw string, a block comment — is
// not included; ask Tokens() directly when that matters.
CRAFTER_API std::span<const LintToken> TokensOnLine(std::size_t line);
// True when a comment token starts on `line`. Replaces `Line(n).contains("//")`,
// which false-positives on a `//` inside a string literal.
CRAFTER_API bool LineHasComment(std::size_t line);
// True when `line` (1-based) is touched by a token that spans more
// than one line — a raw string or a block comment. A transform that
// joins, splits or rewrites such a line changes what is inside that
// token, so this is the guard to consult before reflowing anything.
// Replaces `Line(n).contains("R\"")`, which both misses raw strings
// opened on an earlier line and fires on the characters R" appearing
// inside an ordinary literal.
CRAFTER_API bool LineHasMultiLineToken(std::size_t line);
// Driver wiring — set by RunLint before each check call. Not for rules.
std::string activeRule;
std::vector<LintFinding>* sink = nullptr;
std::optional<std::string> commentStrippedCache;
std::optional<LintSuppressions> suppressionsCache;
std::optional<std::vector<LintToken>> tokenCache;
// Per-line flag, 0-based, for LineHasMultiLineToken. Derived from
// tokenCache and invalidated with it.
std::optional<std::vector<bool>> spannedLineCache;
};
// A named lint rule: `check` runs once per (rule, file) over the project's

7
interfaces/Crafter.Build-Lint.cppm
View file

@ -39,10 +39,13 @@ export namespace Crafter {
std::vector<std::filesystem::path> changedFiles;
std::size_t filesLinted = 0;
std::size_t rulesRun = 0; // rules remaining after glob filter
std::size_t errors = 0; // rule exceptions + write failures
// Rule exceptions, write failures, and infrastructure failures such as
// libclang not loading. Counted in Clean() so a run that could not do
// its job never looks like a run that found nothing.
std::size_t errors = 0;
bool noRulesDefined = false; // project registered no rules at all
// Host-side only (like TestSummary::AllPassed), safe as in-class inline.
bool Clean() const { return findings.empty() && !noRulesDefined; }
bool Clean() const { return findings.empty() && !noRulesDefined && errors == 0; }
};
// Run the project's lint rules over its own sources: module interfaces