feat(lint): libclang-backed token layer
Adds LintContext::Tokens() and friends, backed by clang_tokenize, as the
substrate the rules will move onto. Nothing consumes it yet.
libclang is dlopen'd rather than linked: -lclang would break the mingw and
MSVC cross-builds at link time and would put a libclang.so.NN runtime
dependency into the otherwise self-contained release tarballs. The clang-c
header is used for its declarations only, and the function-pointer table is
typed with decltype so the signatures cannot drift from the real API.
Three properties this buys that the hand-rolled scanners could not have:
- a raw string literal or block comment is ONE token, so the documented
"raw string literals are not recognized" limitation goes away;
- `//` inside a literal is not a comment, so LineHasComment() replaces the
Line(n).contains("//") probes that false-positive on it;
- tokens cover preprocessor branches that are inactive for the host, since
clang_tokenize lexes rather than evaluates #if. Token rules therefore
keep seeing every platform's code, which an AST could not offer.
The parse backing the tokenizer is expected to fail on module units — no
PCMs, no build flags — and that is fine, because lexing has no semantic
prerequisites. Verified in the new tests.
LintSummary::Clean() now counts `errors`. It previously ignored them, so an
infrastructure failure that produced no findings reported clean and exited
0; a missing libclang would have been exactly that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
8892154b28
commit
5888af29ed
4 changed files with 387 additions and 2 deletions
|
|
@ -124,6 +124,31 @@ export namespace Crafter {
|
|||
std::unordered_map<std::size_t, std::unordered_set<std::string>> lineRules;
|
||||
};
|
||||
|
||||
// Lexical class of a LintToken, mirroring clang's token kinds one-to-one.
|
||||
enum class LintTokenKind {
|
||||
Punctuation,
|
||||
Keyword,
|
||||
Identifier,
|
||||
Literal, // string, raw string, character, integer, floating literal
|
||||
Comment, // // ... or /* ... */
|
||||
};
|
||||
|
||||
// One token from LintContext::Tokens(). `offset`/`length` are byte offsets
|
||||
// into LintContext::content and stay valid until the next SetContent.
|
||||
//
|
||||
// A raw string literal or a block comment is ONE token and may span lines,
|
||||
// which is exactly what the hand-rolled scanners could not represent. The
|
||||
// stream also covers text inside preprocessor branches that are inactive
|
||||
// for the host — clang_tokenize lexes, it does not evaluate #if — so token
|
||||
// rules see every platform's code, not just the one being built.
|
||||
struct LintToken {
|
||||
LintTokenKind kind = LintTokenKind::Punctuation;
|
||||
std::size_t offset = 0;
|
||||
std::size_t length = 0;
|
||||
std::size_t line = 0; // 1-based, of the token's first byte
|
||||
std::size_t column = 0; // 1-based, of the token's first byte
|
||||
};
|
||||
|
||||
// Per-file view handed to each LintRule's check callback. Every member
|
||||
// function is out-of-line and CRAFTER_API (defined in Crafter.Build:Lint's
|
||||
// implementation unit) because rule lambdas execute from the user's
|
||||
|
|
@ -165,11 +190,31 @@ export namespace Crafter {
|
|||
// re-parsed after SetContent.
|
||||
CRAFTER_API bool Suppressed(std::string_view rule, std::size_t line);
|
||||
|
||||
// The file lexed by clang, in source order. Built on first call, cached
|
||||
// per file, re-lexed after SetContent. Empty for extensions that are
|
||||
// not C or C++ (shaders): lexing GLSL as C++ would produce nonsense.
|
||||
//
|
||||
// Prefer this to scanning characters. It is the only view that gets
|
||||
// raw strings, line splices, digraphs and nested quoting right, and
|
||||
// offsets index straight into `content`, so a transform can find in
|
||||
// the token stream and edit in place.
|
||||
CRAFTER_API std::span<const LintToken> Tokens();
|
||||
// The token's own bytes: content.substr(tok.offset, tok.length).
|
||||
CRAFTER_API std::string_view TokenText(const LintToken& token) const;
|
||||
// Tokens whose FIRST byte is on `line` (1-based). A token that starts
|
||||
// earlier and spans into `line` — a raw string, a block comment — is
|
||||
// not included; ask Tokens() directly when that matters.
|
||||
CRAFTER_API std::span<const LintToken> TokensOnLine(std::size_t line);
|
||||
// True when a comment token starts on `line`. Replaces `Line(n).contains("//")`,
|
||||
// which false-positives on a `//` inside a string literal.
|
||||
CRAFTER_API bool LineHasComment(std::size_t line);
|
||||
|
||||
// Driver wiring — set by RunLint before each check call. Not for rules.
|
||||
std::string activeRule;
|
||||
std::vector<LintFinding>* sink = nullptr;
|
||||
std::optional<std::string> commentStrippedCache;
|
||||
std::optional<LintSuppressions> suppressionsCache;
|
||||
std::optional<std::vector<LintToken>> tokenCache;
|
||||
};
|
||||
|
||||
// A named lint rule: `check` runs once per (rule, file) over the project's
|
||||
|
|
|
|||
Loading…
Reference in a new issue