Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

feat(lint): libclang-backed token layer

Browse source 
Adds LintContext::Tokens() and friends, backed by clang_tokenize, as the
substrate the rules will move onto. Nothing consumes it yet.

libclang is dlopen'd rather than linked: -lclang would break the mingw and
MSVC cross-builds at link time and would put a libclang.so.NN runtime
dependency into the otherwise self-contained release tarballs. The clang-c
header is used for its declarations only, and the function-pointer table is
typed with decltype so the signatures cannot drift from the real API.

Three properties this buys that the hand-rolled scanners could not have:

  - a raw string literal or block comment is ONE token, so the documented
    "raw string literals are not recognized" limitation goes away;
  - `//` inside a literal is not a comment, so LineHasComment() replaces the
    Line(n).contains("//") probes that false-positive on it;
  - tokens cover preprocessor branches that are inactive for the host, since
    clang_tokenize lexes rather than evaluates #if. Token rules therefore
    keep seeing every platform's code, which an AST could not offer.

The parse backing the tokenizer is expected to fail on module units — no
PCMs, no build flags — and that is fine, because lexing has no semantic
prerequisites. Verified in the new tests.

LintSummary::Clean() now counts `errors`. It previously ignored them, so an
infrastructure failure that produced no findings reported clean and exited
0; a missing libclang would have been exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-07-27 02:54:38 +02:00
commit 5888af29ed
4 changed files with 387 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

100
tests/Lint/main.cpp
View file

@ -427,6 +427,106 @@ int main() {
Check(s.Read("a") == "// lint-disable-file flag\nbad\n", "other rules still format");
}
// ---------------- token layer ----------------
//
// The source is spelled with escaped literals rather than a raw string so
// that this file stays lintable by the very rules under test; the scratch
// file it writes does contain a genuine multi-line raw string.
{
constexpr std::string_view Source =
"#ifdef CRAFTER_LINT_NEVER_DEFINED\n" // 1
"void HiddenBranch();\n" // 2
"#endif\n" // 3
"// a real comment\n" // 4
"int url = 1; // https://example.com\n" // 5
"auto raw = R\"raw(spans lines\n" // 6
" // not a comment\n" // 7
" int notADecl;\n" // 8
")raw\";\n"; // 9
Scratch s("tokens");
s.Write("f", Source);
Configuration cfg = s.Config({"f"});
cfg.AddLintRule("tokens", [](LintContext& ctx) {
std::span<const LintToken> toks = ctx.Tokens();
Check(!toks.empty(), "tokens: file lexes to a non-empty stream");
// Every token's offset/length must address its own bytes, or a
// transform editing at an offset would corrupt the file.
bool offsetsSound = true;
for (const LintToken& t : toks) {
if (t.offset + t.length > ctx.content.size() || ctx.TokenText(t).empty()) offsetsSound = false;
}
Check(offsetsSound, "tokens: every offset/length addresses real bytes");
// Ordered by offset, so binary search in TokensOnLine is valid.
bool ordered = std::ranges::is_sorted(toks, {}, &LintToken::offset);
Check(ordered, "tokens: stream is in source order");
// Inactive #ifdef branch is still lexed — this is what keeps token
// rules covering every platform, unlike an AST.
bool sawHidden = std::ranges::any_of(toks, [&](const LintToken& t) {
return t.kind == LintTokenKind::Identifier && ctx.TokenText(t) == "HiddenBranch";
});
Check(sawHidden, "tokens: inactive #ifdef branch is lexed");
// A multi-line raw string is exactly one Literal, comment markers
// and declarations inside it included.
auto isRaw = [&](const LintToken& t) { return ctx.TokenText(t).starts_with("R\"raw("); };
Check(std::ranges::count_if(toks, isRaw) == 1, "tokens: raw string is a single token");
auto raw = std::ranges::find_if(toks, isRaw);
if (raw != toks.end()) {
Check(raw->kind == LintTokenKind::Literal, "tokens: raw string is a Literal");
Check(raw->line == 6, "tokens: raw string starts on line 6");
Check(ctx.TokenText(*raw).contains("// not a comment"), "tokens: raw string body kept intact");
Check(ctx.TokenText(*raw).ends_with(")raw\""), "tokens: raw string spans to its own terminator");
}
// The only comments are the two real ones on lines 4 and 5 — the
// `//` on line 7 lives inside the raw string.
std::vector<std::size_t> commentLines;
for (const LintToken& t : toks) {
if (t.kind == LintTokenKind::Comment) commentLines.push_back(t.line);
}
Check(commentLines == std::vector<std::size_t>{4, 5}, "tokens: only real comments are Comment tokens");
Check(ctx.LineHasComment(4), "tokens: LineHasComment finds a whole-line comment");
Check(ctx.LineHasComment(5), "tokens: LineHasComment finds a trailing comment");
Check(!ctx.LineHasComment(7), "tokens: `//` inside a raw string is not a comment");
Check(!ctx.LineHasComment(2), "tokens: code-only line has no comment");
// TokensOnLine brackets by starting line.
std::span<const LintToken> line2 = ctx.TokensOnLine(2);
Check(!line2.empty() && ctx.TokenText(line2.front()) == "void", "tokens: TokensOnLine starts at the line's first token");
Check(std::ranges::all_of(line2, [](const LintToken& t) { return t.line == 2; }), "tokens: TokensOnLine stays on its line");
// SetContent must invalidate the cache, or offsets point into a
// buffer that no longer exists.
ctx.SetContent("int replaced;\n");
std::span<const LintToken> after = ctx.Tokens();
Check(!after.empty() && ctx.TokenText(after.front()) == "int", "tokens: re-lexed after SetContent");
Check(std::ranges::none_of(after, [&](const LintToken& t) { return ctx.TokenText(t) == "HiddenBranch"; }),
"tokens: stale tokens are dropped after SetContent");
});
RunLint(cfg, Mode(LintMode::Report));
}
// Non-C++ extensions are not lexed: GLSL through a C++ lexer would produce
// plausible-looking nonsense rather than an honest refusal.
{
Scratch s("tokens-foreign");
fs::path shader = s.dir / "f.frag";
{
std::ofstream f(shader, std::ios::binary | std::ios::trunc);
f << "#version 450\nvoid main() { }\n";
}
Configuration cfg = s.Config({});
cfg.shaders.emplace_back(fs::path(shader), "main", ShaderType::Fragment);
cfg.AddLintRule("no-lex", [](LintContext& ctx) {
Check(ctx.Tokens().empty(), "tokens: shaders are not lexed as C++");
});
RunLint(cfg, Mode(LintMode::Report));
}
if (Failures > 0) {
std::println(std::cerr, "{} assertions failed", Failures);
return 1;