diff --git a/AGENTS.md b/AGENTS.md index 1c445f4..eab0c32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ Partitions and what they own: - `:Test` holds `AddTest`, `RunTests` and `TestRunner`. - `:Lint` is the lint/format driver plus the libclang token and AST layer. - `:Script` is the driver behind `crafter-build run`: it looks a name up in `cfg.scripts`, calls the callback and turns what it returns into the exit code. -- `:Agents` keeps the dependency notes section of a project's AGENTS.md: it walks the dependencies `cfg.dependencyNotes` selects, reads each one's `AGENTS.consumer.md` and merges them between the section's markers. Which Configurations belong to another project, and where it lives, comes from the loader (`LoadedProjectOf` in `:External`, `PragmaDependencies` in `:Platform`), because only the Configuration `GitProject` and `LocalProject` return has its `path` rebased. +- `:Agents` writes a project's AGENTS.md from its AGENTS.template.md: it walks the dependencies `cfg.dependencyNotes` selects, reads each one's `AGENTS.consumer.md` and merges them into the template's dependency notes section. Which Configurations belong to another project, and where it lives, comes from the loader (`LoadedProjectOf` in `:External`, `PragmaDependencies` in `:Platform`), because only the Configuration `GitProject` and `LocalProject` return has its `path` rebased. - `:Progress` is console output. `:Shader` compiles shaders through glslang. `:Asset` is the hook into Crafter.Asset. ## Dev environment @@ -194,4 +194,4 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary: - The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib//eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`. - `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`. - `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe---*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both. -- The self-host build and `test` keep the dependency notes section of this very file. None of Crafter.Math, Crafter.Asset and Crafter.LintRules ships an `AGENTS.consumer.md` yet. Once one does, a build here can change AGENTS.md whenever that repository changes its notes, since all three are unpinned. The tree is then dirty, so the next build's `git describe --dirty` version differs, which is a new `crafter.build-lib` variant and a full rebuild. Commit the section to settle it. +- This repository has no `AGENTS.template.md`, so the self-host build and `test` leave this file alone, and the `AGENTS.consumer.md` Crafter.Math ships does not reach it. A template would make this file build output: it would go into `.gitignore`, and a fresh clone or worktree would have no AGENTS.md until its first build. diff --git a/implementations/Crafter.Build-Agents.cpp b/implementations/Crafter.Build-Agents.cpp index f26c8c2..bddd41b 100644 --- a/implementations/Crafter.Build-Agents.cpp +++ b/implementations/Crafter.Build-Agents.cpp @@ -22,12 +22,26 @@ namespace { // Right under the begin marker, for whoever opens the file: an edit made // inside the section is gone after the next build. constexpr std::string_view Preamble = ""; + // The first line of every AGENTS.md a build writes. It tells whoever + // opens the file, an agent asked to change it included, where an edit + // has to be made to outlive the next build, and it is how a build tells + // its own AGENTS.md from one someone wrote, which it never replaces. A + // changed word would make every AGENTS.md written before look like + // someone's own. + constexpr std::string_view GeneratedLine = ""; constexpr std::string_view NotesFileName = "AGENTS.consumer.md"; + constexpr std::string_view TemplateFileName = "AGENTS.template.md"; + constexpr std::string_view AgentsFileName = "AGENTS.md"; void Warn(std::string_view message) { Progress::Diagnostic(std::format("warning: {}", message)); } + // The line ending `text` uses, so a CRLF file does not end up mixed. + std::string_view LineEnd(std::string_view text) { + return text.contains("\r\n") ? "\r\n" : "\n"; + } + std::string ErrnoMessage() { return std::error_code(errno, std::generic_category()).message(); } @@ -249,9 +263,9 @@ namespace { } // Why `text` cannot go into the section as it is; empty when it can. A - // marker line of its own would end or restart the section once the file - // is read back, and a code block left open would hide the end marker: - // either way the next build could no longer find what this one wrote. + // marker line of its own would end or restart the section for whoever + // reads it, a template copied from the result included, and a code block + // left open would run on over the notes after it and the end marker. // Lines are numbered from `firstLine`. std::string NoteProblem(std::string_view text, std::size_t firstLine) { const MarkerScan scan = ScanMarkers(text); @@ -397,6 +411,33 @@ namespace { return notes; } + // Whether a build may replace `agentsMd`, the AGENTS.md beside a + // template: one a build wrote, or one with no text at all, which is what + // a write cut short leaves of a new file. Anything else someone wrote, + // and beside a template that is easily the case: an AGENTS.md committed + // before the template was, or one an agent kept its own notes in. + bool Replaceable(std::string_view agentsMd) { + if (agentsMd.find_first_not_of(" \t\r\n") == std::string_view::npos) return true; + std::string_view first = agentsMd.substr(0, agentsMd.find('\n')); + if (first.ends_with('\r')) first.remove_suffix(1); + return first == GeneratedLine; + } + + // An AGENTS.md with no template beside it is someone's own, and no build + // writes it. One that still holds a section from the builds that kept the + // notes in a committed AGENTS.md would go stale without a word, under a + // preamble saying every build rewrites it, so each build says so until + // the file has moved to a template or the section is gone. + void WarnStaleSection(const fs::path& agents) { + std::error_code ec; + if (!fs::is_regular_file(agents, ec)) return; + std::string text; + if (!ReadFile(agents, text).empty()) return; + const MarkerScan scan = ScanMarkers(text); + if (scan.begins.empty() && scan.ends.empty()) return; + Warn(std::format("{} has a dependency notes section, which builds no longer update; rename the file to {} and keep {} out of version control to have every build write it with current notes, or delete the section", agents.string(), TemplateFileName, AgentsFileName)); + } + // Bring `fd`, which holds `current`, to `wanted`: from the first byte the // two differ at, so the text in front of the section is never rewritten, // then cut to length and synced. Empty on success, otherwise the error. @@ -405,10 +446,9 @@ namespace { // rename swaps in another file, which drops whatever belongs to the old // one (a hard link such as a CLAUDE.md beside it, ACLs, ownership, being // read-only) and needs fsyncs to be crash safe at all. The price is that - // a reader can catch the section half written, and that a write failing - // partway, on a full disk say, leaves it so. With the section last, - // where it is appended, only the section can be damaged, and the next - // build reports its markers instead of guessing. + // a reader can catch the file half written, and that a write failing + // partway, on a full disk say, leaves it so until the next build writes + // it again. std::string WriteChanged(std::int32_t fd, std::string_view current, std::string_view wanted) { const auto diverge = std::ranges::mismatch(current, wanted); for (auto done = static_cast(diverge.in2 - wanted.begin()); done < wanted.size();) { @@ -427,24 +467,24 @@ namespace { } } -std::string Crafter::MergeDependencyNotes(std::string_view agentsMd, std::span notes) { +std::string Crafter::MergeDependencyNotes(std::string_view templateMd, std::span notes) { for (const DependencyNote& note : notes) { if (const std::string problem = NoteProblem(note.text, 1); !problem.empty()) throw std::runtime_error(std::format("the notes from {}: {}", note.name, problem)); } - const MarkerScan scan = ScanMarkers(agentsMd); + const MarkerScan scan = ScanMarkers(templateMd); const bool section = scan.begins.size() == 1 && scan.ends.size() == 1 && scan.begins[0] < scan.ends[0]; - // The file's own line ending, so a CRLF file does not end up mixed. - const std::string_view eol = agentsMd.contains("\r\n") ? "\r\n" : "\n"; + const std::string_view eol = LineEnd(templateMd); if (notes.empty()) { - // With nothing to write, leaving the file alone loses nothing, so - // anything short of a well-formed section is no concern of this one. - if (!section) return std::string(agentsMd); - std::string_view before = agentsMd.substr(0, scan.lines[scan.begins[0]].begin); - const std::string_view after = agentsMd.substr(scan.lines[scan.ends[0]].next); - // The blank line a section at the end was appended after goes with - // it. One the user moved keeps the text around it apart. + // With nothing to place, nothing can land in the wrong place, so + // anything short of a well-formed section is left as it was written. + if (!section) return std::string(templateMd); + std::string_view before = templateMd.substr(0, scan.lines[scan.begins[0]].begin); + const std::string_view after = templateMd.substr(scan.lines[scan.ends[0]].next); + // A section that ends the template takes the blank line in front of + // it, which an appended one gets. One placed elsewhere keeps the text + // around it apart. if (after.empty() && before.ends_with(std::format("{0}{0}", eol))) before.remove_suffix(eol.size()); return std::format("{}{}", before, after); } @@ -455,13 +495,12 @@ std::string Crafter::MergeDependencyNotes(std::string_view agentsMd, std::span notes = project.dependencyNotes == DependencyNotes::None ? std::vector() : CollectNotes(project, projectFile, projectDir); + std::string wanted; + try { + wanted = std::format("{}{}{}", GeneratedLine, LineEnd(templateMd), MergeDependencyNotes(templateMd, notes)); + } catch (const std::exception& e) { + Warn(std::format("{}: {}, so {} was not written", templateFile.string(), e.what(), agents.string())); + return false; + } - // Locked from the read through the write, so two builds never interleave - // their writes, and on the file itself, which also covers two projects - // that link one AGENTS.md. A read-only descriptor takes the lock as - // well, so a read-only AGENTS.md is still read without complaint. - const Descriptor reading(::open(agents.c_str(), O_RDONLY | O_CLOEXEC)); + // Created when missing, since a fresh clone has no build output, and only + // now, with the text in hand, so a template the notes cannot be placed + // in leaves no empty AGENTS.md behind. Locked from the read through the + // write, so two builds never interleave their writes, and on the file + // itself, which also covers two projects that link one AGENTS.md. A + // read-only descriptor takes the lock as well, so a read-only AGENTS.md + // is still read without complaint. + const Descriptor reading(::open(agents.c_str(), O_RDONLY | O_CREAT | O_CLOEXEC, 0666)); if (!reading.Valid()) { - Warn(std::format("{} could not be read ({}), so its dependency notes were not updated", agents.string(), ErrnoMessage())); + Warn(std::format("{} could not be opened ({}), so it was not written", agents.string(), ErrnoMessage())); return false; } while (::flock(reading.Get(), LOCK_EX) != 0 && errno == EINTR) {} std::string current; if (const std::string error = ReadAll(reading.Get(), current); !error.empty()) { - Warn(std::format("{} could not be read ({}), so its dependency notes were not updated", agents.string(), error)); + Warn(std::format("{} could not be read ({}), so it was not written", agents.string(), error)); return false; } - - std::string merged; - try { - merged = MergeDependencyNotes(current, notes); - } catch (const std::exception& e) { - Warn(std::format("{}: {}, so its dependency notes were not updated", agents.string(), e.what())); + if (current == wanted) return false; + if (!Replaceable(current)) { + Warn(std::format("{} was not written by a build, so it was not replaced by one written from {}; once the template holds what it says, delete it", agents.string(), TemplateFileName)); return false; } - if (merged == current) return false; // Only now, with something to write: a read-only AGENTS.md is refused // here, and not on every build that leaves it as it is. const Descriptor writing(::open(agents.c_str(), O_WRONLY | O_CLOEXEC)); if (!writing.Valid()) { - Warn(std::format("{} could not be opened for writing ({}), so its dependency notes were not updated", agents.string(), ErrnoMessage())); + Warn(std::format("{} could not be opened for writing ({}), so it was not written", agents.string(), ErrnoMessage())); return false; } - if (const std::string error = WriteChanged(writing.Get(), current, merged); !error.empty()) { + if (const std::string error = WriteChanged(writing.Get(), current, wanted); !error.empty()) { // Put back what was there. A full disk that refused the new bytes // usually still takes the old ones, whose space was just freed. - if (WriteChanged(writing.Get(), merged, current).empty()) { - Warn(std::format("{} could not be written ({}), so its dependency notes were not updated", agents.string(), error)); + if (WriteChanged(writing.Get(), wanted, current).empty()) { + Warn(std::format("{} could not be written ({}), so it was left as it was", agents.string(), error)); } else { - Warn(std::format("{} could not be written ({}) and may now end in part of the section; the next build reports it if so", agents.string(), error)); + Warn(std::format("{} could not be written ({}) and may now be half written; the next build writes it again", agents.string(), error)); } return false; } - // A build that edits a file the user commits says so. + // A file that appears beside project.cpp, or changes there, unasked, + // says where it came from. if (Progress::GetVerbosity() != Progress::Verbosity::Quiet) { Progress::Clear(); if (notes.empty()) { - std::println("Removed the dependency notes from {}", agents.string()); + std::println("Wrote {} from {}", agents.string(), TemplateFileName); } else { - std::println("Updated the dependency notes in {} ({} {})", agents.string(), notes.size(), notes.size() == 1 ? "dependency" : "dependencies"); + std::println("Wrote {} from {} and the notes of {} {}", agents.string(), TemplateFileName, notes.size(), notes.size() == 1 ? "dependency" : "dependencies"); } } return true; diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index e0d8329..c37c362 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -1840,11 +1840,13 @@ Clean (after the `clean` subcommand): and dependency artifact under those trees goes with it. AGENTS.md: - When an AGENTS.md sits beside the project file, a build and a test run keep a - section in it holding the AGENTS.consumer.md each dependency ships beside its - project.cpp, and write the file only when that section changes. An AGENTS.md - is never created. cfg.dependencyNotes picks the dependencies: Transitive (the - default), Direct, or None, which removes the section. + When an AGENTS.template.md sits beside the project file, a build and a test + run write AGENTS.md from it, with a section holding the AGENTS.consumer.md + each dependency ships beside its project.cpp, and only when the text changes. + Commit the template and keep AGENTS.md out of version control. An AGENTS.md + no build wrote is never replaced, and without a template AGENTS.md is left + alone. cfg.dependencyNotes picks the dependencies: Transitive (the default), + Direct, or None. Project args: Any flag not consumed above is forwarded verbatim to CrafterBuildProject as diff --git a/interfaces/Crafter.Build-Agents.cppm b/interfaces/Crafter.Build-Agents.cppm index 77e3c4e..11762fe 100644 --- a/interfaces/Crafter.Build-Agents.cppm +++ b/interfaces/Crafter.Build-Agents.cppm @@ -7,10 +7,17 @@ namespace fs = std::filesystem; // Dependency notes. A crafter-build project ships an AGENTS.consumer.md beside // its project.cpp, written for the agents of the projects that depend on it, -// and crafter-build keeps those files in each consumer's AGENTS.md. Without +// and crafter-build merges those files into each consumer's AGENTS.md. Without // that, an agent working on a consumer knows nothing about the libraries it // builds against unless someone copied the knowledge in by hand, and a copy // goes stale the moment the library changes. +// +// The consumer writes its own instructions in an AGENTS.template.md beside its +// project.cpp, and every build writes AGENTS.md from that, so AGENTS.md is +// build output and stays out of version control. Kept in a committed +// AGENTS.md instead, as they once were, the notes changed that file whenever +// a dependency changed its own, on builds that changed nothing else, and +// every diff and rebase carried changes nobody had made. export namespace Crafter { struct Configuration; @@ -23,7 +30,7 @@ export namespace Crafter { // Only the projects this one's own configurations and // `#pragma crafter dependency` lines name. Direct, - // None. A section an earlier build wrote is removed. + // None: AGENTS.md holds the template's text alone. None, }; @@ -34,38 +41,43 @@ export namespace Crafter { std::string name; // The file's content, without leading blank lines or trailing // whitespace. Lines end in LF; the merge writes the line ending the - // document uses. + // template uses. std::string text; }; - // `agentsMd` with its dependency notes section holding `notes`, in order. - // A document without a section gets one appended after a blank line; one - // with a section has it rewritten in place, wherever it was moved to; and - // empty `notes` remove it. Removing a section that ends the document takes - // the blank line in front of it too, so a document that ended in a newline - // before it gained the section gets its own bytes back. The section uses - // the document's line ending, so a CRLF file stays CRLF. + // `templateMd` with its dependency notes section holding `notes`, in + // order: AGENTS.md as it reads below the line that says a build wrote it. + // A template without a section gets one appended after a blank line; one + // with a section, which is how a template places it, has it rewritten + // where it stands, whatever it held; and empty `notes` remove it. + // Removing a section that ends the template takes the blank line in front + // of it too, so the result ends as the text before it did. The section + // uses the template's line ending, so a CRLF file stays CRLF. // // The two marker lines count only as whole lines outside fenced code // blocks, which is what lets a document quote them. Throws, naming the // line, when there are notes to write and the markers are unbalanced, or - // when the section would have to be appended to a document that ends - // inside a code block: guessing which part of someone's file is generated - // would overwrite their text, and missing the section would add another on - // every build. With no notes it never throws, since leaving the document - // alone loses nothing. Also throws for a note that would break the section - // when it is read back: one with a marker line of its own, or a code block - // it never closes. - std::string MergeDependencyNotes(std::string_view agentsMd, std::span notes); + // when the section would have to be appended to a template that ends + // inside a code block: a lone or doubled marker is a slip in the + // template, and any guess at what it meant would put the notes, or cut + // the text, where its author did not, while notes appended inside a code + // block would read as code. With no notes it never throws, since nothing + // has to be placed. Also throws for a note that would break the section + // for whoever reads it: one with a marker line of its own, or a code + // block it never closes. + std::string MergeDependencyNotes(std::string_view templateMd, std::span notes); - // Refresh the dependency notes section of the AGENTS.md beside - // `projectFile` from the AGENTS.consumer.md of every dependency - // project.dependencyNotes selects. `project` is the Configuration that - // file returned, after LoadProject: loading is what found the - // dependencies. Does nothing when there is no AGENTS.md, which is never - // created. Writes only when the text changes, in place, so hard links, - // ownership and a read-only AGENTS.md are respected; returns whether it - // wrote. A read or write that fails, a section that cannot be placed and + // Write the AGENTS.md beside `projectFile` from the AGENTS.template.md + // beside it, with the AGENTS.consumer.md of every dependency + // project.dependencyNotes selects merged in. `project` is the + // Configuration that file returned, after LoadProject: loading is what + // found the dependencies. Creates AGENTS.md when it is missing, and + // replaces only one a build wrote: an AGENTS.md someone wrote is left + // alone, with a warning. Without a template it writes nothing, and only + // warns about a section an earlier crafter-build left in AGENTS.md. + // Writes only when the text changes, in place, so hard links, ownership + // and a read-only AGENTS.md are respected; returns whether it wrote. A + // read or write that fails, a template the notes cannot be placed in and // a dependency's unusable notes are warnings: the notes never fail a // build, and nothing is reported when nothing would have been written. bool UpdateAgentsMd(const Configuration& project, const fs::path& projectFile); diff --git a/interfaces/Crafter.Build-Clang.cppm b/interfaces/Crafter.Build-Clang.cppm index 0797e88..b4bbc68 100644 --- a/interfaces/Crafter.Build-Clang.cppm +++ b/interfaces/Crafter.Build-Clang.cppm @@ -529,8 +529,9 @@ export namespace Crafter { // Not part of VariantId: a script changes nothing that is compiled. std::vector