diff --git a/.agents/docs/2026-11-12-issue-86-engine-trap-and-start-budget-design.md b/.agents/docs/2026-11-12-issue-86-engine-trap-and-start-budget-design.md new file mode 100644 index 00000000..fb74e3bd --- /dev/null +++ b/.agents/docs/2026-11-12-issue-86-engine-trap-and-start-budget-design.md @@ -0,0 +1,572 @@ +# Issue 86 after 0.22.0: the engine's trap, and a session that cannot run out of engine starts + +Status: proposal for review · not implemented · evidence: the reporter's four diagnostic bundles +(0.0.20, 0.21.0, 0.22.0 twice — comments 6758603816, 6758928222, 6812093165, 6816991554 of #86), +and the temporary harness of draft PR #87 (branch `tmp/issue86-systop-repro`, workflow +`issue86-repro`), which runs the released 0.22.0 payload on the reporter's project +(`yizhinailong/systop` at `b506132`) on hosted Windows runners; its results (runs 43175887538, +43178642084, 43181924438) are the assets of the temporary pre-release `issue86-evidence`. Target: +the next series release (0.23.0) for the server's part; an engine release (mcppls-clangd, +`23.1.0-mcppls.11`) and a platform pin for the other two. + +The goal this plan serves: **a file that makes clangd trap costs the session that file and one +engine start — not the engine, and never the session's ability to start one; and a trap leaves +enough on the machine that the next report names the function it happened in.** + +| # | Item | Decision | Owner | Section | +|---|---|---|---|---| +| E1 | clangd ends by `0xC000001D` while building `src/ui/monitor_view.cppm`: an unchecked `*HeaderID` in its include analysis | check the optional at both sites (engine patch, upstreamable); until the engine ships, the server turns include analysis off where the defect can occur | engine + server | §1.2, §1.3 | +| P1 | On Windows 11 24H2 builds (26100, 26200) the ok image gives an open file another identity than its name | one identity for a file, however it was reached (platform); the engine's CI gets a runner of that build | platform + CI | §1.4 | +| E2 | A trap inside the ok image says nothing: no code, no file, no frame | the runner reports a trap (condition, image offset, frame-pointer chain) on stderr; the server keeps it | platform + server | §2 | +| S1 | Every abnormal exit is recorded as `-1` | keep the condition's number the platform already gives; name it in the log, the issue and the incident | server | §3.1 | +| S2 | The file set aside is guessed from requests and recent edits, and guessed wrong | the file whose AST build was in flight when the engine ended is the suspect; a named suspect needs no second exit | server | §3.2 | +| S3 | A file set aside comes back on every edit, on a rebuilt closure, and in every new session | a crash ledger keyed by file and engine, kept in the workspace cache; one retry per term, whatever the reason | server | §3.3 | +| S4 | Crash restarts spend the sandbox's grant table to its last slot | a start budget the engine knows: automatic restarts stop while a reserve is left; the refused start offers *Restart Server* | server | §3.4 | +| W4 | The grant table never returns a slot (64 a process, ever) | one slot per distinct grant for the life of the process, in the platform; a pin bump | platform | §4 | +| D | What a report carries, and what the logs say | the reproducer command in the bundle; true counts in messages; the release payload not `dirty`; the kit line names the project's target | server | §5 | +| T | Tests and CI | a Windows module-unit fixture, the restart loop past the table's capacity, a trap fixture | server + engine | §6 | + +## 0. What the evidence says + +### 0.1 The reports + +All four bundles are one machine, one project: Windows 10.0.26200, VS Code, `systop` (target +`x86_64-pc-windows-msvc`, clang 23.1.3, MSVC STL 14.44.35207). The last two are the released 0.22.0 +(payload commit `9e31bf6`, clangd `23.1.0-mcppls.10`, sha256 `47a09679…`), so the reporter is not on +a stale build. (The report's "target x86_64-w64-mingw32" is the log's semantic-kit line, not the +project's target — §5.) + +- **The engine exits, on every version.** 38 engine exits across the bundles: 11 on 0.0.20 (clangd + `mcppls.9`), 7 on 0.21.0, 20 on 0.22.0 (`mcppls.10`). One signature: clangd's log ends without an + error, one to three seconds after `Built preamble of size ~257 KB` for a file that imports + modules (the size of a skipped preamble: the module path, UP-23). In 31 of the 38, + `src/ui/monitor_view.cppm` was being built in the last eight seconds; the other seven are + `main.cpp`, `test_monitor_view.cpp` and other units on the same path. The reporter's own + `okrun clangd.okb --check=src/ui/monitor_view.cppm` ends in `Building AST...` with exit + `0xC000001D` (illegal instruction), every time. +- **The exit is silent by construction.** The image is built with libc++'s fast hardening, a global + stack guard and frame pointers (mcppls-clangd `openkal/clangd/mcpp.toml`): a failed hardening + check and a smashed guard both end in a trap instruction, with no message. Nothing in the image + catches it, so clangd's own crash context ("crashed while building X", LLVM's stack dump) is + never printed, and the server's `crashContexts_` stays empty: `crashFile: null`, `stack: []` in + all 38 incidents. +- **0.22.0's part held, as far as it went.** 0.0.20/0.21.0: 13 directories a start, 9 read-only, + 7 starts fit and the 8th is refused. 0.22.0: 8 directories, 4 read-only (Windows Kits, the MSVC + tools, the payload, mcpp's data), 16 starts fit (16 × 4 = 64, the table) and the 17th is refused, + 26 minutes into a session whose 15 exits were all the trap above. The refusal carries 0.22.0's + sentence, with a wrong count: "this session's 33 engine starts" is `generation_`, which advances + at a start and at a stop (2n − 1); the session had 17. +- **So #86 was never the grant table first.** The table is what a crash loop runs into; W1–W3 of + 0.22.0 moved the wall from the 8th start to the 17th, and W4 (recycling) was deferred. The loop + itself — the trap — was not looked at, because no bundle could say where it was. + +Three more things the bundles show, each a defect of the server's own: + +1. **The exit code is thrown away.** `Connection::exit_code()` returns `by_environment ? -1 : code` + (`src/lsp/connection.cpp:144`); mcpplibs.os gives the condition's number in `code`. The + incidents say `exitCode: -1`; the reporter found `0xC000001D` by hand. +2. **The wrong file is set aside.** 01:21:32 on 11-12, 0.22.0: the one file clangd was building + when it ended was `monitor_view.cppm`; the suspect recorded is `main.cpp`, and at 01:21:47 + `main.cpp` is set aside while `monitor_view.cppm` stays. Suspects are the files of unanswered + requests and of edits in the last ten seconds (`clangd.cpp:2944`), which in an editor at rest + is every visible file. +3. **Setting aside does not hold.** In that session `monitor_view.cppm` was set aside five times + and was back with clangd before each next exit: on an edit (`Quarantine::release`, "the file + changed"), on "the modules it imports were built again since clangd exited on it" (06:48:53 — built again *by the restart*, + identical; five seconds later the engine ended on it again), and with each new session. Every + return cost one engine start, four table slots. + +### 0.2 The harness (draft PR #87) + +The harness replays the report on a clean machine: the reporter's mcpp (2026.11.10.2) and LLVM +(23.1.3) build and test `systop`; the released payload's server is started as the extension starts +it (`okrun mcppls.okb serve --payload …`, the reporter's initialization options) and +`drive_lsp.py` keeps up an editor's requests on the files the reporter had open; then +`check_matrix.py` runs `clangd --check` on `src/ui/monitor_view.cppm` as a matrix, any run that +ends by an exception again under `cdb`, and `symbolize.py` names the frames (it finds the image's +base by matching the bytes at the instruction pointer against the image's ELF — the image is a +static-pie ELF inside the okb, not stripped — and reads the ELF's own symbol table). + +**The trap reproduces, on one of the two runners.** + +| | `windows-2025` (Server 2025, build 26100, MSVC 14.51) | `windows-2022` (Server 2022, build 20348, MSVC 14.44) | +|---|---|---| +| `--check` on the project's file, 3 runs | **`0xC000001D` ×3**, in `Building AST...` | exit 3 ×3 (one error, below) | +| one ftxui header, `import std;`, its type used in the purview | **`0xC000001D`** | exit 3 | +| the same, the type not used | exit 3 | exit 3 | +| a one-struct header beside the file (guarded, unguarded, `#pragma once`), used | exit 3 | exit 3 | +| the same header included as `"../ui/"` (another spelling of its path), used | exit 3 | exit 3 | +| the trapping variant with `.clangd`: `UnusedIncludes: None`, `MissingIncludes: None` | **exit 3 — no trap** | exit 3 | +| … the same fragment as clangd's *user* configuration, its directory named by `XDG_CONFIG_HOME` or by `HOME` | **exit 3 — no trap** | exit 3 | +| … named by `LOCALAPPDATA` | `0xC000001D` (not read) | exit 3 | +| … without `--experimental-modules-support` | exit 3 — no trap | exit 3 | +| … `--clang-tidy=0`, `--enable-config=0`, `--pch-storage=memory`/`disk`, in-process module builds, `OKRUN_SNAPSHOT=0`, `OKRUN_PLACE=0`, a fresh module cache, the server's own arguments | `0xC000001D`, each | exit 3 | +| `src/main.cpp`, `tests/test_monitor_view.cpp`, `src/systop.cppm`, `src/core/refresh_clock.cppm` | exit 0 | exit 0 | +| `src/core/sampler.cppm` (Windows SDK headers in its global module fragment) | exit 3 | exit 3 | +| clangd's own line about its module cache | "…the owned cache is not used here: **an open file is not the node its name refers to**" | "owned, 1024 MiB of payloads" | +| the replayed session (7 minutes) | 7 engine exits, each recorded `-1`; 9 files set aside | 0 exits | + +(Exit 3 is `--check`'s "one error": `module_decl_not_at_start`, which `--check` reports for a +module unit with a global module fragment on the skipped-preamble path and the LSP path does not; +it is a defect of `--check` alone, noted for the engine's list, and not this issue.) + +The reporter's own logs carry the same line as the trapping runner — "an open file is not the node +its name refers to", 155 times in the last bundle — and the reporter's Windows is build 26200. The +split is the Windows build, not the MSVC version (the reporter and the runner that does *not* trap +share 14.44.35207), not the processor (AMD EPYC and Intel Xeon here), and not the machine. + +**The trap, named** (windows-2025, the project's own file; `matrix/original.stack.txt`): + +``` +exception c000001d rip image+0x188e505 bytes at rip: 0f 0b (ud2) +TRAP function_ref<…>::callback_fn+0x665 + function_ref<…>::callback_fn + RecursiveASTVisitor::TraverseTypedefTypeLoc + RecursiveASTVisitor::TraverseFunctionProtoTypeLoc + RecursiveASTVisitor::TraverseFunctionHelper + RecursiveASTVisitor::TraverseDecl + clang::include_cleaner::walkUsed + clang::clangd::computeIncludeCleanerFindings + clang::clangd::ParsedAST::build + clang::clangd::check → clangdMain → ok_main +``` + +and the instructions before it, in the image (`llvm-objdump` at `image+0x188df50`): + +``` +callq include_cleaner::Includes::match(Header) const +… +callq clang::clangd::IncludeStructure::getID(const FileEntry*) const +movq %rax, %r14 +shrq $0x20, %rax ; the optional's "has a value" +je 188e505 ; → ud2 +``` + +**The session shows the server's three defects on a clean machine.** In six minutes the server +set aside, among others, `refresh_clock.cppm` ("clangd was idle" — a unit with no `#include` at +all) and `systop.cppm` ("clangd was file is queued" — eight lines of imports), neither of which +can reach the trap, and handed `monitor_view.cppm` back +three times ("the modules it imports were built again since clangd exited on it"), each return an +exit. Every exit is `-1` in the log and the incidents. Windows Error Reporting, asked for a dump of +every `okrun.exe` that ends by an exception, wrote none: nothing on the machine records the trap +unless a debugger is attached. + +**The grant table reproduces without any crash, to the slot.** Three sessions on the two runners +(three read-only grants a start: Windows Kits, the MSVC tools, the runner's temporary directory) +were restarted in a loop with `mcppls.restartEngine`: in each, 21 starts fit and the 22nd was +refused, `granting …/VC/Tools/MSVC/: out of memory … (3 read-only grants each)` — 21 × 3 = +63, the 22nd start's first grant took the 64th slot and its second was refused; the reporter's is +16 × 4 = 64. The wall is at start `floor(64 / r) + 1` on any Windows machine, crash or no crash. +Two details of the refusal: its sentence says "43 engine starts" (22 were made), and the server's +log has no line for it at all — the engine logs `is given 8 directories` and nothing after; the +sentence exists only in the status issue and the bundle. Every later restart the person asks for +logs `is given 8 directories` again and fails as silently. + +**What CI could not have seen.** Every Windows job of `ci.yml` runs on `windows-2022`; the defect +needs a 26100-or-later build. No Windows 11 24H2 user's first module unit with a third-party +header in its global module fragment was ever exercised before a release. + +## 1. E1 — the trap + +### 1.1 The cause + +Three facts, each established by §0.2, in a chain: + +1. **The same file has two identities in one clangd process.** On Windows builds 26100 and 26200 + the ok image's runtime reports another node identity (device and file number — what LLVM's + `UniqueID` is made of) for a file that was *opened* than for the same file asked about *by + name*; on build 20348 they agree. The engine's own probe says so at every start (patch 0039, + UP-03: "an open file is not the node its name refers to"), and for its module cache alone it + already steps aside. Where exactly the two numbers come apart in openkal's Windows + implementation is not established here (§1.4). +2. **clangd's include analysis identifies a header both ways.** `IncludeStructure` gives a header + its ID by the `UniqueID` of the file the preprocessor entered (`getOrCreateID`, `Headers.cpp`). + `computeIncludeCleanerFindings` later takes each include of the main file, gets a `FileEntry` + for its *canonical resolved path* (`convertIncludes`: `FileManager::getFileRef(Inc.Resolved)`), + and asks for that entry's ID (`IncludeStructure::getID`, by `UniqueID`). When the second lookup + yields another identity than the first — which (1) makes possible — `getID` answers + `std::nullopt`. The disassembly of §0.2 is this call and the test of its result. +3. **Upstream dereferences that optional behind an `assert` alone** (`IncludeCleaner.cpp:411–415` + and `:434–438`: `assert(HeaderID.has_value() && "ConvertedIncludes only contains resolved + includes."); Used.insert(*HeaderID);`). A release build has no assert; an unhardened one reads + an indeterminate ID (wrong, silent); the image's hardened libc++ checks `optional::operator*` + and traps. The trap is the hardening doing its work. + +Why these files, as far as the matrix says — and where it stops saying: + +- The callback runs only for a reference in the main file to a symbol that a main-file include + provides: a purview *use* of a type from the header. `using namespace ftxui;` alone, an unused + include, a `std::` type, a type of an imported module do not reach it (the used/not-used rows; + the reporter's reduction says the same). +- The header must have been *entered in this parse*: the skipped-preamble path of a file that + imports modules (UP-23), where the global module fragment's headers are parsed with the main + file. Without `--experimental-modules-support`, or with no import, there is a textual preamble + and no trap; `main.cpp` in `--check` has one too (its includes precede its imports) and ends 0. + In a session `main.cpp` can be on the skipped path as well (the reporter's logs show it with a + 258 KB preamble), which is how exits came to be attributed to it. +- **Which headers get two identities is not established.** On the trapping runner the ftxui header + (in mcpp's store, `D:\a\_temp\…\xpkgs\compat-x-ftxui\…`, found through `-I`) does; a + one-struct header beside the source (`D:\a\\…`) does not, by its own name or through + `../ui/` (so "a path that is not the canonical one" is not the condition — the fourth run was + made to test exactly that, and refuted it); `sampler.cppm`'s Windows SDK headers do not. The + engine's probe, which reports the split, runs in the server's cache (`D:\a\_temp\…`). The + reporter's reduction found a one-struct header that traps, on their paths. What the affected + locations have in common is the platform's question (§1.4); the engine's fix (§1.2) does not + depend on the answer, and the server's stopgap (§1.3) keys on the engine's own probe line, which + both trapping machines print and the non-trapping one does not. + +Who is affected: any Windows 11 24H2-or-later user whose module unit includes a library header in +its global module fragment, imports a module, and uses a type from that header — the ordinary +shape of an mcpp module unit over a C or C++ library. The engine ends at the first build of such a +file, every time. + +### 1.2 The fix, in the engine + +At both sites the optional is checked; an include whose entry has no ID is not marked used by ID: + +```cpp +for (auto *Inc : ConvertedIncludes.match(H)) { + Satisfied = true; + if (auto HeaderID = AST.getIncludeStructure().getID(&Inc->Resolved->getFileEntry())) + Used.insert(*HeaderID); + else + UsedUnidentified = true; // the reference is satisfied; which include it was is unknown +} +``` + +and, so that the unchecked path's *wrong answer* does not replace the trap, an include structure in +which some include could not be identified reports no unused includes for this build +(`getUnused` is skipped when `UsedUnidentified`): a missing "unused include" hint is the honest +result when the analysis cannot tell headers apart; a false "unused" on a header in use would be a +diagnostic that offers to delete a needed line. Missing-include findings are unaffected (they are +computed from providers, not IDs). + +One patch in mcppls-clangd's series (`UP-41`), written to go upstream as it is: the assert states an +invariant a file system can break (two paths, two identities — a network share, a substituted +drive, a case-insensitive volume do it without openkal), and a release build must not have +undefined behavior there. Engine version `23.1.0-mcppls.11`. + +Rejected: resolving the include's entry by the path it was entered by (it repairs this site and +leaves every other by-name lookup of the analysis as it is); turning the image's hardening off (it +found a real defect on its first outing; §2 makes its report readable instead). + +### 1.3 Until the engine ships: the server keeps the analysis off where it traps + +A workaround the matrix proves (the `.clangd` row of §0.2): a `.clangd` beside the project with + +```yaml +Diagnostics: + UnusedIncludes: None + MissingIncludes: None +``` + +ends the trap; the reporter can use it today, and it goes into the issue and +`docs/50-troubleshooting.md` now. For everyone who has not read the issue, the server does the +same thing itself, as a workaround of the engine's version (`WA-CLANGD-015`, in +`src/engine/clangd/workarounds.cppm`; off for an engine whose identity lists the fix's feature): + +- **How.** The image determines no user configuration directory on Windows ("Couldn't determine + user config file, not loading", in every log); it reads one where `XDG_CONFIG_HOME` or `HOME` + points (the user-configuration rows of §0.2 — verified: the trap stops; `LOCALAPPDATA` is not read). The server + starts the engine with `XDG_CONFIG_HOME=/engine-config` and writes + `/engine-config/clangd/config.yaml` with the fragment above. Nothing in the project is + touched. In clangd's documented order a user fragment takes precedence over a project's + `.clangd`, so a project that sets `UnusedIncludes: Strict` does not bring the trap back; the + fixture of §6 asserts that order rather than trusting it. +- **When.** Only where the defect can occur: Windows, an engine without the fix, and the engine's + start-up line saying an open file is not the node its name refers to. The line is known only + after the first start, so the first session on such a machine writes the fragment and restarts + the engine once (one start of the budget, §3.4); the fact is kept in the workspace cache with + the engine's sha256, and later sessions start with the fragment in place. A simpler rule — + always on Windows until `mcppls.11` — costs include hints on machines that are not affected + (Windows 10, Server 2022); the plan takes the probe's rule and keeps the simpler one as the + fallback if the restart proves awkward in review. +- **What the person sees.** One log line and one notice in `mcppls report`: "include hints + (unused/missing) are off on this system: clangd 23.1.0-mcppls.10 ends on them here (#86); they + return with the next engine". No status issue: nothing is broken that the person can act on. +- **What it does not do.** A person whose own `HOME/.config/clangd/config.yaml` exists is not + reading it through the image today either (no `HOME` on Windows), so nothing is taken away; on + Linux and macOS the workaround is never on. + +It costs the unused-include hint on affected machines for one release and is removed with the pin +of `mcppls.11`. This is a server patch release (`server 0.22.1`), independent of everything else +in this plan, and the first thing to ship. + +### 1.4 P1 — the identity itself + +The trap is one consequence of (1); clang uses a file's identity wherever it must know that two +paths are one file — `FileManager`'s table of entries, `#pragma once` and include guards' +"already entered", module input validation, the preamble's stat cache. A header reached by two +spellings is, to all of them, two files on an affected machine. The engine's patch 0039 already +treats the condition as a property of the system to step around; it is a defect of the platform to +fix: + +- **Platform (openkal-windows).** One identity per file, the same from an open handle and from a + name, on every Windows build — in the runtime's `stat`/`fstat` family. The harness gives the + test: the image's probe must print "owned" on a `windows-2025` runner. What differs between the + two builds (a by-name query that exists only on 24H2-era builds is the first place to look) is + for the platform's own analysis; this plan does not guess it into a design. +- **Engine CI.** `mcppls-clangd`'s Windows rows, and this repository's `conformance`, + `vscode-e2e` and `clean-machine` Windows rows, run on `windows-2025` as well as `windows-2022` + until the platform fix is pinned, then on `windows-2025`. The image's probe line is asserted + ("owned") once it is. +- **Server.** `mcppls report` and the bundle say which it is (`engine.fileIdentity: "consistent" | + "split"`, from the engine's start-up line), so a report from an affected machine says so in its + first screen. + +## 2. E2 — a trap that says where it was + +A native clangd that crashes prints its crash context and a stack; the server reads both +(`crash-context` events, K-3) and the incident carries them. The ok image prints nothing, Windows +Error Reporting writes nothing for it (§0.2), and from 2026-11-10 to 2026-11-12 four bundles of 38 +exits could say only *when*. The function was found in one CI run once a debugger was attached and +the frames were named — work the runner can do itself. Three layers, each sufficient +for the next report to be better than this one: + +1. **The runner reports a trap** (openkal-image / okrun, every system). When the image's code ends + by a condition the runner can see — an exception on Windows (a vectored handler, first in line), + a signal on Linux and macOS — it writes one block to stderr before the process ends as it would + have: + + ``` + okrun: trap: illegal instruction (0xC000001D) at image+0x1dc42cf + okrun: build-id ed8cd464126f670098a4b710c8908a5579c0a5ac x86_64 + okrun: frames: image+0x1dc42cf image+0xa93c67 image+0x2b10f4 … (frame-pointer chain, at most 64) + ``` + + The image is built with frame pointers, so the chain is a walk of `rbp`, bounded and + async-signal-safe: no allocation, no symbol lookup, offsets relative to the image's base, which + the runner knows. The harness of PR #87 proves the method from outside (cdb's registers, + `symbolize.py` finds the base from the bytes at the instruction pointer and names frames from + the image's own symbol table); the runner doing it from inside needs no debugger on the + reporter's machine. Rejected: unwinding with `.eh_frame` in the handler (too much code to run in + a process that just trapped), and symbolizing in the runner (the symbol table is 2.8 MB the + runner would have to map; the server or a developer does it offline). +2. **The server keeps it.** The engine's stderr reader recognizes the `okrun: trap:` block as it + recognizes LLVM's stack dump today: `lastExit_.exception` becomes the condition + (`illegal instruction (0xC000001D)`), `lastExit_.stack` the offsets, and the incident and the + bundle carry them with the image's build-id. A developer names the frames with the image of that + build-id (`symbolize.py`, promoted from the harness to `tools/`); the report is actionable + without the reporter's project. +3. **A failed hardening check says which.** With libc++'s hardening in a mode that reports before + it ends the program (an assertion handler that writes `file:line: assertion … failed` and then + traps), the block above is preceded by the check that failed. Its cost is code size in the + image; to be measured on the engine's side before it is taken (the image's cold start is a + budgeted number there). Layer 1 does not wait for it. + +clangd's own notion of "which file" (its `ThreadCrashReporter`) runs in a signal handler the image +does not have; §3.2 gives the server that knowledge from what clangd already logs, so layer 1 does +not need to call back into the image. + +## 3. S — the server + +### 3.1 S1 — the exit keeps its number + +`Connection::exit_code()` returns the platform's `exit_status` whole (`code`, `by_environment`) +instead of folding it to `-1`. From it the engine derives, once, an `ExitCondition { number, kind, +name }`: on Windows an NTSTATUS (`0xC000001D`, "illegal instruction"; `0xC0000005`, "access +violation"; `0xC00000FD`, "stack overflow"; `0xC0000409`, "stack buffer overrun"), elsewhere a +signal (`SIGILL`, `SIGSEGV`, `SIGABRT`, `SIGKILL`). A table of a dozen names in +`src/engine/clangd/`, unit-tested; an unknown number is printed as it is. + +- Log: `clangd ended by an illegal instruction (0xC000001D) while building src/ui/monitor_view.cppm` + — not `clangd exited with -1`. +- Incident and `engine-exit` event: `exitCode` (the number, unsigned on Windows), `exitCondition` + (the name), `endedByEnvironment` (bool). `exitCode: -1` is no longer written. +- Issue text (`engine-crashed`, `engine-crash-loop`): the condition and the file, e.g. "clangd ended + by an illegal instruction while building monitor_view.cppm; that file is answered by mcppls's own + engine meanwhile". +- `SIGKILL` / a console interrupt stay what they are today (W5: a killed clangd keeps the file): + the condition's kind says "ended from outside", and such an exit names no suspect. + +### 3.2 S2 — the suspect is the file being built + +The server already reads clangd's `ASTWorker building file version ` lines (the +`ast-build` events, `clangd.cpp:2035`) and its file-status notifications. From them it keeps, per +file, whether a build is in flight: begun by `ast-build`, ended by the file's next diagnostics or +an idle file status. At an exit: + +- **Named**: clangd printed a crash context (native engines) or the runner's trap block is + attributed (§2) — as today, that file alone. +- **In flight**: exactly the files whose build was in flight. One file: it is set aside at once — + the second-exit strike (W5, #59 U9) exists because a guess deserves a second chance, and this is + not a guess. Several (the engine runs builds in parallel): each gets a strike, and the ledger of + §3.3 separates them by the next exit, since only the guilty one is in flight both times. +- **Neither** (no build in flight, an exit while idle or during indexing): today's rule — files of + unanswered requests and recent edits, a strike each. + +Measured against the bundles: the in-flight rule names `monitor_view.cppm` in the 31 exits where it +was building, and never `main.cpp` alone at 06:29:16. An exit ended from outside (§3.1) names +nobody under any rule. + +Background builds count too: a unit the server opens to index its definitions +(`primeImplementationUnits`) is in flight like any other, and one that ends the engine is kept +from priming by the same ledger (§3.3) instead of being opened again by the next start. + +### 3.3 S3 — set aside means set aside: the crash ledger + +One record per file that ended the engine, in the workspace's cache +(`workspaces//crash-ledger.json`), keyed by the file's path key and holding: + +``` +{ "file": "…/src/ui/monitor_view.cppm", + "engine": "47a09679…", // the image's sha256: another engine starts clean + "condition": "illegal instruction (0xC000001D)", + "exits": 5, "firstAt": "…", "lastAt": "…", + "term": 16, "retryAt": "…", // minutes, doubled each time: 2, 4, 8, 16, then 60, then 24 h + "contentAtLastExit": "", + "closureAtLastExit": "" } +``` + +Rules, replacing the three ways back of §0.1: + +- **One retry per term, whatever the reason.** An edit, a rebuilt closure, a new session, a + changed command: each makes the file *eligible*; it goes back to clangd only when `retryAt` has + passed, and an exit on it doubles the term. A person typing in a file that traps clangd costs + one engine start per term, not one per keystroke burst. Until then the file is answered by + mcppls's own engine, as a file set aside is today. +- **Unchanged means not eligible.** Same content, same closure, same engine: nothing has changed + that could change the outcome, and the term's end alone does not send it back after the second + exit (`exits >= 2`): it waits for a change or for the person. +- **The closure is what it was built from, not how often.** `closure_identity_` counts module + builds (`moduleBuilds_`), so a restart that rebuilds the same BMIs looks like a change. The + ledger's identity is the modules' input identity (the source hash and command the engine already + reports for a rebuild, UP-32/UP-35), which a restart does not move. +- **A new session reads the ledger.** A file with a live record is not handed to clangd at open; + the status says so once ("monitor_view.cppm made clangd end 5 times (illegal instruction); it is + answered by mcppls's engine. Try again"). Today every editor restart starts the loop over — the + third bundle holds six sessions between 01:20 and 01:53 on 11-12, each ended by the same exits. +- **The person decides otherwise in one click.** The quarantine issue's action (`Try again`) clears + the record's term for that file; `mcppls.restartEngine` does not (a restart is not a claim that + the file is fine). A record is dropped when the engine's sha256 changes, when the file is deleted, + or after 30 days. + +The existing `Quarantine` keeps the stall cases (timeouts, a build that will not finish) as they +are; the ledger is the crash case's own state, and `set_aside_` consults it. `crashRetried_` and +`aside.crashClosure` go. + +### 3.4 S4 — the start budget + +The engine knows what a start costs and what is left, on every system whose sandbox counts +(§4 makes the number large; until every pinned platform has it, and for whatever fixed table comes +next, the server does not walk off the edge): + +- `startsMade_` (a counter of its own — the refusal's sentence says "17 engine starts", not + `generation_`) and `slotsSpent_` (read-only grants summed over starts); the capacity is the + platform's (`os::process::grant_capacity()` when the pin has it, else the measured 64). +- **A reserve.** While fewer than `RESERVE = 3` starts' worth of slots is left, *automatic* + restarts — a crash recovery, a crash-loop retry, a plan that names an ungranted directory — are + not made. The engine goes to a state of its own, `paused`: mcppls's engine answers, the status + says "clangd is paused: it ended N times this session and M starts are left before the server + must be restarted", and the reserve belongs to the person (`Restart clangd`). +- **The crash loop ends at its cause.** With §3.2 and §3.3 a file that traps clangd is out after + one or two starts, so a loop of fifteen is not expected again; the budget is the backstop for the + loop nobody attributed (an exit while idle, a trap in the background index). +- **The refused start offers the way out.** `engine-start-failed` from a spent table gets the + action `mcppls.restartServer` (it has `mcppls.showLogs` today), and its sentence says "Restart + Server restores it" — the table is the server process's, and the extension's restart of the + server takes a second, where "restarting the editor" loses the person's windows. No automatic + server restart (0.22.0's §3.3 stands: a server that restarts itself is harder to trust than a + sentence that says what to press). +- A plan change that needs a new grant mid-session (`uncovered`) prefers, when the budget is in + its reserve, to keep the running engine and report the directory as ungranted (the file that + needs it degrades) over spending a start on it. + +## 4. W4 — the table + +0.22.0 left it to the platform; the reproduction on a clean runner (§0.2) says it is not specific +to the reporter's machine, so every Windows user who restarts the engine twenty-odd times in a +session — a toolchain switch, a branch change, a long day — meets it without any crash. Two changes +in openkal's rights table, either sufficient, both wanted: + +1. **One slot per distinct grant.** `attenuate()` looks up a live root with the same directory and + rights before taking a slot, and counts references. The engine's grant set is the same eight + directories at every start; the table then holds four entries for the life of the server, and + a plan change adds the one directory it names. No caller changes. +2. **`forget()` returns the slot** when the last handle closes (0.22.0 §3.4's reading of + `rights.cpp:395`: `names`/`listed`/`base` are released there, `live` is not cleared). + +mcppls takes it as a pin bump (openkal-platform, mcpplibs.os), with the `engine-restarts` fixture +raised past the old capacity as the proof (§6). Until the pin moves, §3.4 holds the line. macOS +gets the same fix and loses its exemption from the fixture. + +## 5. D — what a report carries + +- **The reproducer, in the bundle.** `engine/root-/reproduce.json`: the runner, the image and + the arguments of `clangd --check=` against the bundle's engine + database, as a command line for the reporter's shell — what the reporter of #86 worked out by + hand. `mcppls report --reproduce ` runs it and prints how it ended, with §3.1's name. +- **The exit, named** (§3.1), and **the trap's block** (§2) in `incident.json`. +- **Counts that are counts.** The refusal's "N engine starts" (§3.4). +- **A refused start is in the log.** `clangd could not be started: …` is written as a warning when + it happens, and once more for each later attempt — today it reaches only the status issue and + the bundle, and the log's last word is `is given 8 directories` (§0.2). +- **The release payload is not `dirty`.** Every bundle says `"dirty": true` for the payload's + build, the released ones included; the release job packs from a tree its own steps have written + into. The stamp is taken before those steps (or ignores the files they write), so `dirty` means + what a reader takes it to mean. +- **The kit line names what it is for.** `semantic kit mcppls-kit-openkal-0.24.4 … for + x86_64-w64-mingw32, standing in for mcppls-kit-x86_64-w64-mingw32` is written before the model + is known and names the host's default; the reporter copied it into the report as the project's + target. The line is written when the model's target is known, or says "for this host's default + target, until the project's is known". + +## 6. T — tests and CI + +- **Engine (mcppls-clangd).** A unit test of the include findings over a file system that gives a + header one identity when opened and another by name (the test VFS can; no Windows needed): no + crash, no "unused" finding for the header in use, the missing-include findings unchanged. An e2e + case shaped like `ftxui-used` (a module unit, a header under `-I` in its global module fragment, + an import, a purview use) run with `--check` on `windows-2025`: it ends by no exception. + `--check`'s own `module_decl_not_at_start` on the skipped-preamble path gets its case too. +- **Unit (server).** `ExitCondition` from an `exit_status` (NTSTATUS and signal tables, unknown + numbers); the in-flight set from a sequence of `ast-build` and diagnostics events, and the + suspects each exit yields (one in flight, several, none, ended from outside); the ledger's rules + (one retry per term across an edit, a rebuilt closure and a reload; unchanged is not eligible; a + new engine sha256 clears; `Try again` clears); the start budget (reserve, paused, a person's + restart inside the reserve, the refusal's sentence and action). +- **Conformance.** `engine-trap` (every system): a file whose opening makes the engine end — + the conformance client ends the engine's process while that file's build is in flight (its + `kill-clangd` action, given a condition to end it by and the `ast-build` event to wait for) — asserting: the file is set aside after the + *first* exit, the other files are answered by clangd again, the incident names the condition, + a second session does not hand the file to clangd, and the session's starts stay under three. + `engine-restarts`: raised from 8 restarts to 40 once W4's pin is in (past the old capacity on + every system, macOS included); until then it asserts `paused` with the reserve intact instead of + a refused start. +- **The stopgap (`server 0.22.1`).** On `windows-2025`: the systop-shaped fixture below, opened + through the server with engine `mcppls.10` — the engine does not end, the log carries the + workaround's line, `/engine-config/clangd/config.yaml` exists, and a project `.clangd` + that turns the analysis on does not change the outcome. On `windows-2022` and Linux: the + workaround is off and unused-include hints are still published. +- **Windows runners.** `conformance`, `vscode-e2e`, `clean-machine` and `bench` get a + `windows-2025` row beside `windows-2022` (§1.4): the eleven Windows rows of `ci.yml` are all + build 20348 today. +- **Windows module corpus (CI, win32-x64, both runners).** A fixture shaped like the reporter's unit — a module + interface unit with a global module fragment that includes a third-party header, `import std;` + with the MSVC STL, a purview use of the header's types — under the real MSVC toolchain of the + runner (`mcpp-target-windows-real` already has one). The harness of PR #87 becomes its seed: + `drive_lsp.py`'s session and `check_matrix.py`'s rows are what the fixture asserts. +- **The harness itself** stays out of main: its scripts move to `tools/` only as far as §2 and §5 + use them (`symbolize.py`). + +## 7. Order, and what each step buys + +0. **Today**: the `.clangd` workaround in the issue (§1.3) — the reporter's editor works again. +1. **`server 0.22.1`** — WA-CLANGD-015 (§1.3): every affected Windows 11 machine stops trapping + without anyone reading an issue. Small, self-contained, testable on `windows-2025`. +2. **E1** (engine, `mcppls.11`) — the trap is gone for good and include hints come back. + **S1 + S2** (server, small) ride the same series release — the next crash report names the + condition and the file; one wrongly-set-aside file fewer per exit. +3. **S3 + S4** (server) — any future file that ends the engine costs one or two starts and never + the session. +4. **W4 + P1** (platform pin) — the table stops being a limit anyone meets, and a file is one file. +5. **E2** (platform + server) — the report after next names the function. +6. **D, T** with the steps they belong to — the `windows-2025` rows first, with step 1; the PR #87 + branch and the `issue86-evidence` pre-release are deleted when E1 is fixed. + +## 8. Not in this plan + +- No change to what a start grants (0.22.0's W1/W2 stand). +- No automatic restart of the server. +- No general "crash budget" setting: the ledger's terms and the reserve are constants until a + report says otherwise. diff --git a/.agents/docs/2026-11-12-issue-86-implementation-plan.md b/.agents/docs/2026-11-12-issue-86-implementation-plan.md new file mode 100644 index 00000000..cd13d8e0 --- /dev/null +++ b/.agents/docs/2026-11-12-issue-86-implementation-plan.md @@ -0,0 +1,190 @@ +# Issue 86: implementation plan and task dependencies + +Implements `2026-11-12-issue-86-engine-trap-and-start-budget-design.md` (the design). One pull +request per repository, each by that repository's own process; this file is the split, the order +and what moved against the design while implementing. + +## 1. Repositories and pull requests + +| | Repository | Pull request | Carries | Released as | +|---|---|---|---|---| +| A | `Sunrisepeak/mcppls-clangd` | #22 `mcppls-clangd-0.0.22` (then #23, issue 89) | E1, the engine's part of T | `v0.0.22`, engine `23.1.0-mcppls.11`; C pins `v0.0.23` (`-mcppls.12`), which carries it | +| B | `mcpplibs/openkal-windows` | #47 `fix/one-identity-and-returned-grants` | P1, W4 (the platform's code) | `0.22.0`; reaches mcppls with a later platform pin | +| C | `Sunrisepeak/mcpp-language-server` | `fix/issue-86-engine-exits` | S1–S4, D, the server's part of T, the pin of A | `0.23.0` | + +## 2. What was found while implementing, and what it changes + +1. **P1's cause is established** (the design left it open, §1.4). openkal-windows writes the + volume's serial number into a node's identity from two places: by handle + `FileFsVolumeInformation` (32 bits), by name — where the system has Windows 11 24H2's + `FileStatBasicInformation`, which builds 26100 and later do — that record's + `VolumeSerialNumber` (64 bits). Every file on such a system has two identities. Which headers + trapped (§1.1's open question) follows: those whose resolved path is another *string* than the + path they were entered by (a header under `-I` with mixed separators or `..`), since clang's + `FileManager` answers a path it has seen from its own table and asks the system only for a new + spelling. B corrects it in `info_by_name`. +2. **E1 is fixed more precisely than §1.2 proposed.** An include is marked used by the ID it was + entered with — the `HeaderID` the include itself carries — not by the reverse lookup through its + resolved entry; "report no unused include" remains only for an include with neither. No hint is + lost on an affected machine. +3. **WA-CLANGD-015 (§1.3, the stopgap) is not built.** It exists for the time between a server + release and the engine's; A and C ship together, C pinning A's release, so the stopgap would + never run. The `.clangd` fragment stays documented in `docs/50-troubleshooting.md` for anyone on + 0.22.0 or earlier. +4. **E2 (§2, the runner's trap block) is not in this round.** It is okrun's code (openkal-image) + and a new platform release train; S1 and S2 already give a report the condition's name and the + file. Follow-up, with the platform pin below. +5. **The platform pin does not move in C.** mcppls builds over openkal-platform 0.24.4; B's + correction reaches a payload through openkal-image (okrun), a platform release and the index — + a train that is at 0.27.0 and moving on its own schedule. Until C's pin takes a platform with + openkal-windows ≥ 0.22.0: E1 makes the two identities harmless to the include analysis, and S4 + holds the line on the grant table (§3.4, as the design says). `engine-restarts` therefore + asserts `paused` with the reserve intact, not 40 restarts (§6). + +6. **"Try again" is Restart clangd.** §3.3 gives the quarantine issue an action of its own that + clears a file's term, and says `mcppls.restartEngine` does not. A new command is three editors' + work and a protocol addition; the status issue of a file that ended clangd (`file-ended-engine`) + carries **Restart clangd**, and the person's restart asks the ledger to try every such file + again -- except inside the start reserve, where a file that ends clangd again would spend what + is left: there they stay with mcppls's engine and clangd comes back for everything else. +7. **The runner already reports a fault on Linux and macOS** (found running the trap fixture): + okrun prints `openkal: fault: an instruction the program may not execute (SIGILL) at …` and a + line a frame (`clangd.okb+0x5b3fb2f`) before the process ends. The server now reads that block + into `lastExit.exception` and `lastExit.stack` -- layer 2 of §2, done; layer 1 is missing on + Windows only, which is the system the reports came from. +8. **A server defect the fixture found.** `Workspace::serve_kept` detached the core engine's + request before finishing the job; a clangd between an exit and its restart had not sent the + request yet and answered the detachment at once, as cancelled, which finished the job under the + function: the editor was told `cancelled` instead of the kept answer and the job was written to + after it was erased (SIGSEGV under the debugger, first run). The job is finished first. +9. **`clangd-crash-closure` changes with the rule it tested.** 0.0.17's W5 sent a file back the + moment its modules were rebuilt; §3.3 replaces that ("one retry per term, whatever the reason"). + The fixture now asserts the file is still aside after the rebuild and back when the term ends. +10. **The grant table is every system's.** openkal-linux and openkal-macos carry the same 64-root + table and the same `forget()`; B corrects Windows alone. The budget (S4) therefore counts on + every system with a sandboxed engine. The same correction for the other two is follow-up. +11. **`windows-2025` rows: conformance only.** §6 names four jobs; this round adds the conformance + row (every `on:win32-x64` case on build 26100) and the engine's own gate with its control. The + `vscode-e2e`, `clean-machine` and `bench` rows follow once that row has a history. +12. **D, as built.** The reproducer is `lastExit.reproduce` (report, incident, bundle), not a + separate `reproduce.json`, and there is no `mcppls report --reproduce` yet. The payload's + `dirty` ignores `mcpp.lock` (what the build tool rewrites for its host) and names what does + differ. The kit line says when its target is the host's default. + +13. **The budget counts a workspace's own starts.** The table belongs to the server process, and a + server with several workspace roots has an engine for each; each counts what it started. With + one root -- every report so far -- the count is the table's. Where it is not, a refused start + still says what it is (`engine-start-failed`, with Restart Server). One count for the process + is follow-up, and unnecessary once the platform returns its slots. +14. **`windows-2025` is another toolchain too.** The image has CMake 4.4.5 and MSVC 14.51 where + `windows-2022` has 3.31.6 and Visual Studio 2022's MSVC. `cmake-msvc-std`'s own project is written for 3.31.6's + experimental `import std`: on the new row its `prepare` failed twice over -- the gate is another + UUID, and with the right one CMake 4.4.5 builds the standard module's library and leaves it + off the program's link line (LNK2001). No server runs before that. The row leaves that one + case out by a tag (74 of 75 cases run there); writing the project for both CMakes is + follow-up. + +15. **The candidate was named and not tested.** Since mcppls-clangd 0.0.21 the engine's run has one + artifact; `download-artifact` puts one match straight into its path, the payload job looked + for it a directory deeper, found no part and assembled the payload from the lock's engine + under `--candidate`, saying nothing. Three runs of this pull request were green on + `23.1.0-mcppls.10`. It showed in the status a failing case printed (`"features": 15`). The + job now finds the part wherever it is, prints the engine of every platform, and stops when a + named candidate is not what the payload carries; `mcppls-devtools payload --candidate` refuses + to run without an engine. A7 waits for a run that is green on the candidate in fact. +16. **One case is not steady on the new row.** `mcpp-polling-watch-inputs@polling` failed once in + three runs on `windows-2025` (a deleted external input not seen within its 30 s) and passed + when run again at once. It is this repository's case and polling's timing, not this change; + a failed job is run again (attempt 2 runs the failed cases), as on every row. + +17. **macOS has the fewest starts.** There the engine is given the system's conventional roots + besides the project's own: 14 directories in CI's sessions, of which nine to eleven are + read-only (counted from the grant rules, not read from a session). The table then holds five + to seven starts and the reserve of three leaves two to four made automatically, after which + a restart is withheld (`engine-restart-withheld`) and the person has three. That is §3.4 as + written, and tight; `engine-restarts` does not run on macOS for the same table. Narrowing + macOS's grants as 0.22.0 narrowed Windows' and Linux's, or the platform returning its slots, + is what widens it. +18. **A's comparison run (A6).** Linux ARM64 and Linux x64 pass the warm-prime gate (x64 on its + repetition), Windows records its ratios, and darwin-arm64 is invalid three times running, as + it was for 0.0.21. The engine's policy admits a release then on its comparison with the one + before: the two images alternated in eight pairs on one runner of each system (a temporary + workflow, run 43245970422) -- on macOS 0.0.22's cold readiness and processor time are below + 0.0.21's beside it, on Linux x64 the two are within 2% in every gated quantity. The engine's + changelog states it. +19. **The engine's owned cache, once.** One cold session of the image on darwin-arm64 (1 of 24 in + the comparison run) did not become ready: a module's lock anchor could not be opened while + three files' prerequisites were built at once. The code is 0.0.20's; recorded as known in the + engine's changelog, and follow-up there. + +20. **C pins 0.0.23, not 0.0.22.** Issue 89 (no refactorings in the image) was found while C was open; + its engine PR (#23) was released as `v0.0.23` on top of 0.0.22, and C's candidate followed it. + The lock takes `v0.0.23`'s archive (size and SHA-256 checked against two downloads and the + published `.sha256`), and the candidate file is gone in the same commit. + +## 3. Tasks + +### A — mcppls-clangd (engine) + +| Task | What | Depends on | +|---|---|---| +| A1 | patch 0048 (UP-41): mark by entered ID; `mayConsiderUnused` checks its lookup | — | +| A2 | unit test `IncludeCleaner.HeaderWithAnotherIdentityByName` (a VFS with two identities) | A1 | +| A3 | e2e `gmf_include_analysis.py` in `e2e_suite.py` and `build-test` | A1 | +| A4 | `openkal.yml` job on `windows-2025`: the case as a gate, 0.0.21's image as its control | A3 | +| A5 | ledger row, `PATCHES.md` section, changelog, `RELEASE_VERSION` 0.0.22, `-mcppls.11`, feature `include-ids-by-entry` | A1 | +| A6 | comparison run (`build-test` dispatch, `all`); the row becomes `steady`, the changelog names the run | A1–A5 green | +| A7 | squash-merge, tag `v0.0.22`, release workflow | A6, **C's CI green on A's candidate** | + +### B — openkal-windows (platform) + +| Task | What | Depends on | +|---|---|---| +| B1 | tests `one_identity_for_a_node`, `attenuation_returns_its_place`; a `windows-2025` row | — | +| B2 | negative control: B1's run fails where it should (first test on 26100 only, second everywhere) | B1 | +| B3 | `info_by_name`: the record's volume word translated to the handle's, learned per volume | B2 | +| B4 | `rights::forget`: a root without a record gives its place back | B2 | +| B5 | README, 0.22.0; squash-merge (done: `6a2da47`, release `0.22.0`) | B3, B4 green | + +### C — mcpp-language-server (server) + +| Task | What | Depends on | +|---|---|---| +| C1 | S1: `Connection::exit_status()`; `ExitCondition` (NTSTATUS and signal names); log, incident, event, issue text | — | +| C2 | S2: in-flight AST builds from `ast-build` events; suspects by rule (named / in flight / neither); an exit ended from outside names nobody | C1 | +| C3 | S3: crash ledger in the workspace cache; one retry per term; unchanged is not eligible; a new session reads it; `Try again` clears | C2 | +| C4 | S4: `startsMade_`, slots spent, reserve, `paused`; refused start logged, counted truly, offers `mcppls.restartServer` | C1 | +| C5 | D: `reproduce.json` in the bundle; `engine.fileIdentity`; the kit line; the payload's `dirty` stamp | C1 | +| C6 | T: unit tests of C1–C4 (the pure rules, and the engine with a process that ends by a signal: set aside at the first exit, across an edit and a session; paused at the reserve); conformance `engine-trap`; a `windows-2025` conformance row | C1–C4 | +| C7 | pin A: `packaging/engine-candidate.json` → A's CI artifact; after A7, `payload.lock.json` → the release, candidate file dropped | A's CI artifact; A7 | +| C8 | docs (`50-troubleshooting.md`, spec/schema where an event or issue changes), changelog, version 0.23.0 | C1–C7 | +| C9 | squash-merge; main CI cancelled, release workflow first; main CI re-run after the release; the release verified locally | C8, A7 | + +## 4. Order across repositories + +``` +A1..A5 ──► A's CI (image + acceptance, windows-2025 gate) ──► artifact clangd-openkal + │ │ + └─► A6 comparison run ▼ +B1 ─► B2 ─► B3,B4 ─► B5 (merge) C1..C6 ──► C7 candidate ──► C's CI green + │ + A6 green + C green ──► A7 merge, tag v0.0.22, release + │ + C7 pin of the release ──► C's CI green ──► C9 +``` + +A's image build is the long pole (hours), so A goes first and B and C are written while it runs. +B is independent of A and C. C cannot merge before A is released; A is not merged before C's CI has +run on A's candidate (the integration rule of both repositories). + +## 5. After this round + +- The platform pin (openkal-platform with openkal-windows ≥ 0.22.0) in mcppls and in the engine's + `openkal/clangd/mcpp.toml`: W4 and P1 reach users; `engine-restarts` goes to 40; the + `windows-2025` control in the engine's CI turns to "one identity". +- E2: the runner's trap block, and the server's reading of it. +- The same correction of `forget()` in openkal-linux and openkal-macos; one start count for the + server process (§2.13). +- macOS's grant set (§2.17); the engine's lock anchor at a cold start (§2.19); `cmake-msvc-std` + for CMake 4.4 and `mcpp-polling-watch-inputs@polling` on `windows-2025` (§2.14, §2.16). +- Housekeeping: draft PR #87, branch `tmp/issue86-systop-repro`, pre-release `issue86-evidence`. diff --git a/.github/versions.env b/.github/versions.env index 40684222..8d33eb0f 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -22,8 +22,8 @@ XLINGS_VERSION=v2026.10.26.1 # The mtest that runs the conformance cases (tests/cts): an xlings package (xim:mtest), which the index # verifies; the release bundle's sha256 checks the bundle taken when xlings cannot install it # (.github/actions/mtest). -MTEST_VERSION=0.5.2 -MTEST_SHA256=be443f8c546b3f4628dcd3f1eb6f45b4d698a7f805232f966349faf8250863c6 +MTEST_VERSION=0.6.3 +MTEST_SHA256=12336d7c94d5dc215d28d21469648bc9c21d985da2dd9e7ac014bc9ae7d16d89 # The xmake the xmake conformance fixtures run (installed from xlings; 0.0.8 part 2 X-7). XMAKE_VERSION=3.1.1 VSCODIUM_VERSION=1.135.06055 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e8164bfc..68c2db0e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -791,23 +791,38 @@ jobs: set -euo pipefail version=$(sed -n 's/^version *= *"\(.*\)"$/\1/p' mcpp.toml | head -1) engine=() + images=() if [ -d candidate ]; then - # The ok image's part (clangd-openkal) is its release archive; its engine.json names openkal. - for archive in candidate/clangd-openkal/*.tar.gz; do - [ -f "$archive" ] && tar -xzf "$archive" -C candidate/clangd-openkal - done + # The ok image's part (clangd-openkal) is its release archive; its engine.json names openkal. The one + # artifact the engine's run has since 0.0.21 is put straight into `candidate` (download-artifact gives + # several a directory each, one none): the archive is looked for wherever it is. From 0.0.21 to + # 0.23.0's pull request it was looked for in `candidate/clangd-openkal` alone, was not there, and the + # payload was assembled from the lock's engine without a word -- a candidate named and never tested. + while IFS= read -r archive; do + tar -xzf "$archive" -C "$(dirname "$archive")" + done < <(find candidate -type f -name 'clangd-*.tar.gz') for part in $(find candidate -path '*clangd/engine.json' -exec dirname {} \;); do platform=$(jq -r .platform "$part/engine.json") chmod +x "$part"/bin/* jq '{platform, version: .["engine-version"], fork: .["fork-commit"], sha256}' "$part/engine.json" engine+=(--clangd "$platform=$part") + images+=("$(jq -r .sha256 "$part/engine.json")") done + [ ${#images[@]} -gt 0 ] || { echo "::error::packaging/engine-candidate.json names a run, and its artifacts hold no engine part (clangd/engine.json)"; find candidate -maxdepth 3; exit 1; } engine+=(--candidate) fi stage/mcppls-devtools payload --server-image "server/mcppls-$version.okb" --server-tree "server/mcppls-$version-openkal.zip" \ --kit kit-openkal ${engine[@]+"${engine[@]}"} --out payload --cache "$PAYLOAD_CACHE" stage/mcppls-devtools payload --verify payload --json > "$RUNNER_TEMP/verify.json" cat "$RUNNER_TEMP/verify.json" + # The engine every later job tests: its version and image, as the payload states them. With a candidate + # they are the candidate's, or the run stops here. + jq -c '.platforms | to_entries[] | {platform: .key, engine: .value.engines.clangd.identity["engine-version"], sha256: .value.engines.clangd.identity.sha256}' payload/payload.json + if [ ${#images[@]} -gt 0 ]; then + carried=$(jq -r '[.platforms[].engines.clangd.identity.sha256] | unique | join(" ")' payload/payload.json) + wanted=$(printf '%s\n' "${images[@]}" | sort -u | paste -sd' ') + [ "$carried" = "$wanted" ] || { echo "::error::the payload carries the engine $carried, the candidate is $wanted"; exit 1; } + fi # The payload carries the one okb byte for byte, and an entry for every platform. want=$(sha256sum "server/mcppls-$version.okb" | cut -d' ' -f1) got=$(jq -r '.server.sha256' "$RUNNER_TEMP/verify.json") @@ -895,6 +910,14 @@ jobs: # semantic kit; the SDK-free target fixtures probe distribution clang, installed below. - { platform: linux-arm64, os: ubuntu-24.04-arm, cross-tools: true } - { platform: win32-x64, os: windows-2022 } + # Windows 11 24H2's build (26100). Issue 86: clangd ended on every module unit with a library header there + # and on no earlier build, and every Windows row of this workflow ran on build 20348 (design 2026-11-12, 6). + # Its image is another toolchain too (CMake 4.4.5, MSVC 14.51): cmake-msvc-std's own project, written for + # CMake 3.31.6's experimental `import std`, does not build there -- the gate is another UUID, and with that + # UUID (f35a9ac6-...) CMake 4.4.5 builds `@cmake_cxx_std.lib` and leaves it off app's link line (LNK2001, + # `std::_General_precision_tables_2::_Max_P`). That is the fixture's `prepare`, before any server + # runs; the row leaves the case out by its tag until the project is written for both. + - { platform: win32-x64, os: windows-2025, part: build-26100, exclude-tag: cmake-3.31-import-std } defaults: run: shell: bash @@ -941,6 +964,7 @@ jobs: exe=''; [ "$RUNNER_OS" != Windows ] || exe='.exe' select=(--tag "on:${{ matrix.platform }}") [ -z "${{ matrix.shard }}" ] || select+=(--shard "${{ matrix.shard }}") + [ -z "${{ matrix.exclude-tag }}" ] || select+=(--exclude-tag "${{ matrix.exclude-tag }}") echo "CTS_SELECT=${select[*]}" >> "$GITHUB_ENV" echo "CTS=stage/mcppls-cts$exe" >> "$GITHUB_ENV" "$MTEST" plan --json "${select[@]}" "stage/mcppls-cts$exe" > "$RUNNER_TEMP/cts-plan.json" @@ -1175,7 +1199,7 @@ jobs: - uses: actions/upload-artifact@v7 if: always() && contains(env.CTS_FIXTURES, ' mcpp-target-windows-real ') with: - name: real-mcpp-target-cdb-${{ matrix.platform }} + name: real-mcpp-target-cdb-${{ matrix.platform }}${{ matrix.part && format('-{0}', matrix.part) || '' }} path: | ${{ runner.temp }}/evidence/mcpp-target-windows-real*/workspace/mcpp-target-windows-real/recorded-*.json ${{ runner.temp }}/evidence/mcpp-target-windows-real*/workspace/mcpp-target-windows-real/compile_commands.json @@ -1207,7 +1231,7 @@ jobs: - uses: actions/upload-artifact@v7 if: always() && contains(env.CTS_FIXTURES, ' diagnostic-bundle ') with: - name: diagnostic-bundle-${{ matrix.platform }} + name: diagnostic-bundle-${{ matrix.platform }}${{ matrix.part && format('-{0}', matrix.part) || '' }} path: ${{ runner.temp }}/bundles/*.zip if-no-files-found: ignore diff --git a/CHANGELOG.md b/CHANGELOG.md index bd36db36..0b7f5c15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,107 @@ Versions are three-part semantic versions, `X.Y.Z`: `X.Y` is the series of the s patch is `## [ X.Y.Z]` (`server`, `vscode`, `clion`, `zed`, `claude-code`). +## [0.23.0] — 2026-11-12 + +### Fixed + +- **clangd ended on every module unit that includes a library header, on Windows 11 24H2** + ([#86](https://github.com/Sunrisepeak/mcpp-language-server/issues/86)). The bundled clangd ended by + `0xC000001D` (illegal instruction) on the first build of any module unit that includes a header in + its global module fragment, imports a module and uses a type of the header -- on Windows builds + 26100 and 26200, and on no earlier one. Its include analysis identified a used include by the file + entry of the include's resolved path; under the runner an open file and its name had two + identities on those builds, the lookup came back empty, and the dereference trapped in the image's + hardened standard library, without a word. Since `23.1.0-mcppls.11` (mcppls-clangd 0.0.22) the engine + marks an include by the ID it was entered with; unused-include hints keep working. The two + identities themselves are corrected in openkal-windows 0.22.0 (a 32-bit and a 64-bit volume serial + number), which a later platform pin brings to the runner. For 0.22.0 and earlier, a `.clangd` + with `Diagnostics: { UnusedIncludes: None, MissingIncludes: None }` avoids it + (docs/50-troubleshooting.md). +- **No refactoring was offered: "No code actions available" on a class's name, an expression, `auto`** + ([#89](https://github.com/Sunrisepeak/mcpp-language-server/issues/89)). clangd's refactorings + ("Define constructor", "Declare implicit copy/move members", "Extract to function", "Expand macro" + and the rest) are units that register themselves when clangd starts; the bundled engine's image was + linked from clangd's libraries alone and carried none of them, on every system, since the engine is + an ok image. A request for code actions was answered with the fixes of its diagnostics and + otherwise with an empty list; the server passed on what it was given. The engine is + `23.1.0-mcppls.12` (mcppls-clangd 0.0.23), whose image links them, and which is not packaged unless + it answers such a request. +- **A file that ends clangd costs the session that file, not the engine.** The reports of #86 showed + the server's own part in the loop: every exit was recorded as `-1`; the files set aside were the + ones last asked about (in an editor at rest, every visible file -- `main.cpp`, which could not + reach the fault, while the file that did stayed); and a file set aside went back on an edit, on + "the modules it imports were built again" (built again by the restart), and in every new session, + each return one more engine start. Now: + - the exit keeps the number the system gave it and its name: `clangd ended by an illegal + instruction (0xC000001D) while building src/ui/monitor_view.cppm`, `SIGSEGV (signal 11)` + elsewhere; `lastExit` and the `engine-exit` event carry `exitCode`, `exitCondition`, + `endedByEnvironment`, and where the runner reports a fault (Linux, macOS) its condition and + frames as image offsets; + - the suspect is the file whose build was in flight when clangd ended. One such file is set aside + at that first exit; several, or none, cost each a strike as before; a clangd ended from outside + (killed, terminated) is nobody's; + - a file that ended clangd is kept in a crash ledger in the workspace's cache and goes back once + a term (2, 4, 8, 16 minutes, an hour, a day), whatever happens meanwhile: an edit, a rebuilt + module, a changed command and a new session make it eligible and nothing more. After a second + exit an unchanged file is not tried again until it or its modules' sources change. **Restart + clangd** tries it again at once; a build that does not end clangd clears the record; another + engine starts clean. The status names the file, the count and the condition + (`file-ended-engine`). +- **A session keeps a reserve of engine starts.** The sandbox's table of granted directories (64 a + server process, a slot per read-only grant per start, none returned up to openkal-windows 0.21.0 + and its Linux and macOS counterparts) is counted: while three starts are left the server makes no + restart by itself -- clangd is paused (`engine-paused`), mcppls's own engine answers, and the + three are the person's. The refused start says how many starts were made (it said "33" for 17: a + generation counter), is written to the log each time (it reached only the status and a bundle), + and offers **Restart Server**, which restores the table in a second. openkal-windows 0.22.0 + returns the slots; the count stays as the backstop until the platform pin has it. +- **A kept answer sent while clangd was between an exit and its restart** was answered `cancelled` + to the editor, and the server could write to memory it had freed (found by the new `engine-trap` + fixture): the job is finished before its engine requests are detached. +- A released payload's `build.dirty` was `true` for the lock file the build tool rewrites; it now + means a tracked source differs, and names the files. The semantic-kit log line no longer reads as + the project's target when it is the host's default. +- **`mcppls query symbol` could answer without the definition, and `query refs` with "no symbol + named ...", when its own start was slow** (a cold machine, or a first query beside a session that + had just begun). A query by name waits for clangd's index before it asks; it knew the index's + report by any progress report, and the server's own ("loading", "preparing modules", shown when a + start takes long enough to be worth showing) counted as one. The query then asked a clangd that had + not read its database yet, and answered with what the module index alone knows -- the declaration. + Only an engine's reports count now. CI met it as the first round of `stability` on a fresh runner; + a failed `cli` check of the conformance runner now says what the command printed. + +### Added + +- `engines[].details`: `buildsInFlight`, `crashLedger`, `startBudget` (starts made, slots spent, + starts left, the reserve, `paused`), `fileIdentity` (`consistent` or `split`, the engine's own + word on whether an open file is the node its name refers to), and under `lastExit` `reproduce`: + the `clangd --check=` command that builds the suspect alone as the session does. +- Status issues `file-ended-engine`, `engine-paused`, `engine-restart-withheld` (S3, additive). +- Conformance `code-actions` (every system), as a client that declares code action literals: the + refactorings on a class's name, a selected expression and `auto`, in a source file and a module's + interface unit, each one carried out -- the action's command goes to clangd and the edit clangd asks + the client to apply holds the text. The runner's new check `code-action` does what an editor does + with an action, and the runner answers `workspace/applyEdit`. +- Conformance `engine-trap` (Linux, macOS): a clangd ended by a signal whenever it builds one file + -- the file is set aside at the first exit and named with the condition, the others keep clangd, + an edit does not send it back, a second session does not hand it to clangd, three starts in all. + `clangd-crash-closure` follows the ledger's rule. The conformance cases also run on + `windows-2025` (Windows build 26100): every Windows row ran on build 20348, where #86's fault + cannot occur. One case stays on the old image alone, `cmake-msvc-std`, whose own project does not + build with the new image's CMake 4.4.5. + +### Changed + +- A file set aside after clangd crashed on it no longer goes back the moment the modules it imports + are built again (0.0.17 W5, #59 U7a); that makes it eligible, and it goes back when its term ends. +- The tests are mtest 0.6.3's: the library (`mcpplibs.mtest`, `mcpplibs.mtest.manifest`) and the engine + CI runs the cases with (`.github/versions.env`). +- CI: a pull request that names an engine candidate (`packaging/engine-candidate.json`) is tested + with it, or its run is red. Since the engine's run has one artifact (mcppls-clangd 0.0.21) the + payload job looked for the part where it no longer was and assembled the payload from the lock's + engine without a word; `mcppls-devtools payload --candidate` now refuses to run without an engine. + ## [0.22.0] — 2026-11-11 ### Fixed diff --git a/docs/50-troubleshooting.md b/docs/50-troubleshooting.md index c8cefa48..d4327e92 100644 --- a/docs/50-troubleshooting.md +++ b/docs/50-troubleshooting.md @@ -293,17 +293,65 @@ argument 3: -O0 -> -O2)`), so a burst names its cause. **"clangd crashed while building NormalJsonTranslator.Core.cpp".** clangd names the file it crashed on (its crash context), and that file is set aside: answered by mcppls's own engine while clangd is -restarted without it. It goes back to clangd when it changes, when its term (two minutes, doubling -if it crashes clangd again) is up, or once, as soon as the modules it imports have been built again: -it may have crashed on one of them half rebuilt. A clangd that exits without naming a file (killed, -or out of memory) costs the files it was working on a strike, not their place: only a second such -exit on the same file within ten minutes sets it aside. `engines[].details.lastExit` in the report -has the exit code, the file, what clangd was doing and, on Windows, the exception code. After five exits in five -minutes clangd is started again after 1, 2, 4, then 8 minutes, and the status says so -(`engine-crash-loop`, with **Restart clangd**); it is not given up for the session. The crash and -timeout issues clear by themselves once clangd has stayed up for a minute. The last resort is +restarted without it. A clangd that ends without saying anything -- the bundled clangd traps where a +hardened library check fails, and prints nothing -- is judged by what it was building: the one file +whose build was in flight is set aside at that first exit, and the log and the status name how +clangd ended (`clangd ended by an illegal instruction (0xC000001D) while building +monitor_view.cppm`, `SIGSEGV (signal 11)` elsewhere). Several builds in flight, or none, cost each +file a strike, not its place: only a second such exit on the same file within ten minutes sets it +aside. A clangd ended from outside (killed, terminated, out of memory) is nobody's fault and costs +nothing. + +**"monitor_view.cppm made clangd end 2 times (an illegal instruction (0xC000001D))".** A file that +ended clangd is kept in a ledger in the workspace's cache (`crash-ledger.json`), and stays with +mcppls's own engine until its term is over: two minutes, then 4, 8, 16, an hour, a day, doubling +each time it ends clangd again. An edit, a rebuilt module it imports, a changed command and a new +editor session each make it *eligible*; none sends it back before the term ends, so a file clangd +cannot build costs one engine start a term and not one a burst of typing. After a second exit an +unchanged file is not tried again at all until its text or what its modules were built from changes. +**Restart clangd** is "try again": it gives every such file back at once (unless the session is down +to its reserve of starts, below). A file that then builds is forgotten by the ledger; another engine +(a new mcppls) starts clean, and a record nobody met for thirty days is dropped. +`engines[].details.lastExit` in the report has the number, the condition (`exitCondition`), the +builds in flight, the rule that named the suspects (`suspectRule`) and, under `reproduce`, the +`clangd --check=` command that builds that file alone the way the session does; on Linux and +macOS `exception` and `stack` carry the runner's own report of the fault (`clangd.okb+0x…` offsets, +which name functions with the image of that version). After five exits in five minutes clangd is +started again after 1, 2, 4, then 8 minutes, and the status says so (`engine-crash-loop`, with +**Restart clangd**); it is not given up for the session. The crash and timeout issues clear by +themselves once clangd has stayed up for a minute. The last resort is [below](#when-mcppls-cannot-recover-by-itself). +**"clangd is paused: it was started 13 times this session and 3 starts are left".** The sandbox that +holds clangd to its directories keeps a table of 64 granted directories for the life of the server +process and, up to openkal-windows 0.21.0 (and its Linux and macOS counterparts), returns no slot: +each start spends one for each read-only directory it grants. The server counts them +(`engines[].details.startBudget`), and while only three starts are left it makes no restart by +itself -- not for a crash, not for a plan that names a new directory: mcppls's own engine answers, +and the three are yours (**Restart clangd**). When none is left the start is refused +(`engine-start-failed`: "…this session's 17 engine starts spent it…"), and **Restart Server** brings +the whole table back in a second, without restarting the editor; `mcppls.grants: ["all"]` does +without the sandbox. How many starts the table holds depends on the system: a start is given three +or four read-only directories on Windows and Linux (sixteen to twenty-one starts), and more on +macOS, whose system directories are granted by convention, so the reserve is reached there after +fewer; `startBudget.readOnlyGrantsEach` and `startsLeft` say it for the session. With the crash +ledger above a file that ends clangd costs one or two starts, so this is the backstop for a long +session of many restarts, not something a crash loop reaches. + +**clangd ends on a module unit with a library header (Windows 11 24H2, mcppls 0.22.0 and earlier).** +On Windows builds 26100 and later the bundled clangd through `23.1.0-mcppls.10` ended by +`0xC000001D` on the first build of any module unit that includes a header in its global module +fragment, imports a module and uses a type of the header +([#86](https://github.com/Sunrisepeak/mcpp-language-server/issues/86)). The engine does not +since `23.1.0-mcppls.11`; 0.23.0 carries `23.1.0-mcppls.12`. On an earlier version, a `.clangd` +beside the project turns the analysis that trapped off: + +```yaml +Diagnostics: + UnusedIncludes: None + MissingIncludes: None +``` + **"mcpp could not describe tools/updater/mcpp.toml".** The build tool described the rest of the workspace and said which part it could not (a member whose build program failed, say); that part's files are read with what the rest gives them, and the rest works as usual (`producer-partial`, S2 diff --git a/docs/92-release.md b/docs/92-release.md index 6f1e5e05..6efca92c 100644 --- a/docs/92-release.md +++ b/docs/92-release.md @@ -189,7 +189,9 @@ releases has already passed this repository's CI with it, and the release brings CI's `payload` job then assembles the payload from that run's parts instead of the lock's archives (`payload --clangd = ... --candidate`: a fork version other than the - lock's is taken for the same LLVM base, its identity still checked), and every later job tests it. Fix what it + lock's is taken for the same LLVM base, its identity still checked), and every later job tests it. The job + prints the engine the payload carries and stops when the run's artifacts hold no part, or when the payload's + engine is not the candidate's image: a candidate that is named is tested, or the run is red. Fix what it finds in either PR, and point the file at the new run, until both are green. 3. **Merge and release the engine**, then switch the lock to the released archives (their size and SHA-256, each checked against two downloads and the published `.sha256`) and **delete the diff --git a/docs/specs/CHANGELOG.md b/docs/specs/CHANGELOG.md index 8a43d968..711e2929 100644 --- a/docs/specs/CHANGELOG.md +++ b/docs/specs/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## 2026-11-12 — S3: an engine that ends, and a session's engine starts + +Three issue codes are named. `file-ended-engine`: files that made the engine end (a fault, not an end from outside) are +answered from the module index until their term is over or the person tries them again; the message names them and the +condition, and the command restarts the engine. `engine-paused`: the engine ended and the server does not restart it by +itself, because the environment allows this server process only a few more engine starts and leaves them to the +person; module-level features meanwhile. `engine-restart-withheld`: the same, for a restart the server wanted of an +engine that still runs, which is kept. `engine-start-failed` may carry the command that restarts the server where that +is what restores the environment. All additive: a client that knows none of them shows the message and its command as +it does for any issue, and protocol version stays 1. + +S6: a payload's `build` may name the tracked files that made it `dirty` (`dirtyFiles`). Informative, as `build` is. + ## 2026-11-07 — S6: the payload manifest S6 states `payload.json` of payload versions 4 (mcppls 0.0.18, 0.0.19: one platform, the server's runner under diff --git a/docs/specs/s3-lsp-extensions.md b/docs/specs/s3-lsp-extensions.md index e23c769b..280eb4d7 100644 --- a/docs/specs/s3-lsp-extensions.md +++ b/docs/specs/s3-lsp-extensions.md @@ -157,6 +157,9 @@ interface CxxModulesIssue { | "implementation-unreadable" // an implementation unit does not build, so the definitions in it are not reached | "engine-crash-loop" // the engine keeps exiting; it is started again after a growing delay, module-level features meanwhile | "engine-start-failed" // the engine cannot be started in this environment (a sandbox refused it); module-level features remain + | "engine-paused" // the engine ended and is not restarted by itself: the session's remaining starts are left to the person + | "engine-restart-withheld" // a restart the server wanted was not made, for the same reason; the running engine is kept + | "file-ended-engine" // some files made the engine end; they are answered from the module index until their term is over or the person tries them again | "preparation-stalled" // preparing modules made no progress for a while | "payload-corrupt" // the files the server was installed with do not match their manifest, or carry nothing for this platform; only syntax-level features are trusted | "kit-missing" // the semantic kit of the project's target is not installed; another kit reads the files (S3-4-37) diff --git a/docs/specs/s6-payload.md b/docs/specs/s6-payload.md index 11153423..06fa4434 100644 --- a/docs/specs/s6-payload.md +++ b/docs/specs/s6-payload.md @@ -44,7 +44,7 @@ Every path in `payload.json` **MUST** be a payload path. S6-3.1-2 | | `kit` | object | MUST | `name` (string) and `path` (payload path) of the payload's semantic kit, an S4 kit. S6-3.1-3 | | `files` | object | MUST | Integrity of files every platform uses: payload path to `{ "size": integer, "sha256": string }`. S6-3.1-4 | -| `build` | object | MAY | Provenance: `builtAt`, `commit`, `dirty`, `source`. Not read to start the server. | +| `build` | object | MAY | Provenance: `builtAt`, `commit`, `dirty` (with `dirtyFiles`, the tracked files that differ from the commit, when it is), `source`. Not read to start the server. | ### 3.2 Version 4: one platform diff --git a/docs/zh-CN/50-troubleshooting.md b/docs/zh-CN/50-troubleshooting.md index 4a3b6930..14bf2abe 100644 --- a/docs/zh-CN/50-troubleshooting.md +++ b/docs/zh-CN/50-troubleshooting.md @@ -129,6 +129,18 @@ mcppls report --bundle problem.zip --root path/to/project # 可加 --hide-proj **“clangd crashed while building NormalJsonTranslator.Core.cpp”。** clangd 会说明它在哪个文件上崩溃（崩溃上下文），隔离的就是这个文件：它改由 mcppls 自己的引擎应答，同时重启一个不带它的 clangd。它在以下情况交回 clangd：文件本身有改动；隔离期满（两分钟，再次让 clangd 崩溃则翻倍）；或者它导入的模块重新构建完成（只交回一次，因为它可能正是在某个重建到一半的模块上崩溃的）。clangd 退出时没有指明文件（被杀掉或内存不足），它当时在处理的文件只记一次警告，不会被隔离；同一个文件在十分钟内第二次碰到这种退出，才会被隔离。报告里的 `engines[].details.lastExit` 有退出码、文件、clangd 当时在做什么，Windows 上还有异常码。五分钟内退出五次之后，clangd 会在 1、2、4、8 分钟后重新启动，状态会说明（`engine-crash-loop`，带 **Restart clangd**）；它不会在本次会话里被放弃。clangd 稳定运行满一分钟后，崩溃和超时这两类 issue 会自行清除。最后的办法见[下文](#mcppls-无法自行恢复时)。 +**“monitor_view.cppm made clangd end 2 times (an illegal instruction (0xC000001D))”。** clangd 不留任何输出就结束时（内置 clangd 在加固的库检查失败处直接陷入，什么也不打印），按它当时正在构建的文件来判断：只有一个文件在构建中，就在第一次退出时隔离这个文件，日志和状态会写明 clangd 是怎么结束的（`clangd ended by an illegal instruction (0xC000001D) while building monitor_view.cppm`，其它系统上是 `SIGSEGV (signal 11)` 这样的信号名）。同时有几个文件在构建、或者一个也没有，每个文件只记一次“警告”，十分钟内第二次才隔离。被外部结束的 clangd（被杀、被终止、内存不足）不算任何文件的责任。让 clangd 结束过的文件记在工作区缓存的账本里（`crash-ledger.json`），在期限结束前一直由 mcppls 自己的引擎回答：2 分钟，之后 4、8、16 分钟，1 小时，1 天，每再结束一次翻倍。编辑、它导入的模块重新构建、命令变化、重新打开编辑器，都只是让它“有资格”回去，期限没到不会回去——所以一个 clangd 构建不了的文件每个期限只花一次引擎启动，而不是每敲一阵键盘花一次。第二次退出之后，文件内容和它导入的模块的来源都没变，就不再自动尝试。**Restart clangd** 就是“再试一次”：把这些文件立刻交还（除非本次会话的启动次数只剩保留的那几次，见下一条）。之后能正常构建的文件会从账本里删掉；换了引擎（新版 mcppls）从头开始；三十天没再遇到的记录会被丢弃。报告里 `engines[].details.lastExit` 有退出的数值、原因（`exitCondition`）、当时在构建的文件、嫌疑文件是按哪条规则定的（`suspectRule`），以及 `reproduce` 下面单独构建这个文件的 `clangd --check=` 命令；在 Linux 和 macOS 上，`exception` 和 `stack` 是运行器自己报告的故障（`clangd.okb+0x…` 偏移，配合同一版本的映像可以对出函数名）。 + +**“clangd is paused: it was started 13 times this session and 3 starts are left”。** 把 clangd 限制在它的目录里的沙箱，在服务端进程的生命周期内只有一张 64 项的授权目录表，而且 openkal-windows 0.21.0 及之前（Linux、macOS 的对应实现也一样）从不归还表项：每次启动为每个只读目录花一项。服务端会计数（`engines[].details.startBudget`），只剩三次启动时不再自己重启——崩溃不重启，计划里出现新目录也不重启：由 mcppls 自己的引擎回答，这三次留给你（**Restart clangd**）。一次都不剩时启动会被拒绝（`engine-start-failed`：“…this session's 17 engine starts spent it…”），**Restart Server** 一秒钟就能把整张表拿回来，不用重启编辑器；`mcppls.grants: ["all"]` 则不用沙箱。这张表能容纳多少次启动因系统而异：Windows 和 Linux 上每次启动授予三到四个只读目录（十六到二十一次启动），macOS 按惯例授予的系统目录更多，所以更早用到保留的三次；`startBudget.readOnlyGrantsEach` 和 `startsLeft` 给出当前会话的数字。有了上面的账本，一个让 clangd 结束的文件只花一两次启动，所以这是长会话里多次重启的兜底，而不是崩溃循环会走到的地方。 + +**clangd 在带库头文件的模块单元上结束（Windows 11 24H2，mcppls 0.22.0 及之前）。** 在 Windows 26100 及之后的版本上，`23.1.0-mcppls.10` 及之前的内置 clangd，会在第一次构建这样的模块单元时以 `0xC000001D` 结束：全局模块片段里包含头文件、导入了模块、并且用到了头文件里的类型（[#86](https://github.com/Sunrisepeak/mcpp-language-server/issues/86)）。`23.1.0-mcppls.11` 起的引擎已经修复（0.23.0 带的是 `23.1.0-mcppls.12`）。更早的版本可以在项目旁边放一个 `.clangd`，关掉出问题的分析： + +```yaml +Diagnostics: + UnusedIncludes: None + MissingIncludes: None +``` + **“mcpp could not describe tools/updater/mcpp.toml”。** 构建工具描述了工作区的其余部分，并说明了它没能描述的那一部分（例如某个成员的构建程序失败）；那部分的文件按其余部分提供的信息来读，其余部分照常工作（`producer-partial`，S2 0.3.0）。修好消息里指出的问题，下次重新加载就会一并描述它。 **“clangd rejected the compile command for module scanning”。** clangd 在构建模块之前，会用数据库里每个单元自己的编译命令扫描它的 import；编译器驱动拒绝的命令会让扫描失败，模块也就一个都不会构建——issue #23 的 `LTO requires -fuse-ld=lld` 就是这种情况。状态会用驱动的原话写出第一处拒绝（类别 `environment`），`engines[].details.scanFailures` 记录次数。命令找不到头文件时也这样说明（类别 `project`）。正在输入的文件扫描失败是常态，只计数。 diff --git a/docs/zh-CN/92-release.md b/docs/zh-CN/92-release.md index cf823509..68372ef8 100644 --- a/docs/zh-CN/92-release.md +++ b/docs/zh-CN/92-release.md @@ -134,7 +134,7 @@ nightly 运行还多一个 `termux` job：termux 自己的用户空间和 PRoot 服务器编译可与尚未结束的上游构建并行。`engine candidate parts` 任务仅在两个架构的映像、`clangd-openkal` 部件及其身份检查通过、run/head 元数据匹配且存在不可变构件摘要后允许装配；已知的上游失败或取消会拒绝准入。 - CI 的 `payload` 任务随后用那次运行的部件（`payload --clangd =<部件> ... --candidate`）组装 payload，而不是锁文件里的归档，之后的所有任务都测试它。发现的问题在哪个 PR 就在哪个 PR 修，并把文件指向新的运行，直到两边都通过。 + CI 的 `payload` 任务随后用那次运行的部件（`payload --clangd =<部件> ... --candidate`）组装 payload，而不是锁文件里的归档，之后的所有任务都测试它。该任务会打印 payload 实际携带的引擎；那次运行的产物里找不到部件，或 payload 携带的不是候选映像时，任务直接失败：写了候选就一定测到候选，否则这次运行是红的。发现的问题在哪个 PR 就在哪个 PR 修，并把文件指向新的运行，直到两边都通过。 3. **合入并发布引擎**，再把锁文件切换到发布的归档（大小和 SHA-256，各自用两次下载和发布的 `.sha256` 核对），并在同一个提交里**删除候选文件**。CI 在发布的字节上再跑一次。 4. 合入本 PR 并发布。 diff --git a/editors/claude-code/.claude-plugin/marketplace.json b/editors/claude-code/.claude-plugin/marketplace.json index d8fb1439..61c122f7 100644 --- a/editors/claude-code/.claude-plugin/marketplace.json +++ b/editors/claude-code/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "displayName": "C++ Modules Language Server", "source": "./mcppls-lsp", "description": "Registers mcppls as the language server for C and C++ sources, including C++20/23 named modules.", - "version": "0.22.0", + "version": "0.23.0", "author": { "name": "Sunrisepeak", "url": "https://github.com/Sunrisepeak/mcpp-language-server" diff --git a/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json b/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json index 7929a46d..6b0d2f4b 100644 --- a/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json +++ b/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "mcppls-lsp", "displayName": "C++ Modules Language Server", - "version": "0.22.0", + "version": "0.23.0", "description": "Registers mcppls as the language server for C and C++ sources, including C++20/23 named modules, and its MCP tools (symbols, references, modules, verification, review). Replaces clangd-lsp for a project; do not enable both at once.", "author": { "name": "Sunrisepeak", diff --git a/editors/clion/gradle.properties b/editors/clion/gradle.properties index 05e84df5..d255fed5 100644 --- a/editors/clion/gradle.properties +++ b/editors/clion/gradle.properties @@ -2,7 +2,7 @@ # ones within the same major line; sinceBuild/untilBuild in plugin.xml is what actually gates it. platformType = CL platformVersion = 2026.2.3 -pluginVersion = 0.22.0 +pluginVersion = 0.23.0 org.gradle.jvmargs = -Xmx2g # The IDE ships the Kotlin standard library; bundling a second copy in the plugin is what JetBrains # asks plugins not to do. diff --git a/editors/vscode/package.json b/editors/vscode/package.json index b22dfc5e..7f6309f4 100644 --- a/editors/vscode/package.json +++ b/editors/vscode/package.json @@ -2,7 +2,7 @@ "name": "mcpp-language-server", "displayName": "%displayName%", "description": "%description%", - "version": "0.22.0", + "version": "0.23.0", "publisher": "sunrisepeak", "license": "Apache-2.0", "icon": "icon.png", diff --git a/editors/zed/extension.toml b/editors/zed/extension.toml index 6de7f940..7b19b430 100644 --- a/editors/zed/extension.toml +++ b/editors/zed/extension.toml @@ -1,6 +1,6 @@ id = "mcppls" name = "C++ Modules Language Server" -version = "0.22.0" +version = "0.23.0" schema_version = 1 description = "mcppls - C++20/23 named modules that just work: navigation, completion, hover and diagnostics across modules for any compiler" repository = "https://github.com/Sunrisepeak/mcpp-language-server" diff --git a/editors/zed/src/resolve.rs b/editors/zed/src/resolve.rs index 7c807e38..6ae1a336 100644 --- a/editors/zed/src/resolve.rs +++ b/editors/zed/src/resolve.rs @@ -25,7 +25,7 @@ pub struct Launch { /// This extension's version, as extension.toml states it (`mcppls-devtools version --check` holds the /// two together). The server serves a plugin of its own series only (docs/92-release.md "Versions"). -pub const PLUGIN_VERSION: &str = "0.22.0"; +pub const PLUGIN_VERSION: &str = "0.23.0"; /// The initialization options the server receives: the user's (`lsp.mcppls.initialization_options`), /// with `plugin` naming this extension and its version, so a server of another series refuses with diff --git a/mcpp.lock b/mcpp.lock index 53a7d60e..fd0b962c 100644 --- a/mcpp.lock +++ b/mcpp.lock @@ -17,24 +17,28 @@ namespace = "compat" version = "1.0.8" source = "index+compat@1.0.8" hash = "fnv1a:e0e808c73d8f562d" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.libarchive"] namespace = "compat" version = "3.8.7" source = "index+compat@3.8.7" hash = "fnv1a:c4f73335678a009e" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.lz4"] namespace = "compat" version = "1.10.0" source = "index+compat@1.10.0" hash = "fnv1a:c1f0fef08251ee13" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.mbedtls"] namespace = "compat" version = "3.6.1" source = "index+compat@3.6.1" hash = "fnv1a:bb700e69e973ccd0" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.mimalloc"] namespace = "compat" @@ -42,7 +46,7 @@ version = "3.6.1" source = "index+compat@3.6.1" hash = "fnv1a:44a30e7e4403e477" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."compat.xxhash"] namespace = "compat" @@ -56,18 +60,21 @@ namespace = "compat" version = "5.8.3" source = "index+compat@5.8.3" hash = "fnv1a:8556dee38703a791" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.zlib"] namespace = "compat" version = "1.3.2" source = "index+compat@1.3.2" hash = "fnv1a:9f13dbeaae3ef4c0" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."compat.zstd"] namespace = "compat" version = "1.5.7" source = "index+compat@1.5.7" hash = "fnv1a:cc855914d41f9bfb" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."mcpp.plugins"] namespace = "mcpp" @@ -75,27 +82,27 @@ version = "0.20.0" source = "index+mcpp@0.20.0" hash = "fnv1a:9438fb3cbe77fdc7" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."mcpplibs.cmdline"] namespace = "mcpplibs" version = "0.0.2" source = "index+mcpplibs@0.0.2" hash = "fnv1a:aad61299abc2c4b7" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."mcpplibs.mtest"] namespace = "mcpplibs" -version = "0.5.2" -source = "index+mcpplibs@0.5.2" -hash = "fnv1a:3c8c0386dc9ba899" +version = "0.6.3" +source = "index+mcpplibs@0.6.3" +hash = "fnv1a:23330f86ce6ee24f" [package."openkal"] namespace = "mcpplibs" version = "0.24.0" source = "index+mcpplibs@0.24.0" hash = "fnv1a:7264ed077cd83703" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."openkal-ld"] namespace = "mcpplibs" @@ -110,6 +117,7 @@ namespace = "mcpplibs" version = "0.25.0" source = "index+mcpplibs@0.25.0" hash = "fnv1a:34912a331a4386f5" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."openkal-llvm-runtime"] namespace = "mcpplibs" @@ -117,7 +125,7 @@ version = "0.19.2" source = "index+mcpplibs@0.19.2" hash = "fnv1a:3e60b833793b7bb6" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."openkal-macos"] namespace = "mcpplibs" @@ -132,7 +140,7 @@ version = "0.28.1" source = "index+mcpplibs@0.28.1" hash = "fnv1a:a96807df66880000" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."openkal-platform"] namespace = "mcpplibs" @@ -140,7 +148,7 @@ version = "0.24.4" source = "index+mcpplibs@0.24.4" hash = "fnv1a:8527642a77391fed" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."openkal-tools"] namespace = "mcpplibs" @@ -148,7 +156,7 @@ version = "0.5.3" source = "index+mcpplibs@0.5.3" hash = "fnv1a:17b68873cd803c9b" origin = "platform" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] [package."openkal-windows"] namespace = "mcpplibs" @@ -162,25 +170,27 @@ namespace = "mcpplibs" version = "0.4.1" source = "index+mcpplibs@0.4.1" hash = "fnv1a:c06e49a4f73f1d74" -targets = ["aarch64-macos-musl", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-musl", "x86_64-windows-gnu"] [package."mcpplibs.tinyhttps"] namespace = "mcpplibs" version = "0.3.1" source = "index+mcpplibs@0.3.1" hash = "fnv1a:c9bc1a873a07579e" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."mcpplibs.mtest.manifest"] namespace = "mcpplibs.mtest" -version = "0.5.2" -source = "index+mcpplibs.mtest@0.5.2" -hash = "fnv1a:0caef37dd958cfef" +version = "0.6.3" +source = "index+mcpplibs.mtest@0.6.3" +hash = "fnv1a:153d237dde298771" [package."mcpplibs.mwrap.libarchive"] namespace = "mcpplibs.mwrap" version = "3.8.7" source = "index+mcpplibs.mwrap@3.8.7" hash = "fnv1a:cd2622aff5395ee4" +targets = ["x86_64-linux-gnu", "x86_64-linux-musl"] [package."mcpplibs.mwrap.xxhash"] namespace = "mcpplibs.mwrap" @@ -194,5 +204,5 @@ namespace = "nlohmann" version = "3.12.0" source = "index+nlohmann@3.12.0" hash = "fnv1a:65af05f998b1b9d7" -targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-openkal", "x86_64-windows-gnu"] +targets = ["aarch64-macos-musl", "x86_64-linux-gnu", "x86_64-linux-musl", "x86_64-openkal", "x86_64-windows-gnu"] diff --git a/mcpp.toml b/mcpp.toml index 29839d3e..6afba705 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -59,12 +59,12 @@ mcpplibs.os = "0.4.1" nlohmann.json = "3.12.0" # The test library every package's tests are written with, and the conformance cases (tests/cts): # mcpp-community/mtest, which speaks the mtest protocol to mcpp, mtest and the editors. -mcpplibs.mtest = "0.5.2" -"mcpplibs.mtest.manifest" = "0.5.2" +mcpplibs.mtest = "0.6.3" +"mcpplibs.mtest.manifest" = "0.6.3" [package] name = "mcpp-language-server" -version = "0.22.0" +version = "0.23.0" description = "Compiler-agnostic C++ modules language server" license = "Apache-2.0" authors = ["Sunrisepeak"] diff --git a/modules/base/src/version.cppm b/modules/base/src/version.cppm index 07dcf4bd..c4040c27 100644 --- a/modules/base/src/version.cppm +++ b/modules/base/src/version.cppm @@ -9,9 +9,9 @@ export namespace mcppls::base { // checked against mcpp.toml (the one source) by `mcppls-devtools version --check`, not kept in step by // hand. Three constants that lived here and nothing read were removed rather than left to drift: // the S1 profile version is spec::PROFILE_VERSION, the kit manifest version is spec::KIT_VERSION. -inline constexpr std::string_view VERSION { "0.22.0" }; +inline constexpr std::string_view VERSION { "0.23.0" }; // The clangd the payload ships (mcppls-clangd). Checked against packaging/payload.lock.json by the same command. -inline constexpr std::string_view CLANGD_VERSION { "23.1.0-mcppls.10" }; +inline constexpr std::string_view CLANGD_VERSION { "23.1.0-mcppls.12" }; // Maintained-engine identity stays intact in logs and caches. Its LLVM base // decides libc++ kit compatibility; unknown vendor suffixes remain unchanged. inline std::string_view llvm_base_version(std::string_view engineVersion) { diff --git a/modules/pack/src/payload.cpp b/modules/pack/src/payload.cpp index 367faea4..78e084dc 100644 --- a/modules/pack/src/payload.cpp +++ b/modules/pack/src/payload.cpp @@ -133,7 +133,24 @@ nlohmann::json provenance(const std::string& root) { // R-2 (plan 0.0.9): whether a tracked file differs from the commit. Files the build itself leaves in the checkout // (CI's `cross/` and `host-devtools`) are untracked and say nothing about the source: counting them marked every // released payload dirty (0.0.8's included). - record["dirty"] = !git_output(root, { "status", "--porcelain", "--untracked-files=no" }).empty(); + // Issue 86 (design 2026-11-12, 5): nor does the lock the build tool rewrites for the host it runs on (`mcpp.lock` + // gains the building host's targets at every build): every released payload from 0.0.9 to 0.22.0 said `dirty` + // for it, and a reader took that for a build from changed sources. What does differ is named, so `dirty` can be + // read without the checkout. + const std::string changed { git_output(root, { "status", "--porcelain", "--untracked-files=no", "--", ".", ":(exclude,glob)**/mcpp.lock" }) }; + record["dirty"] = !changed.empty(); + if (!changed.empty()) { + nlohmann::json files = nlohmann::json::array(); + std::size_t begin { 0 }; + while (begin < changed.size() && files.size() < 8) { + const std::size_t end { std::min(changed.find('\n', begin), changed.size()) }; + std::string line { changed.substr(begin, end - begin) }; + while (!line.empty() && line.back() == '\r') line.pop_back(); + if (line.size() > 3) files.push_back(line.substr(3)); // "XY " + begin = end + 1; + } + record["dirtyFiles"] = std::move(files); + } } return record; } diff --git a/packaging/payload.lock.json b/packaging/payload.lock.json index ccdd3515..2fb2af72 100644 --- a/packaging/payload.lock.json +++ b/packaging/payload.lock.json @@ -1,14 +1,14 @@ { "lock-version": 1, "comment": "Every upstream input of the payload and the target kits, and the one table of platforms. Change versions here only; CI fetches, verifies and assembles. Each sha256 was checked against two independent downloads and against the digest the release publishes beside the asset. The server version is read from mcpp.toml. A platform is named - as VS Code names extension targets; each row names the engine its payload entry carries, the target kit built for it (published as mcppls-kit-) and the target its test tools are built for. The VS Code extension, the CLion plugin, the release manifest and CI's matrices list the same platforms, which `mcppls-devtools check platforms` holds them to.", - "clangd-version": "23.1.0-mcppls.10", + "clangd-version": "23.1.0-mcppls.12", "libcxx-version": "23.1.0", "entries": { "mcppls-clangd-openkal": { - "url": "https://github.com/Sunrisepeak/mcppls-clangd/releases/download/v0.0.21/clangd-23.1.0-mcppls.10-openkal.tar.gz", - "file": "clangd-23.1.0-mcppls.10-openkal.tar.gz", - "size": 50382093, - "sha256": "2a9a978ba93fc65bdbd064aae113f7c4d2f86beb070a4ce1a505022a60fc9618", + "url": "https://github.com/Sunrisepeak/mcppls-clangd/releases/download/v0.0.23/clangd-23.1.0-mcppls.12-openkal.tar.gz", + "file": "clangd-23.1.0-mcppls.12-openkal.tar.gz", + "size": 50971795, + "sha256": "9c53256b7c05aeb1cbc59de0fa6d28b4f0c496720c27abf969ed62156a2c92de", "license": "Apache-2.0 WITH LLVM-exception" }, "llvm-project-src": { diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index affc40d4..170cc4db 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -565,6 +565,8 @@ class Client { // Every `workspace/semanticTokens/refresh` the server asked for, when it arrived: an editor asks for the tokens of the // files it shows again on one (the tokens-latency check does). std::vector tokensRefreshes; + // Issue 89: the edits the server asked this client to apply (workspace/applyEdit), each answered as applied. + std::vector appliedEdits; // The server's OS pid where the platform can say; read by the clangd watcher's thread, so atomic. std::atomic serverPid { -1 }; @@ -762,6 +764,9 @@ class Client { } } else if (method == "workspace/semanticTokens/refresh") { tokensRefreshes.push_back(Clock::now()); + } else if (method == "workspace/applyEdit") { + appliedEdits.push_back(message.value("params", Json::object())); + result = Json { { "applied", true } }; } else if (method == "client/unregisterCapability") { // The protocol spells the field "unregisterations". for (const auto& registration : message["params"].value("unregisterations", Json::array())) { @@ -4704,7 +4709,11 @@ class Scenario { const Json value = Json::parse(result->out, nullptr, false); if (value.is_discarded()) return { false, "the output is not JSON: " + clipped(result->out, 200) }; auto [held, detail] = expectations_hold(value, check.value("expect", Json::array())); - return { held, held ? clipped(lsp::dump(value), 160) : detail }; + if (held) return { true, clipped(lsp::dump(value), 160) }; + // What the command said is the evidence of a failed expectation: where its answers came from (provenance), + // the session's state (snapshot), and whatever it warned of. + return { false, std::format("{}\n output: {}{}", detail, clipped(lsp::dump(value), 2000), + result->err.empty() ? std::string {} : "\n stderr: " + clipped(result->err, 1500)) }; } if (kind == "type-text") { // import-hang plan §8: a person typing a line one key at a time. Line `line` of `file` takes each of @@ -5144,6 +5153,84 @@ class Scenario { const bool held { check.value("subset", false) ? std::ranges::includes(expected, lines) : lines == expected }; return { published && held, published ? std::format("{} on lines {}", code.dump(), lines) : std::string { "no diagnostics were published" } }; } + if (kind == "code-action") { + // Issue 89: the actions offered at `at` (through `to`, when a range is selected) hold one titled `title`; with + // `edit-contains`, that action carried out as an editor carries it out -- its own edit, or its command and the + // workspace/applyEdit the server then asks for -- puts that text into this file. `absent` names titles that are + // not offered. + open(file); + const std::string documentUri { uri(file) }; + const std::string title { check.value("title", std::string {}) }; + const Json at = position(check.at("at")); + const Json to = check.contains("to") ? position(check.at("to")) : at; + Json chosen = nullptr; + Json actions = nullptr; + // Asked again while the file is being built, for a minute at most: an engine that has no such action answers + // at once, each time, and the check's whole time would be spent learning nothing more. + const auto deadline = Clock::now() + std::min(timeout_, std::chrono::seconds { 60 }); + bool offered { false }; + while (!offered) { + const auto outcome = client_.request_full("textDocument/codeAction", + Json { { "textDocument", Json { { "uri", documentUri } } }, { "range", Json { { "start", at }, { "end", to } } }, + { "context", Json { { "diagnostics", Json::array() } } } }, timeout_); + if (outcome.timedOut) return { false, "no answer" }; + if (outcome.isError) return { false, lsp::dump(outcome.error) }; + actions = outcome.result; + if (actions.is_array()) { + for (const auto& action : actions) { + if (action.is_object() && action.value("title", std::string {}) == title) { + chosen = action; + offered = true; + break; + } + } + offered = offered || title.empty(); + } + if (offered || Clock::now() >= deadline) break; + client_.drain(std::chrono::milliseconds { 500 }); + } + if (!offered) return { false, lsp::dump(actions) }; + for (const auto& unwanted : check.value("absent", Json::array())) { + for (const auto& action : actions) { + if (action.is_object() && Json(action.value("title", std::string {})) == unwanted) return { false, lsp::dump(actions) }; + } + } + const std::string wanted { check.value("edit-contains", std::string {}) }; + if (wanted.empty()) return { true, lsp::dump(actions) }; + Json edit = nullptr; + if (const auto own = chosen.find("edit"); own != chosen.end() && own->is_object()) { + edit = *own; + } else { + // A Command, or a CodeAction that holds one. + const auto held = chosen.find("command"); + const Json& command { held != chosen.end() && held->is_object() ? *held : chosen }; + const std::size_t before { client_.appliedEdits.size() }; + const auto outcome = client_.request_full("workspace/executeCommand", + Json { { "command", command.value("command", std::string {}) }, { "arguments", command.value("arguments", Json::array()) } }, timeout_); + if (outcome.timedOut) return { false, "the action's command was not answered" }; + if (outcome.isError) return { false, lsp::dump(outcome.error) }; + if (client_.appliedEdits.size() == before) return { false, std::format("the action's command asked for no edit: {}", lsp::dump(outcome.result)) }; + edit = client_.appliedEdits.back().value("edit", Json::object()); + } + // The text the edit gives this file, whichever of the protocol's two shapes it has. + std::string inserted; + const auto take = [&](const Json& edits) { + if (!edits.is_array()) return; + for (const auto& item : edits) { + if (item.is_object()) inserted += item.value("newText", std::string {}) + "\n"; + } + }; + if (const Json* changes { lsp::find(edit, "changes") }; changes != nullptr && changes->is_object()) { + if (const auto own = changes->find(documentUri); own != changes->end()) take(*own); + } + if (const Json* changes { lsp::find(edit, "documentChanges") }; changes != nullptr && changes->is_array()) { + for (const auto& change : *changes) { + const Json* changed { lsp::find_path(change, { "textDocument", "uri" }) }; + if (changed != nullptr && *changed == Json(documentUri)) take(change.value("edits", Json::array())); + } + } + return { inserted.find(wanted) != std::string::npos, lsp::dump(edit) }; + } if (kind == "request") { // mcpp-community/mcpp#797: one request about `file` -- `method`, at `at` when given, with `params` added -- and // its answer: `expect`, the result exactly, or `expect-error`, the code of the error it is refused with. @@ -6586,6 +6673,62 @@ int prepare_crashing_clangd(std::string_view kind, const std::string& payload, s return 0; } +// Issue 86 (design 2026-11-12): a clangd that a fault of its own ends whenever it builds one file -- every start, the +// way an ok image trapped in a hardened library check on the reporter's module unit and printed nothing. The stand-in +// IS the payload's clangd (the script replaces itself with it), with its standard error read by a watch that forwards +// every line and, at "ASTWorker building file <...trapFile>", ends clangd with SIGTRAP: the system reports the signal, +// and nothing names the file but what clangd was building. POSIX only. +int prepare_trapping_clangd(std::string_view kind, const std::string& payload, std::string_view trapFile) { + std::error_code ignored; + if (os::host() == os::platform::windows) { + say("{}: POSIX only", kind); + return 2; + } + const std::string workspace { base::current_directory() }; + const std::string real { payload_clangd(payload) }; + const ClangdCommand realCommand { payload_clangd_command(payload) }; + if (!std::filesystem::exists(real, ignored)) { + say("{}: no clangd at {}", kind, real); + return 1; + } + const std::string directory { base::join_path(workspace, "stand-in") }; + (void)std::filesystem::create_directories(directory, ignored); + const std::string script { std::format( + "#!/bin/sh\n" + "real='{}'\n" + // The ok image's runner takes the image first (S4-4-12); a program engine has none. + "image='{}'\n" + "case \"$1\" in --version|--help) exec \"$real\" ${{image:+\"$image\"}} \"$@\" ;; esac\n" + "echo start >> '{}/starts'\n" + "fifo='{}/log.'$$\n" + "rm -f \"$fifo\"\n" + "mkfifo \"$fifo\" || exit 70\n" + "pid=$$\n" + // The watch: clangd's log, line by line, to the server as it was; the line that begins the file's build ends clangd. + "(\n" + " while IFS= read -r line; do\n" + " printf '%s\\n' \"$line\" >&2\n" + // SIGTRAP: a fault of the program's own that no runner takes for itself. okrun reports SIGILL, SIGSEGV, SIGBUS + // and SIGFPE (openkal's fault report) and returns to the faulting instruction, which a signal sent from outside + // has not got: the first one would be reported and survived, and clangd would finish the build meanwhile. + " case \"$line\" in *'ASTWorker building file '*'{}'*) kill -TRAP \"$pid\" ;; esac\n" + " done < \"$fifo\"\n" + " rm -f \"$fifo\"\n" + ") < /dev/null > /dev/null &\n" + "exec \"$real\" ${{image:+\"$image\"}} \"$@\" 2> \"$fifo\"\n", + realCommand.program, realCommand.prefix.empty() ? std::string {} : realCommand.prefix.front(), directory, directory, base::file_name(trapFile)) }; + const std::string clangd { base::join_path(directory, "clangd") }; + if (auto written = os::fs::write(clangd, script); !written) { + say("{}: {}", kind, written.error().message()); + return 1; + } + if (auto marked = base::make_executable(std::vector { clangd }); !marked) { + say("{}: {}", kind, marked.error().message); + return 1; + } + return 0; +} + // 0.0.8 plan R-4: a producer that answers late. A shell script around the mock mcpp that waits `seconds` before // `emit build-database` and lets every other command through at once, so the server plans with what it scanned itself // meanwhile and then receives the build tool's model. POSIX only. @@ -6736,12 +6879,13 @@ int prepare(const std::string& kind, const std::string& argument) { if (kind == "clangd-crash-context") return prepare_crashing_clangd(kind, argument, "src/crasher.cpp"); if (kind == "clangd-crash-closure") return prepare_crashing_clangd(kind, argument, "src/main.cpp"); if (kind == "clangd-exit-strike") return prepare_crashing_clangd(kind, argument, ""); + if (kind == "clangd-trap") return prepare_trapping_clangd(kind, argument, "src/trap.cppm"); if (kind == "compdb-rejected-command") return prepare_compdb_rejected_command(argument); if (kind == "compdb-mixed-standards") return prepare_compdb_mixed_standards(argument); if (kind == "xmake-stale-compdb") return prepare_marked_compdb(kind, argument.empty() ? std::string { "g++" } : argument, "-DSTALE_COMPDB", "-std=c++17"); if (kind == "compdb-midwrite") return prepare_marked_compdb(kind, argument.empty() ? std::string { "clang++" } : argument, "-DVERSION_ONE", "-std=c++23"); if (kind == "compdb-lto-msvc") return prepare_compdb_lto_msvc(argument); - say("prepare: unknown fixture kind {} (heavy-headers, large-module-file, s1-two-sets, payload-corrupt, producer-candidate, delayed-producer, failure-at-base, compdb-clang-cl-std, compdb-clangxx-msvc-std, generated-module-old-mcpp, clangd-cannot-load, compdb-lto-msvc, clangd-crash-context, clangd-crash-closure, clangd-exit-strike, compdb-rejected-command, compdb-mixed-standards, xmake-stale-compdb, compdb-midwrite)", kind); + say("prepare: unknown fixture kind {} (heavy-headers, large-module-file, s1-two-sets, payload-corrupt, producer-candidate, delayed-producer, failure-at-base, compdb-clang-cl-std, compdb-clangxx-msvc-std, generated-module-old-mcpp, clangd-cannot-load, compdb-lto-msvc, clangd-crash-context, clangd-crash-closure, clangd-exit-strike, clangd-trap, compdb-rejected-command, compdb-mixed-standards, xmake-stale-compdb, compdb-midwrite)", kind); return 2; } diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index 0a4ad6be..74b76742 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -15,6 +15,7 @@ import mcppls.project.scan; import mcppls.normalize.plan; import mcppls.engine; import mcppls.engine.clangd.definition; +import mcppls.engine.clangd.exits; import mcppls.engine.clangd.guard; import mcppls.engine.clangd.primer; import mcppls.engine.clangd.process; @@ -341,6 +342,7 @@ class ClangdEngine final : public Engine { Clock::time_point at; std::set suspects; // what was asked about or changed just before: used when clangd did not say std::vector unanswered; + std::vector inFlight; // issue 86 S2: the files whose builds were in flight }; std::optional pendingExit_; std::map crashContexts_; // by generation, the latest few @@ -497,20 +499,26 @@ class ClangdEngine final : public Engine { // Fix plan F16: set aside for what is on disk, not in the editor: it goes back when the disk is fixed. bool onDisk { false }; std::string modelOrigin; // the model's source then (fix plan F4: a provisional model's verdicts do not outlive it) - // 0.0.17 W5 (#59 U7a): set aside for an exit of clangd, the modules it imports as they were built then - // (closure_identity_). Once they are built again, and differently, it goes back to clangd once. - std::optional crashClosure; }; std::map> aside_; // path key // W5: an exit of clangd that names no file (a kill, the OOM killer) is not yet a file's fault: the files it was busy // with get a strike, and a file that gets a second within EXIT_STRIKE_WINDOW is set aside, as a crash naming it is. static constexpr std::chrono::minutes EXIT_STRIKE_WINDOW { 10 }; std::map> exitStrikes_; // path key -> its strike - // W5: how many times each module has been built (clangd's "Built module"), for the closure identity of a file set - // aside after an exit; and the identities each such file already went back to clangd with. + // How many times each module has been built (clangd's "Built module"): a report's count, and no longer anything's + // identity -- a restart builds the same modules again (issue 86). std::map> moduleBuilds_; - std::map, std::less<>> crashRetried_; // path key -> closure identities tried bool closureRetryCheck_ { false }; + // Issue 86 (design 2026-11-12). S2: the files clangd is building now. S3: the files that ended clangd, kept in the + // workspace's cache across edits, restarts and sessions. S4: what this session's starts spent of the sandbox's + // table of granted directories, and whether automatic restarts have stopped to leave the rest to the person. + BuildsInFlight building_; + CrashLedger ledger_; + bool ledgerRead_ { false }; + std::optional ledgerCheckAt_; + StartBudget budget_; + bool paused_ { false }; + std::string fileIdentity_; // "consistent" | "split": whether an open file is the node its name refers to, in the engine's own words std::map> fileStatus_; // client URI -> clangd's last textDocument/clangd.fileStatus state // robustness design C10: a definition that clangd finds only as a declaration in a module's interface is looked for in that // module's other units, which are opened in clangd without the editor (background units) and closed again when unused. @@ -709,6 +717,12 @@ class ClangdEngine final : public Engine { { "workarounds", workarounds_json_() }, // Fix plan F3, F6, F14, F16, F17. { "lastExit", lastExit_ }, + // Issue 86: the builds in flight, the files that ended clangd, the session's starts, and the engine's word + // on whether a file has one identity here. + { "buildsInFlight", building_.files() }, + { "crashLedger", ledger_.to_json()["files"] }, + { "startBudget", start_budget_json_() }, + { "fileIdentity", fileIdentity_.empty() ? Json(nullptr) : Json(fileIdentity_) }, { "scanFailures", Json { { "count", scanFailures_.count }, { "files", scanFailures_.files }, { "firstReason", scanFailures_.firstReason } } }, { "restartBudget", restart_budget_json_() }, { "filesUnsafeOnDisk", disk_hazards_json_() }, @@ -967,6 +981,9 @@ class ClangdEngine final : public Engine { excluded_ = std::move(newExcluded); // A file set aside that the database now gives another command goes back to clangd with it. for (const auto& file : commandChanged) { + // Issue 86 S3: a file that ended clangd is made eligible by another command, as by any change, and goes back + // when its term is over. In a new session every command is another one: the first plan gave all of them back. + if (quarantine_.contains(file) && ledger_verdict_(file) == CrashLedger::Verdict::held) continue; if (!quarantine_.release(file)) continue; aside_.erase(file); log::info("handing {} back to clangd ({}): the engine database has another command for it", file, host_->root_directory()); @@ -1100,7 +1117,7 @@ class ClangdEngine final : public Engine { break; } } - if (excluded_path_(document.path) || quarantined_(document.path)) { + if (excluded_path_(document.path) || quarantined_(document.path) || kept_by_ledger_(document)) { closeWarm(); break; } @@ -1134,7 +1151,11 @@ class ClangdEngine final : public Engine { // What clangd stopped on is still there: the file stays aside until its time is up or what it imports changes. // A file clangd spun on goes back as soon as its text is any other than the one it spun on; one set aside // for what is on disk goes back only once the disk is fixed (fix plan F16), whatever the editor has. - if (const auto aside = aside_.find(key); aside != aside_.end()) { + // Issue 86 S3: a file that ended clangd is the ledger's to give back. An edit makes it eligible, and it + // goes back when its term is over -- not at every burst of typing, each one an engine start. + const auto held = ledger_verdict_(key, document.text); + if (held == CrashLedger::Verdict::held) break; + if (const auto aside = aside_.find(key); aside != aside_.end() && held == CrashLedger::Verdict::clean) { if (aside->second.onDisk) break; if (aside->second.spunOn ? *aside->second.spunOn == text_hash_(document.text) : aside->second.structure == structure_of_text_(document.text)) break; @@ -1436,6 +1457,13 @@ class ClangdEngine final : public Engine { crashContexts_[generation_] = CrashContext { event.value("action", std::string {}), event.value("file", std::string {}), event.value("exception", std::string {}), event.value("stack", std::vector {}) }; while (crashContexts_.size() > 4) crashContexts_.erase(crashContexts_.begin()); + } else if (kind == "runner-fault") { + // A crash context of clangd's that names its file stays; otherwise the runner's report is the better word on + // what happened and where, and stands in for it. + CrashContext& context { crashContexts_[generation_] }; + if (context.exception.empty() || context.file.empty()) context.exception = event.value("exception", std::string {}); + if (context.file.empty()) context.stack = event.value("stack", std::vector {}); + while (crashContexts_.size() > 4) crashContexts_.erase(crashContexts_.begin()); } else if (kind == "scan-failed") { note_scan_failure_(event.value("file", std::string {}), event.value("reason", std::string {}), event.value("driver", false)); } else if (kind == "module-lock-wait") { @@ -1467,8 +1495,14 @@ class ClangdEngine final : public Engine { for (const auto& document : host_->documents()) { if (!document.path.empty() && base::path_key(document.path) == key) { spin_.began(document.uri, event.value("version", std::int64_t { 0 }), Clock::now()); + file = document.path; } } + // Issue 86 S2: every file's, a unit opened for the index too -- whatever clangd is building when it ends. + if (!key.empty()) building_.began(key, file); + } else if (kind == "file-identity") { + // Split is the finding; a later line about another directory does not take it back. + if (fileIdentity_ != "split") fileIdentity_ = event.value("identity", std::string {}); } } @@ -1561,6 +1595,7 @@ class ClangdEngine final : public Engine { } if (accepting_) consider(spinSampleAt_); if (pendingExit_) consider(pendingExit_->at + EXIT_CONTEXT_WAIT); + consider(ledgerCheckAt_); if (upSince_) consider(*upSince_ + RECOVERED_AFTER); for (const auto& [uri, closing] : closingAfterBuild_) consider(closing.second); consider(diskRecheckAt_); @@ -1704,7 +1739,16 @@ class ClangdEngine final : public Engine { request_restart_("clangd did not answer initialize"); } if (closureRetryCheck_ && !primer_.busy()) retry_after_closure_rebuilt_(); + if (ledgerCheckAt_ && *ledgerCheckAt_ <= now) { + ledgerCheckAt_.reset(); // asked now; hand_back_ledger_files_ sets the next one, if a term is still running + hand_back_ledger_files_(); + } for (const auto& key : quarantine_.due(now)) { + // A file that ended clangd waits for the ledger's term, not this one (issue 86 S3). + if (ledger_verdict_(key) == CrashLedger::Verdict::held) { + quarantine_.put(key, now); + continue; + } // Nor is a file whose text on disk would stall clangd (fix plan F16): it waits for the disk, or the database. if (const auto aside = aside_.find(key); aside != aside_.end() && aside->second.onDisk) { if (const auto hazard = diskHazards_.find(key); hazard != diskHazards_.end()) { @@ -1991,6 +2035,12 @@ class ClangdEngine final : public Engine { } if (!process_) process_ = make_process_(); lastConfig_ = config; + building_.clear(); + paused_ = false; + // Issue 86 S4: what this start spends of the sandbox's table -- one slot for each read-only grant, for the life + // of the server process (openkal through 0.25; a platform that returns them makes the count harmless). + budget_.set_capacity(granted_.empty() ? 0 : StartBudget::CAPACITY); + budget_.started(read_only_grants_()); Json grantsJson = Json::array(); for (const auto& grant : granted_) grantsJson.push_back(Json { { "path", grant.path }, { "writable", grant.writable }, { "unchanging", grant.unchanging } }); @@ -2013,6 +2063,14 @@ class ClangdEngine final : public Engine { auto limited = options_.verboseLog ? std::make_shared(std::numeric_limits::max(), std::numeric_limits::max()) : std::make_shared(40, 200); auto reader = std::make_shared(); // only ever used on the thread reading clangd's standard error + // Design 2 (E2): the runner's own report of a fault in the image, where it makes one (okrun on Linux and macOS: + // "openkal: fault: () at
", then a line for each frame, "openkal: #n
+ // +"). The image itself prints nothing when a hardened check traps; this is all there is. + struct RunnerFault { + std::string what; + std::vector frames; + }; + auto fault = std::make_shared(); // read on that thread alone too auto ring = logRing_; auto buildTimes = buildTimes_; ring->add(std::format("--- clangd {} started (generation {}) ---", options_.version, generation)); @@ -2020,10 +2078,15 @@ class ClangdEngine final : public Engine { config, [sink, generation](Json message) { sink(Json { { "kind", "message" }, { "generation", generation }, { "message", std::move(message) } }); }, [sink, generation] { sink(Json { { "kind", "closed" }, { "generation", generation } }); }, - [sink, generation, root, limited, reader, ring, buildTimes](std::string_view line) { + [sink, generation, root, limited, reader, ring, buildTimes, fault](std::string_view line) { // Fix plan F17.1: every line is kept in memory for an incident, whatever reaches the log. ring->add(line); buildTimes->add(line); + if (const auto told = runner_fault_line(line)) { + if (!told->what.empty()) *fault = RunnerFault { told->what, {} }; + else if (!fault->what.empty() && fault->frames.size() < 64) fault->frames.push_back(told->frame); + sink(Json { { "kind", "runner-fault" }, { "generation", generation }, { "exception", fault->what }, { "stack", fault->frames } }); + } const auto read = reader->read(line); const log::Level level { clangd_log_level(line) }; // Forwarded at clangd's own severity (robustness design C7, real-project plan RP3.3): a @@ -2034,6 +2097,13 @@ class ClangdEngine final : public Engine { if (read.astBuild) { sink(Json { { "kind", "ast-build" }, { "generation", generation }, { "file", read.astBuild->file }, { "version", read.astBuild->version } }); } + // P1 (design 1.4): the engine measures its cache directory at start and says when an open file is not + // the node its name refers to -- the system on which one file has two identities. + if (line.find("an open file is not the node its name refers to") != std::string_view::npos) { + sink(Json { { "kind", "file-identity" }, { "generation", generation }, { "identity", "split" } }); + } else if (line.find("module cache") != std::string_view::npos && line.find("owned, ") != std::string_view::npos) { + sink(Json { { "kind", "file-identity" }, { "generation", generation }, { "identity", "consistent" } }); + } // mcppls-clangd 0.0.13: "Rebuilding module M: ", once per module it builds. if (const std::size_t at { line.find("Rebuilding module ") }; at != std::string_view::npos) { const std::string_view rest { line.substr(at + 18) }; @@ -2102,20 +2172,25 @@ class ClangdEngine final : public Engine { // Issue 86: the Windows sandbox's "out of memory" on a grant is its fixed table of // attenuated directories running out of slots (openkal 0.20.0 recycles none), not memory. std::string message; + bool tableSpent { false }; if (unsupported) { message = std::format("clangd could not be started: {}. This system refused to start a program the way mcppls starts one " "(as PRoot does without support for execveat); only module-level features are available", why); } else { message = std::format("clangd could not be started: {}", why); if (why.starts_with("granting ") && why.find("out of memory") != std::string::npos) { - const auto readOnly { std::ranges::count_if(granted_, [](const Grant& grant) { return !grant.writable; }) }; + tableSpent = true; message += std::format(". This is not memory: the sandbox keeps a small table of granted directories that is never " - "recycled (openkal 0.20.0), and this session's {} engine starts spent it ({} read-only grants " - "each). Restarting the editor restores it, and `mcppls.grants: [\"all\"]` does without the sandbox", - generation, readOnly); + "recycled, and this session's {} engine start{} spent it ({} read-only grants each). Restart Server " + "restores it, and `mcppls.grants: [\"all\"]` does without the sandbox", + budget_.starts(), budget_.starts() == 1 ? "" : "s", read_only_grants_()); } } - add_issue_(Issue { "engine-start-failed", std::move(message), "mcppls.showLogs", "environment" }); + // Design 5: a refused start is in the log, each time; it reached only the status and a bundle. + log::warning("{} ({})", message, host_->root_directory()); + host_->record_event("engine-start-failed", Json { { "why", why }, { "starts", budget_.starts() }, { "tableSpent", tableSpent } }); + std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-start-failed"; }); + add_issue_(Issue { "engine-start-failed", std::move(message), tableSpent ? "mcppls.restartServer" : "mcppls.showLogs", "environment" }); flush_deferred_without_engine_(); host_->engine_settled(ENGINE_ID, Json::object()); host_->status_changed(); @@ -2138,9 +2213,66 @@ class ClangdEngine final : public Engine { (void)send_(lsp::make_request(engineId, "initialize", std::move(params))); } + // ---- issue 86 S4: the start budget --------------------------------------------------------- + + std::size_t read_only_grants_() const { + const auto& grants = granted_.empty() ? lastConfig_.grants : granted_; + return static_cast(std::ranges::count_if(grants, [](const Grant& grant) { return !grant.writable; })); + } + + Json start_budget_json_() const { + const std::size_t each { read_only_grants_() }; + const auto left = budget_.left(each); + return Json { { "starts", budget_.starts() }, { "slotsSpent", budget_.spent() }, { "capacity", budget_.counted() ? Json(StartBudget::CAPACITY) : Json(nullptr) }, + { "readOnlyGrantsEach", each }, { "startsLeft", left ? Json(*left) : Json(nullptr) }, { "reserve", StartBudget::RESERVE }, { "paused", paused_ } }; + } + + // An automatic restart while the sandbox's table is down to its reserve is not made: what is left is the + // person's (Restart clangd), and after that only Restart Server brings the table back. A clangd that is gone + // stays gone and mcppls's own engine answers (`paused`); one that still runs is kept as it is. + bool withhold_restart_(std::string_view reason, RestartCause cause) { + const std::size_t each { read_only_grants_() }; + if (budget_.automatic(each)) return false; + const std::size_t left { budget_.left(each).value_or(0) }; + restartAt_.reset(); + restartWhenQuiet_ = false; + restartDueAt_.reset(); + const bool gone { closedGeneration_ == generation_ || !process_ || !process_->running() }; + if (gone) { + if (!paused_) { + paused_ = true; + unavailable_ = true; + flush_deferred_without_engine_(); + host_->engine_settled(ENGINE_ID, Json::object()); + log::warning("clangd ({}) is paused: it was started {} times this session and {} start{} left before the server must be restarted; " + "not restarting it for: {}", host_->root_directory(), budget_.starts(), left, left == 1 ? " is" : "s are", reason); + host_->record_event("engine-paused", Json { { "starts", budget_.starts() }, { "startsLeft", left }, { "reason", std::string { reason } }, + { "cause", std::string { to_string(cause) } } }); + } + std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-paused" || issue.code == "engine-crash-loop"; }); + issues_.push_back(Issue { "engine-paused", + std::format("clangd is paused: it was started {} times this session and {} start{} left before the server must be restarted. mcppls's own " + "engine answers meanwhile. Restart clangd to use one, or Restart Server to have them all back", + budget_.starts(), left, left == 1 ? " is" : "s are"), + "mcppls.restartClangd", "engine" }); + } else { + log::warning("clangd ({}) is not restarted ({}): {} start{} left this session, kept for the person", host_->root_directory(), reason, left, + left == 1 ? " is" : "s are"); + host_->record_event("engine-restart-withheld", Json { { "startsLeft", left }, { "reason", std::string { reason } }, { "cause", std::string { to_string(cause) } } }); + std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-restart-withheld"; }); + issues_.push_back(Issue { "engine-restart-withheld", + std::format("clangd was not restarted ({}): {} start{} left this session before the server must be restarted. Restart clangd to use one, " + "or Restart Server to have them all back", reason, left, left == 1 ? " is" : "s are"), + "mcppls.restartClangd", "engine" }); + } + host_->status_changed(); + return true; + } + void restart_(std::string_view reason, RestartCause cause = RestartCause::recovery) { if (incompatible_) return; // it cannot run on this machine; another start changes nothing settle_exit_(); // the verdict on the last exit comes first, whatever brings the restart forward + if (cause != RestartCause::user && withhold_restart_(reason, cause)) return; // 0.0.13 W6: `user` is "never counted against clangd" -- a person's command, or a change of theirs (a toolchain, a model // replacing the provisional one); the log says that rather than "(user)", which read as if someone had asked. log::info("restarting clangd ({}): {} ({})", host_->root_directory(), reason, @@ -2829,6 +2961,7 @@ class ClangdEngine final : public Engine { const bool working { engine_working(params->value("state", std::string {})) }; buildingNow_[name] = working; if (working) workingAt_[name] = Clock::now(); + else if (!statusPath.empty()) building_.ended(base::path_key(statusPath)); } if (host_->has_document(uri)) { fileStatus_[uri] = params->value("state", std::string {}); @@ -2848,10 +2981,16 @@ class ClangdEngine final : public Engine { } if (method == lsp::method::TEXT_DOCUMENT_PUBLISH_DIAGNOSTICS) { const Json& params { message["params"] }; + // Issue 86 S2: whoever the diagnostics are for -- a prime unit, a unit opened for the index, the editor's + // file -- the build they end is no longer in flight. + if (const std::string built { host_->path_of_uri(host_->client_uri(params.value("uri", std::string {}))) }; !built.empty()) { + building_.ended(base::path_key(built)); + } if (const Json* published { lsp::find(params, "diagnostics") }; finish_prime_(params.value("uri", std::string {}), published)) return; const std::string uri { host_->client_uri(params.value("uri", std::string {})) }; const std::string diagnosedPath { host_->path_of_uri(uri) }; const std::string diagnosedKey { diagnosedPath.empty() ? std::string {} : base::path_key(diagnosedPath) }; + if (!diagnosedKey.empty()) built_without_ending_(diagnosedKey, diagnosedPath); // A doomed file's diagnostics are mcppls's own (real-project plan RP1.1, design P1): // mark_doomed_ already closed it in clangd, but a publish clangd had queued before that // close can still arrive after, and would otherwise clobber the module-failed diagnostic @@ -2953,7 +3092,8 @@ class ClangdEngine final : public Engine { const auto touched = touchedAt_.find(base::path_key(document.path)); if (!document.path.empty() && touched != touchedAt_.end() && now - touched->second < std::chrono::seconds { 10 }) suspects.insert(document.path); } - pendingExit_ = PendingExit { generation_, now, std::move(suspects), std::move(unanswered) }; + pendingExit_ = PendingExit { generation_, now, std::move(suspects), std::move(unanswered), building_.files() }; + building_.clear(); // An exit before the handshake leaves the server's own initialize waiting on this engine. One // may be a fluke the restart below mends; a second is not, and the editor is not kept // waiting for the restarts after it: initialize is answered with mcppls's own capabilities. @@ -2989,6 +3129,8 @@ class ClangdEngine final : public Engine { restartWhenQuiet_ = false; restartAt_ = std::max(now + std::chrono::seconds { 1 << std::min(crashes_.size() - 1, 6) }, restartGate_.earliest(now, RestartCause::crash)); } + // Issue 86 S4: inside the reserve no restart is coming, and the status says so now rather than when it was due. + (void)withhold_restart_(restartReason_, RestartCause::crash); host_->status_changed(); } @@ -3001,28 +3143,42 @@ class ClangdEngine final : public Engine { } // Fix plan F3: an exit settled with what clangd said about it. Its crash context names the file it crashed - // on, and only that file is set aside; without one (a signal with no context, an exit code), what it was - // asked about or given just before is, as before. The exit code is read now, when the process is surely gone. + // on, and only that file is set aside. Issue 86 (design 2026-11-12): without one, the exit's own condition and + // the builds in flight say whose it was (exit_suspects) -- an ok image that traps prints nothing, and what clangd + // was asked about or given just before is every visible file in an editor at rest. The status is read now, when + // the process is surely gone. void settle_exit_() { if (!pendingExit_) return; PendingExit exit { std::move(*pendingExit_) }; pendingExit_.reset(); - const std::optional code { process_ ? process_->exit_code() : std::nullopt }; + const auto status = process_ ? process_->exit_status() : std::nullopt; + const std::optional condition { status + ? std::optional { exit_condition(status->code, status->byEnvironment, os::host() == os::platform::windows ? ExitHost::windows : ExitHost::posix) } + : std::nullopt }; + const bool ended { condition && condition->kind != ExitKind::returned }; // by the system or from outside, not by a status of its own const auto context = crashContexts_.find(exit.generation); const bool known { context != crashContexts_.end() && !context->second.file.empty() }; - std::set suspects; + std::optional named; if (known) { std::string path { base::canonical_path(context->second.file) }; if (path.empty()) path = context->second.file; for (const auto& document : host_->documents()) { if (!document.path.empty() && base::same_path(document.path, path)) path = document.path; } - suspects.insert(path); - } else { - suspects = std::move(exit.suspects); + named = std::move(path); } + const ExitSuspects verdict { exit_suspects(condition.value_or(ExitCondition {}), named, exit.inFlight, + std::vector { exit.suspects.begin(), exit.suspects.end() }) }; + const std::set suspects { verdict.files.begin(), verdict.files.end() }; + // S1: the number the system gave, whole. A status the program returned is its own number; a condition is the + // system's (an NTSTATUS on Windows, a signal elsewhere), and `-1` is no longer written for it. + Json exitCode = nullptr; + if (condition) exitCode = condition->kind == ExitKind::returned ? Json(static_cast(condition->number)) : Json(condition->number); lastExit_ = Json { { "at", std::format("{:%FT%TZ}", std::chrono::floor(std::chrono::system_clock::now())) }, - { "exitCode", code ? Json(*code) : Json(nullptr) }, + { "exitCode", exitCode }, + { "exitCondition", ended ? Json(condition->text()) : Json(nullptr) }, + { "endedByEnvironment", ended }, + { "endedFromOutside", condition && condition->kind == ExitKind::outside }, { "crashFile", known ? Json(context->second.file) : Json(nullptr) }, { "crashAction", known ? Json(context->second.action) : Json(nullptr) }, { "exception", context != crashContexts_.end() && !context->second.exception.empty() ? Json(context->second.exception) : Json(nullptr) }, @@ -3031,31 +3187,51 @@ class ClangdEngine final : public Engine { { "stack", context != crashContexts_.end() ? Json(context->second.stack) : Json::array() }, { "clangd", Json { { "version", options_.version }, { "sha256", executable_sha256_() } } }, { "unansweredRequests", exit.unanswered }, + { "buildsInFlight", exit.inFlight }, + { "suspectRule", std::string { to_string(verdict.rule) } }, { "suspects", Json(std::vector { suspects.begin(), suspects.end() }) } }; + // D: the command that builds the suspect alone, for the reporter's shell. + if (!verdict.files.empty()) lastExit_["reproduce"] = reproduce_command_(verdict.files.front()); host_->record_event("engine-exit", Json { { "recentExits", crashes_.size() }, { "suspects", lastExit_["suspects"] }, { "exitCode", lastExit_["exitCode"] }, + { "exitCondition", lastExit_["exitCondition"] }, { "suspectRule", lastExit_["suspectRule"] }, { "crashFile", lastExit_["crashFile"] }, { "crashAction", lastExit_["crashAction"] }, { "exception", lastExit_["exception"] } }); + const std::string building { verdict.rule == ExitSuspects::Rule::in_flight + ? std::format(" while building {}", verdict.files.size() == 1 ? verdict.files.front() : std::format("{} files", verdict.files.size())) + : std::string {} }; if (known) { - log::warning("clangd crashed while building {} ({}; {}{})", context->second.file, host_->root_directory(), context->second.action, - context->second.exception.empty() ? std::string {} : std::format(", exception {}", context->second.exception)); + log::warning("clangd crashed while building {} ({}; {}{}{})", context->second.file, host_->root_directory(), context->second.action, + context->second.exception.empty() ? std::string {} : std::format(", exception {}", context->second.exception), + ended ? std::format("; ended by {}", condition->text()) : std::string {}); std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-crashed"; }); add_issue_(Issue { "engine-crashed", std::format("clangd crashed while building {}; it was restarted without it", base::file_name(context->second.file)), "mcppls.exportDiagnosticBundle" }); - } else if (code) { - log::warning("clangd exited with {} ({})", *code, host_->root_directory()); + } else if (ended) { + log::warning("clangd ended by {}{} ({})", condition->text(), building, host_->root_directory()); + std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-crashed"; }); + add_issue_(Issue { "engine-crashed", + verdict.certain + ? std::format("clangd ended by {} while building {}; that file is answered by mcppls's own engine meanwhile", condition->text(), + base::file_name(verdict.files.front())) + : std::format("clangd ended by {}{}", condition->text(), + verdict.rule == ExitSuspects::Rule::in_flight ? std::format(" while building {} files", verdict.files.size()) : std::string {}), + "mcppls.exportDiagnosticBundle" }); + } else if (condition) { + log::warning("clangd exited with {}{} ({})", static_cast(condition->number), building, host_->root_directory()); } incident_("engine-crash", lastExit_, std::vector { suspects.begin(), suspects.end() }); const auto now = Clock::now(); std::erase_if(exitStrikes_, [&](const auto& item) { return now - item.second > EXIT_STRIKE_WINDOW; }); - for (const auto& path : suspects) { + for (const auto& path : verdict.files) { const std::string key { base::path_key(path) }; // A file whose text on disk is known to stall clangd (fix plan F16) is what crashed it on Windows, where // UP-01 crashes instead of spinning: it goes back when the disk is fixed, not when a crash's term is up. const bool onDisk { diskHazards_.contains(key) }; // W5 (#59 U9): an exit nothing names a file for was set aside for two minutes on the first time: a clangd - // killed while a file was being typed in took that file from clangd. Now the first is a strike. - if (!known && !onDisk && !exitStrikes_.contains(key)) { + // killed while a file was being typed in took that file from clangd. Now the first is a strike. S2: the one + // file whose build was in flight when a fault ended clangd is not a guess, and needs no second exit. + if (!verdict.certain && !onDisk && !exitStrikes_.contains(key)) { exitStrikes_.emplace(key, now); log::info("clangd exited while working on {} ({}); it stays with clangd, and a second exit within {} minutes sets it aside", path, host_->root_directory(), EXIT_STRIKE_WINDOW.count()); @@ -3063,13 +3239,152 @@ class ClangdEngine final : public Engine { continue; } exitStrikes_.erase(key); - set_aside_(path, known ? std::format("clangd crashed while building it ({})", context->second.action) : std::string { "clangd exited while working on it" }, + set_aside_(path, known ? std::format("clangd crashed while building it ({})", context->second.action) + : ended && verdict.rule == ExitSuspects::Rule::in_flight ? std::format("clangd ended by {} while building it", condition->text()) + : std::string { "clangd exited while working on it" }, Reclaim::no, false, std::nullopt, onDisk); - if (const auto aside = aside_.find(key); aside != aside_.end() && !onDisk) aside->second.crashClosure = closure_identity_(key); + // The ledger is for a file clangd named or was building, twice over if it was not alone. A file set aside on + // the guess (it was asked about or edited lately, and clangd ended twice) keeps the short term and the way + // back on an edit it always had: holding an innocent file for a day, across sessions, would be the guess + // made worse. + if (!onDisk && verdict.rule != ExitSuspects::Rule::guessed) { + record_crash_(key, path, ended ? condition->text() : known ? std::string { "crash" } : std::string { "exit" }); + } } host_->status_changed(); } + // ---- issue 86 S3: the crash ledger --------------------------------------------------------- + + std::string ledger_path_() const { return base::join_path(host_->cache_directory(), "crash-ledger.json"); } + + // Read once a session, when something first asks: most workspaces never have one. + void read_ledger_() { + if (ledgerRead_) return; + ledgerRead_ = true; + const auto text = os::fs::read(ledger_path_()); + if (!text) return; + ledger_.from_json(Json::parse(*text, nullptr, false)); + if (ledger_.empty()) return; + // Another engine starts clean, and a record nobody met for a month is forgotten. + if (ledger_.prune(executable_sha256_(), LedgerClock::now())) write_ledger_(); + } + + void write_ledger_() { + std::error_code ignored; + if (ledger_.empty()) { + (void)std::filesystem::remove(ledger_path_(), ignored); + return; + } + if (auto written = os::fs::write_atomic(ledger_path_(), ledger_.to_json().dump(2) + "\n"); !written) { + log::info("the crash ledger of {} could not be written: {}", host_->root_directory(), written.error().message()); + } + } + + // The text clangd had, or would be given: the editor's where the file is open, the disk's otherwise. + std::string content_identity_(std::string_view key, std::string_view path) const { + for (const auto& document : host_->documents()) { + if (!document.path.empty() && base::path_key(document.path) == key) return base::sha256_hex(document.text); + } + const auto disk = os::fs::read(std::string { path }); + return disk ? base::sha256_hex(*disk) : std::string {}; + } + + CrashLedger::Verdict ledger_verdict_(const std::string& key, std::optional text = std::nullopt) { + read_ledger_(); + const CrashRecord* record { ledger_.find(key) }; + if (record == nullptr) return CrashLedger::Verdict::clean; + return ledger_.verdict(key, executable_sha256_(), text ? base::sha256_hex(*text) : content_identity_(key, record->file), + closure_identity_(key), LedgerClock::now()); + } + + // The soonest end of a term among the files set aside: when the ledger is asked again without anything changing. + void schedule_ledger_check_() { + ledgerCheckAt_.reset(); + const auto wall = LedgerClock::now(); + for (const auto& key : ledger_.keys()) { + const CrashRecord* record { ledger_.find(key) }; + if (record == nullptr || !quarantine_.contains(key) || record->retryAt <= wall) continue; + const auto at = Clock::now() + std::chrono::duration_cast(record->retryAt - wall) + std::chrono::seconds { 1 }; + if (!ledgerCheckAt_ || at < *ledgerCheckAt_) ledgerCheckAt_ = at; + } + } + + void record_crash_(const std::string& key, const std::string& path, std::string_view condition) { + read_ledger_(); + const CrashRecord& record { ledger_.exited(key, path, executable_sha256_(), condition, content_identity_(key, path), closure_identity_(key), LedgerClock::now()) }; + write_ledger_(); + schedule_ledger_check_(); + log::info("{} has ended clangd {} time{} ({}); it is not given back for {} minutes, whatever changes ({})", path, record.exits, + record.exits == 1 ? "" : "s", record.condition, record.term.count(), host_->root_directory()); + host_->record_event("crash-ledger", Json { { "file", path }, { "exits", record.exits }, { "termMinutes", record.term.count() }, { "condition", record.condition } }); + update_quarantine_issue_(); + } + + // clangd published the file's diagnostics: its build ended, and clangd did not. Whatever the ledger held + // against the file is over. + void built_without_ending_(const std::string& key, const std::string& path) { + building_.ended(key); + if (!ledgerRead_ || ledger_.empty() || quarantine_.contains(key) || ledger_.find(key) == nullptr) return; + ledger_.survived(key); + write_ledger_(); + log::info("{} built in clangd without ending it: nothing is held against it any more ({})", path, host_->root_directory()); + host_->record_event("crash-ledger-cleared", Json { { "file", path } }); + } + + // A file the editor opens that ended clangd in an earlier session, or earlier in this one: it is not handed to + // clangd until the ledger says so. True when it was kept. + bool kept_by_ledger_(const DocumentView& document) { + if (document.path.empty()) return false; + const std::string key { base::path_key(document.path) }; + if (ledger_verdict_(key, document.text) != CrashLedger::Verdict::held) return false; + const CrashRecord* record { ledger_.find(key) }; + if (!quarantine_.contains(key)) quarantine_.put(key, Clock::now()); + if (!aside_.contains(key)) aside_[key] = Aside { structure_of_text_(document.text), false, std::nullopt, false, modelOrigin_ }; + log::info("{} stays with mcppls's own engine ({}): it ended clangd {} time{} ({})", document.path, host_->root_directory(), record->exits, + record->exits == 1 ? "" : "s", record->condition); + host_->record_event("file-kept-aside", Json { { "file", document.path }, { "exits", record->exits }, { "condition", record->condition } }); + schedule_ledger_check_(); + update_quarantine_issue_(); + return true; + } + + // The files whose term is over and that may go back (CrashLedger::verdict), handed back. + void hand_back_ledger_files_() { + if (!ledgerRead_ || ledger_.empty()) return; + std::vector back; + for (const auto& [key, aside] : aside_) { + if (aside.onDisk || aside.spunOn || !quarantine_.contains(key) || ledger_.find(key) == nullptr) continue; + if (ledger_verdict_(key) == CrashLedger::Verdict::retry) back.push_back(key); + } + for (const auto& key : back) { + quarantine_.release(key); + aside_.erase(key); + for (const auto& document : host_->documents()) { + if (document.path.empty() || base::path_key(document.path) != key) continue; + log::info("handing {} back to clangd ({}): its term is over", document.path, host_->root_directory()); + host_->record_event("file-handed-back", Json { { "file", document.path }, { "why", "its term is over" } }); + if (accepting_ && !excluded_path_(document.path)) open_or_hold_(document, true); + } + } + schedule_ledger_check_(); + if (!back.empty()) update_quarantine_issue_(); + } + + // D (design 5): `clangd --check=` as this engine is started, against this workspace's engine database -- + // what a reporter had to work out by hand to show that one file ends clangd. + Json reproduce_command_(const std::string& file) const { + std::vector command; + if (!lastConfig_.runner.empty()) command.push_back(lastConfig_.runner); + command.push_back(lastConfig_.executable); + command.push_back("--check=" + file); + for (const auto& argument : clangd_arguments(lastConfig_)) { + if (argument.starts_with("--compile-commands-dir=") || argument == "--experimental-modules-support" || argument.starts_with("--modules-builder-") + || argument.starts_with("--pch-storage=") || argument.starts_with("--managed-input-root=")) command.push_back(argument); + } + return Json { { "file", file }, { "command", command }, { "directory", lastConfig_.workDirectory } }; + } + // The loader said why clangd exited: before its exit was handled, it waits for it; after, it is // handled now (the restart that exit scheduled is not wanted any more). void handle_load_failure_(const std::string& line) { @@ -3625,41 +3940,27 @@ class ClangdEngine final : public Engine { return closure; } + // What the modules a file imports were built from: each one's source, as the disk has it. Not how often they + // were built -- a restart builds the same modules again, and that is no change (issue 86 S3). std::string closure_identity_(std::string_view key) const { std::string identity; for (const auto& module : closure_of_(key)) { - const auto built = moduleBuilds_.find(module); - identity += std::format("{}={};", module, built == moduleBuilds_.end() ? 0 : built->second); + const auto source = moduleSources_.find(module); + identity += std::format("{}=", module); + if (source != moduleSources_.end()) { + const auto stamp = base::file_stamp(source->second); + identity += std::format("{}@{}:{}", source->second, stamp ? stamp->size : 0, stamp ? stamp->modified_ns : 0); + } + identity += ';'; } - return identity; + return identity.empty() ? identity : base::sha256_hex(identity); } - // W5 (#59 U7a): a file set aside after clangd exited on it goes back once its modules have been built again and - // differently -- what it crashed on may well have been one of them, in the middle of being rebuilt -- and only once - // for each such closure: an exit on it after that sets it aside for a doubled term, as before. Asked when modules - // were built, once preparation is done; a closure with a module that failed to build stays as it is. + // W5 (#59 U7a), under the ledger's rule (issue 86 S3): modules were built again, which may be the change a file + // set aside after an exit was waiting for. It makes the file eligible; it goes back when its term is over. void retry_after_closure_rebuilt_() { closureRetryCheck_ = false; - std::vector retried; - for (const auto& [key, aside] : aside_) { - if (!aside.crashClosure || aside.crashClosure->empty() || !quarantine_.contains(key)) continue; - const std::string identity { closure_identity_(key) }; - if (identity == *aside.crashClosure || crashRetried_[key].contains(identity)) continue; - if (std::ranges::any_of(closure_of_(key), [&](const std::string& module) { return modulesFailedAt_.contains(module); })) continue; - crashRetried_[key].insert(identity); - retried.push_back(key); - } - for (const auto& key : retried) { - quarantine_.release(key); - aside_.erase(key); - for (const auto& document : host_->documents()) { - if (document.path.empty() || base::path_key(document.path) != key) continue; - log::info("handing {} back to clangd ({}): the modules it imports were built again since clangd exited on it", document.path, host_->root_directory()); - host_->record_event("file-handed-back", Json { { "file", document.path }, { "why", "the modules it imports were built again" } }); - if (accepting_ && !excluded_path_(document.path)) open_or_hold_(document, true); - } - } - if (!retried.empty()) update_quarantine_issue_(); + hand_back_ledger_files_(); } void set_aside_(const std::string& path, std::string_view why, Reclaim reclaim, bool moduleFailed = false, @@ -3754,6 +4055,8 @@ class ClangdEngine final : public Engine { detail["restartBudget"] = restart_budget_json_(); detail["filesSetAside"] = quarantine_.members(); detail["lastExit"] = lastExit_; + detail["startBudget"] = start_budget_json_(); + detail["fileIdentity"] = fileIdentity_.empty() ? Json(nullptr) : Json(fileIdentity_); detail["lastPlanDiff"] = lastPlanDiff_; detail["scanFailures"] = Json { { "count", scanFailures_.count }, { "files", scanFailures_.files }, { "firstReason", scanFailures_.firstReason } }; std::vector attached { IncidentFile { "clangd.log", logRing_->text() } }; @@ -4056,7 +4359,8 @@ class ClangdEngine final : public Engine { std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-crashed" || issue.code == "engine-restart-capped"; }); std::vector released; for (auto it = aside_.begin(); it != aside_.end();) { - if (it->second.modelOrigin == "inferred" && !it->second.onDisk && !it->second.spunOn) { + if (it->second.modelOrigin == "inferred" && !it->second.onDisk && !it->second.spunOn + && ledger_verdict_(it->first) != CrashLedger::Verdict::held) { quarantine_.release(it->first); released.push_back(it->first); it = aside_.erase(it); @@ -4264,8 +4568,11 @@ class ClangdEngine final : public Engine { } void update_quarantine_issue_() { - std::erase_if(issues_, [](const Issue& issue) { return issue.code == "file-quarantined" || issue.code == "file-unsafe-on-disk"; }); + std::erase_if(issues_, [](const Issue& issue) { + return issue.code == "file-quarantined" || issue.code == "file-unsafe-on-disk" || issue.code == "file-ended-engine"; + }); std::vector members; + std::vector ended; for (const auto& key : quarantine_.members()) { // Fix plan F16, F17.5: set aside for what an autosave of a half-typed line put on disk. That is the code being // typed, not the server losing anything: said as such, with why, and never a degraded status. @@ -4274,8 +4581,25 @@ class ClangdEngine final : public Engine { std::format("{} is answered by mcppls's own engine until it is saved again: {}", base::file_name(hazard->second.path), hazard->second.why), "", "code" }); continue; } + // Issue 86 S3: a file that ended clangd says so, with how and how often -- "stopped responding" it did not. + if (const CrashRecord* record { ledger_.find(key) }) { + ended.push_back(record); + continue; + } members.push_back(key); } + if (!ended.empty()) { + const CrashRecord& first { *ended.front() }; + std::string names; + for (std::size_t i { 0 }; i < ended.size() && i < 3; ++i) names += std::format("{}{}", i == 0 ? "" : ", ", base::file_name(ended[i]->file)); + if (ended.size() > 3) names += std::format(" and {} more", ended.size() - 3); + issues_.push_back(Issue { "file-ended-engine", + ended.size() == 1 + ? std::format("{} made clangd end {} time{} ({}); it is answered by mcppls's own engine. Restart clangd tries it again", + names, first.exits, first.exits == 1 ? "" : "s", first.condition) + : std::format("{} made clangd end ({}); they are answered by mcppls's own engine. Restart clangd tries them again", names, first.condition), + "mcppls.restartClangd" }); + } if (!members.empty()) { // import-hang plan §6: the files by name, and what they still get. std::string names; @@ -4371,8 +4695,14 @@ class ClangdEngine final : public Engine { bool restart_on_request() override { if (options_.payloadCorrupt || incompatible_ || options_.executable.empty()) return false; std::vector released; + // Issue 86 S3, S4: Restart clangd is the person's "try again" for the files that ended it -- unless the session + // is down to its reserve of starts, where a file that ends clangd again would spend what is left: those stay + // with mcppls's engine and clangd comes back for everything else. + read_ledger_(); + const bool reserve { paused_ || !budget_.automatic(read_only_grants_()) }; + if (!reserve) ledger_.try_all_again(); for (auto it = aside_.begin(); it != aside_.end();) { - if (it->second.onDisk) { + if (it->second.onDisk || (reserve && ledger_.find(it->first) != nullptr)) { ++it; continue; } @@ -4385,7 +4715,11 @@ class ClangdEngine final : public Engine { } deferredReclaims_.clear(); stuckAtCap_ = false; - std::erase_if(issues_, [](const Issue& issue) { return issue.code == "engine-crashed" || issue.code == "engine-timeout" || issue.code == "engine-crash-loop"; }); + paused_ = false; + std::erase_if(issues_, [](const Issue& issue) { + return issue.code == "engine-crashed" || issue.code == "engine-timeout" || issue.code == "engine-crash-loop" || issue.code == "engine-paused" + || issue.code == "engine-restart-withheld"; + }); crashes_.clear(); exitStrikes_.clear(); crashLoops_ = 0; @@ -4710,6 +5044,8 @@ class ClangdEngine final : public Engine { backgroundRefused_.erase(refused); // it changed: it may build now } if (excluded_.contains(key) || quarantine_.contains(key) || !writtenArguments_.contains(key)) return false; + // Issue 86 S2: a unit that ended clangd when it was opened for the index is not opened for it again. + if (ledger_verdict_(key) == CrashLedger::Verdict::held) return false; if (const auto joined = joinedAt_.find(key); joined != joinedAt_.end() && now < joined->second + DATABASE_REREAD) return false; // A unit of a module that did not compile can be one clangd never finishes (robustness design C6). if (modulesFailedAt_.contains(module) || reportedFailures_.contains(module)) return false; diff --git a/src/engine/clangd/exits.cpp b/src/engine/clangd/exits.cpp new file mode 100644 index 00000000..fd2732d4 --- /dev/null +++ b/src/engine/clangd/exits.cpp @@ -0,0 +1,340 @@ +module mcppls.engine.clangd.exits; + +import std; +import nlohmann.json; + +namespace mcppls::engine::clangd { + +namespace { + +struct Named { + std::uint32_t number; + std::string_view name; + ExitKind kind; +}; + +// The statuses a program is ended with on Windows that a report of clangd has shown or can show. +constexpr std::array NTSTATUS_NAMES { { + { 0xC000001Du, "illegal instruction", ExitKind::fault }, + { 0xC0000005u, "access violation", ExitKind::fault }, + { 0xC00000FDu, "stack overflow", ExitKind::fault }, + { 0xC0000409u, "stack buffer overrun", ExitKind::fault }, + { 0xC0000374u, "heap corruption", ExitKind::fault }, + { 0xC0000094u, "integer division by zero", ExitKind::fault }, + { 0xC0000096u, "privileged instruction", ExitKind::fault }, + { 0xC0000006u, "in-page error", ExitKind::fault }, + { 0xC0000017u, "no memory", ExitKind::fault }, + { 0xC000012Du, "commit limit", ExitKind::fault }, + { 0xC0000135u, "DLL not found", ExitKind::fault }, + { 0xC0000142u, "DLL initialization failed", ExitKind::fault }, + { 0x80000003u, "breakpoint", ExitKind::fault }, + { 0x80000002u, "datatype misalignment", ExitKind::fault }, + { 0xC000013Au, "console interrupt", ExitKind::outside }, + { 0xFFFFFFFFu, "terminated", ExitKind::outside }, +} }; + +// Signals by the numbers Linux and macOS share, and the few they do not (the name then says both). +constexpr std::array SIGNAL_NAMES { { + { 1, "SIGHUP", ExitKind::outside }, + { 2, "SIGINT", ExitKind::outside }, + { 3, "SIGQUIT", ExitKind::outside }, + { 4, "SIGILL", ExitKind::fault }, + { 5, "SIGTRAP", ExitKind::fault }, + { 6, "SIGABRT", ExitKind::fault }, + { 7, "SIGBUS or SIGEMT", ExitKind::fault }, + { 8, "SIGFPE", ExitKind::fault }, + { 9, "SIGKILL", ExitKind::outside }, + { 10, "SIGUSR1 or SIGBUS", ExitKind::fault }, + { 11, "SIGSEGV", ExitKind::fault }, + { 13, "SIGPIPE", ExitKind::outside }, + { 15, "SIGTERM", ExitKind::outside }, + { 31, "SIGSYS", ExitKind::fault }, +} }; + +std::int64_t seconds_of(LedgerClock::time_point at) { + return std::chrono::duration_cast(at.time_since_epoch()).count(); +} + +LedgerClock::time_point time_of(std::int64_t seconds) { + return LedgerClock::time_point { std::chrono::duration_cast(std::chrono::seconds { seconds }) }; +} + +} // namespace + +std::string ExitCondition::number_text() const { + if (kind == ExitKind::returned) return std::format("{}", static_cast(number)); + return windows ? std::format("0x{:08X}", number) : std::format("{}", number); +} + +std::string ExitCondition::text() const { + if (kind == ExitKind::returned) return std::format("status {}", static_cast(number)); + if (windows) { + if (name.empty()) return std::format("status 0x{:08X}", number); + const bool vowel { name.front() == 'a' || name.front() == 'i' }; + return std::format("{} {} (0x{:08X})", vowel ? "an" : "a", name, number); + } + return name.empty() ? std::format("signal {}", number) : std::format("{} (signal {})", name, number); +} + +ExitCondition exit_condition(int code, bool byEnvironment, ExitHost host) { + ExitCondition condition; + condition.windows = host == ExitHost::windows; + if (!byEnvironment) { + condition.number = static_cast(code); + // A program on Windows that ends by a status of the system's is reported so by the platform; one whose starter + // does not go through it (a runner that returns its image's status) still shows the number. + if (condition.windows && (condition.number & 0xF0000000u) == 0xC0000000u) { + condition.kind = ExitKind::fault; + for (const auto& named : NTSTATUS_NAMES) { + if (named.number == condition.number) { + condition.name = named.name; + condition.kind = named.kind; + } + } + } + return condition; + } + if (condition.windows) { + // The platform reports `status & 0x7FFFFFFF`: the top bit is put back. Every status a fault ends a program with + // has it -- the errors (0xC...) and the two warnings (0x80000002, 0x80000003) alike; 0x7FFFFFFF is a termination. + const std::uint32_t reported { static_cast(code) }; + condition.number = reported == 0x7FFFFFFFu ? 0xFFFFFFFFu : (reported | 0x80000000u); + condition.kind = ExitKind::fault; + for (const auto& named : NTSTATUS_NAMES) { + if (named.number == condition.number) { + condition.name = named.name; + condition.kind = named.kind; + } + } + return condition; + } + condition.number = static_cast(code) & 0x7Fu; + condition.kind = ExitKind::fault; + for (const auto& named : SIGNAL_NAMES) { + if (named.number == condition.number) { + condition.name = named.name; + condition.kind = named.kind; + } + } + return condition; +} + +std::optional runner_fault_line(std::string_view line) { + constexpr std::string_view FAULT { "openkal: fault: " }; + constexpr std::string_view FRAME { "openkal: #" }; + while (!line.empty() && (line.back() == '\r' || line.back() == '\n' || line.back() == ' ')) line.remove_suffix(1); + if (const auto at = line.find(FAULT); at != std::string_view::npos) { + std::string_view what { line.substr(at + FAULT.size()) }; + // The address is the process's own and names nothing: the condition is what a report keeps. + if (const auto address = what.rfind(" at 0x"); address != std::string_view::npos) what = what.substr(0, address); + if (what.empty()) return std::nullopt; + return RunnerFaultLine { std::string { what }, {} }; + } + if (const auto at = line.find(FRAME); at != std::string_view::npos) { + // "#n
+": the last word, without its directories. + const auto word = line.rfind(' '); + if (word == std::string_view::npos || word < at + FRAME.size()) return std::nullopt; + std::string_view where { line.substr(word + 1) }; + if (where.find("+0x") == std::string_view::npos) return std::nullopt; + if (const auto slash = where.find_last_of("/\\"); slash != std::string_view::npos) where = where.substr(slash + 1); + return RunnerFaultLine { {}, std::string { where } }; + } + return std::nullopt; +} + +void BuildsInFlight::began(std::string_view key, std::string_view path) { + for (auto& [known, spelled] : order_) { + if (known == key) { + spelled = std::string { path }; + return; + } + } + order_.emplace_back(std::string { key }, std::string { path }); +} + +void BuildsInFlight::ended(std::string_view key) { + std::erase_if(order_, [&](const auto& item) { return item.first == key; }); +} + +void BuildsInFlight::clear() { order_.clear(); } + +std::vector BuildsInFlight::files() const { + std::vector files; + for (const auto& item : order_) files.push_back(item.second); + return files; +} + +bool BuildsInFlight::contains(std::string_view key) const { + return std::ranges::any_of(order_, [&](const auto& item) { return item.first == key; }); +} + +std::string_view to_string(ExitSuspects::Rule rule) { + switch (rule) { + case ExitSuspects::Rule::named: return "named"; + case ExitSuspects::Rule::in_flight: return "in-flight"; + case ExitSuspects::Rule::guessed: return "guessed"; + case ExitSuspects::Rule::nobody: return "nobody"; + } + return "nobody"; +} + +ExitSuspects exit_suspects(const ExitCondition& condition, std::optional named, std::vector inFlight, + std::vector guessed) { + ExitSuspects suspects; + // clangd's own word comes first, whatever ended it: it printed the context because it was crashing. + if (named && !named->empty()) { + suspects.rule = ExitSuspects::Rule::named; + suspects.files.push_back(std::move(*named)); + suspects.certain = true; + return suspects; + } + // Ended from outside: whatever it was building, the build did not end it. + if (condition.kind == ExitKind::outside) return suspects; + if (!inFlight.empty()) { + suspects.rule = ExitSuspects::Rule::in_flight; + suspects.certain = inFlight.size() == 1 && condition.kind == ExitKind::fault; + suspects.files = std::move(inFlight); + return suspects; + } + if (!guessed.empty()) { + suspects.rule = ExitSuspects::Rule::guessed; + suspects.files = std::move(guessed); + } + return suspects; +} + +const CrashRecord& CrashLedger::exited(std::string_view key, std::string_view file, std::string_view engine, std::string_view condition, + std::string_view content, std::string_view closure, LedgerClock::time_point now) { + auto found = records_.find(key); + if (found != records_.end() && found->second.record.engine != engine) { + records_.erase(found); + found = records_.end(); + } + if (found == records_.end()) { + found = records_.emplace(std::string { key }, Entry {}).first; + found->second.record.firstAt = now; + } + Entry& entry { found->second }; + CrashRecord& record { entry.record }; + entry.asked = false; + record.file = std::string { file }; + record.engine = std::string { engine }; + record.condition = std::string { condition }; + record.exits += 1; + record.lastAt = now; + record.term = TERMS[std::min(static_cast(record.exits) - 1, TERMS.size() - 1)]; + record.retryAt = now + record.term; + record.content = std::string { content }; + record.closure = std::string { closure }; + return record; +} + +CrashLedger::Verdict CrashLedger::verdict(std::string_view key, std::string_view engine, std::string_view content, std::string_view closure, + LedgerClock::time_point now) const { + const auto found = records_.find(key); + if (found == records_.end()) return Verdict::clean; + const CrashRecord& record { found->second.record }; + if (record.engine != engine) return Verdict::clean; + if (found->second.asked) return Verdict::retry; + if (now < record.retryAt) return Verdict::held; + if (record.exits >= 2 && record.content == content && record.closure == closure) return Verdict::held; + return Verdict::retry; +} + +bool CrashLedger::survived(std::string_view key) { return erase(key); } + +bool CrashLedger::try_again(std::string_view key) { + const auto found = records_.find(key); + if (found == records_.end()) return false; + found->second.asked = true; + return true; +} + +void CrashLedger::try_all_again() { + for (auto& [key, entry] : records_) entry.asked = true; +} + +bool CrashLedger::erase(std::string_view key) { + const auto found = records_.find(key); + if (found == records_.end()) return false; + records_.erase(found); + return true; +} + +bool CrashLedger::prune(std::string_view engine, LedgerClock::time_point now) { + return std::erase_if(records_, [&](const auto& item) { + return item.second.record.engine != engine || now - item.second.record.lastAt > KEPT; + }) > 0; +} + +const CrashRecord* CrashLedger::find(std::string_view key) const { + const auto found = records_.find(key); + return found == records_.end() ? nullptr : &found->second.record; +} + +std::vector CrashLedger::keys() const { + std::vector keys; + for (const auto& item : records_) keys.push_back(item.first); + return keys; +} + +Json CrashLedger::to_json() const { + Json files = Json::object(); + for (const auto& [key, entry] : records_) { + const CrashRecord& record { entry.record }; + files[key] = Json { { "file", record.file }, { "engine", record.engine }, { "condition", record.condition }, { "exits", record.exits }, + { "firstAt", seconds_of(record.firstAt) }, { "lastAt", seconds_of(record.lastAt) }, { "term", record.term.count() }, + { "retryAt", seconds_of(record.retryAt) }, { "contentAtLastExit", record.content }, { "closureAtLastExit", record.closure } }; + } + return Json { { "version", 1 }, { "files", std::move(files) } }; +} + +void CrashLedger::from_json(const Json& value) { + records_.clear(); + if (!value.is_object() || value.value("version", 0) != 1 || !value.contains("files") || !value["files"].is_object()) return; + const Json& files = value["files"]; + for (auto at = files.begin(); at != files.end(); ++at) { + const std::string key { at.key() }; + const Json& item = at.value(); + if (!item.is_object()) continue; + Entry entry; + CrashRecord& record { entry.record }; + const auto text = [&](const char* name) { return item.contains(name) && item[name].is_string() ? item[name].get() : std::string {}; }; + const auto number = [&](const char* name) { return item.contains(name) && item[name].is_number_integer() ? item[name].get() : std::int64_t { 0 }; }; + record.file = text("file"); + record.engine = text("engine"); + record.condition = text("condition"); + record.exits = static_cast(number("exits")); + record.firstAt = time_of(number("firstAt")); + record.lastAt = time_of(number("lastAt")); + record.term = std::chrono::minutes { number("term") }; + record.retryAt = time_of(number("retryAt")); + record.content = text("contentAtLastExit"); + record.closure = text("closureAtLastExit"); + if (record.file.empty() || record.engine.empty() || record.exits <= 0) continue; + records_.emplace(key, std::move(entry)); + } +} + +void StartBudget::started(std::size_t readOnlyGrants) { + ++starts_; + spent_ += readOnlyGrants; +} + +std::optional StartBudget::left(std::size_t readOnlyGrants) const { + if (capacity_ == 0) return std::nullopt; + if (readOnlyGrants == 0) return std::nullopt; + return spent_ >= capacity_ ? 0 : (capacity_ - spent_) / readOnlyGrants; +} + +bool StartBudget::automatic(std::size_t readOnlyGrants) const { + const auto fits = left(readOnlyGrants); + return !fits || *fits > RESERVE; +} + +bool StartBudget::by_person(std::size_t readOnlyGrants) const { + const auto fits = left(readOnlyGrants); + return !fits || *fits >= 1; +} + +} // namespace mcppls::engine::clangd diff --git a/src/engine/clangd/exits.cppm b/src/engine/clangd/exits.cppm new file mode 100644 index 00000000..461180be --- /dev/null +++ b/src/engine/clangd/exits.cppm @@ -0,0 +1,178 @@ +// What an exit of clangd costs the session, and no more (issue 86, design 2026-11-12): the exit keeps the number the +// system gave it, the file being built when it happened is the suspect, a file that ended the engine stays set aside +// across edits and sessions, and automatic restarts stop while the sandbox still has starts left for the person. +// Logic only: no process, no clock of its own, no file -- the engine gives each what it asks about. +export module mcppls.engine.clangd.exits; + +import std; +import nlohmann.json; + +export namespace mcppls::engine::clangd { + +using Json = nlohmann::json; + +// ---- S1: how the engine ended ---------------------------------------------------------------------- + +enum class ExitHost { posix, windows }; + +enum class ExitKind { + returned, // the program returned a status of its own choosing + fault, // the system ended it for something it did: an instruction, an access, a stack, a failed check + outside, // it was ended from outside: killed, terminated, interrupted -- nothing it was working on is to blame +}; + +struct ExitCondition { + std::uint32_t number { 0 }; // the status; an NTSTATUS on Windows, a signal's number elsewhere, for a fault or an end from outside + ExitKind kind { ExitKind::returned }; + std::string name; // "illegal instruction", "SIGSEGV", ...; empty for a returned status + // "an illegal instruction (0xC000001D)", "SIGILL (signal 4)", "status 3". + std::string text() const; + // The number as a report writes it: hexadecimal for an NTSTATUS, decimal otherwise. + std::string number_text() const; + bool windows { false }; +}; + +// What mcpplibs.os reports for a program that ended: `code`, and whether the environment ended it. On Windows the +// platform clears the status's top bit when it reports an end by the system (0xC000001D arrives as 0x4000001D), and +// gives 0x7FFFFFFF for a program terminated by its starter. +ExitCondition exit_condition(int code, bool byEnvironment, ExitHost host); + +// The runner's report of a fault in the image it runs, a line at a time (okrun where openkal reports faults): +// openkal: fault: an instruction the program may not execute (SIGILL) at 0x3e8002928f9 +// openkal: #1 0x1a3db4dafb30 /payload/clangd/openkal/bin/clangd.okb+0x5b3fb2f +// The first gives `what` ("an instruction the program may not execute (SIGILL)"), each of the others a `frame` +// ("clangd.okb+0x5b3fb2f": the image's file name and the offset in it, which is what names a function offline). +struct RunnerFaultLine { + std::string what; + std::string frame; +}; +std::optional runner_fault_line(std::string_view line); + +// ---- S2: the builds in flight ----------------------------------------------------------------------- + +// clangd says when it begins building a file ("ASTWorker building file ..."); the build is over when the file's +// diagnostics come, when its status goes idle, or when it is closed. What is in flight when clangd ends is what it +// was doing. +class BuildsInFlight { +public: + void began(std::string_view key, std::string_view path); + void ended(std::string_view key); + void clear(); + // The files, by path, in the order their builds began. + std::vector files() const; + bool contains(std::string_view key) const; + std::size_t size() const { return order_.size(); } + +private: + std::vector> order_; // (key, path) +}; + +// Who an exit is laid on (design 3.2). +struct ExitSuspects { + enum class Rule { named, in_flight, guessed, nobody }; + Rule rule { Rule::nobody }; + std::vector files; + // The files are set aside at once; otherwise each gets a strike, and the second sets it aside. + bool certain { false }; +}; +// `named`: the file clangd's own crash context names. `inFlight`: the builds in flight at the exit. `guessed`: the +// files of unanswered requests and recent edits. +ExitSuspects exit_suspects(const ExitCondition& condition, std::optional named, std::vector inFlight, + std::vector guessed); +std::string_view to_string(ExitSuspects::Rule rule); + +// ---- S3: the crash ledger --------------------------------------------------------------------------- + +using LedgerClock = std::chrono::system_clock; + +struct CrashRecord { + std::string file; + std::string engine; // the engine's sha256: another engine starts clean + std::string condition; + int exits { 0 }; + LedgerClock::time_point firstAt; + LedgerClock::time_point lastAt; + std::chrono::minutes term { 0 }; + LedgerClock::time_point retryAt; + std::string content; // sha256 of the text clangd had + std::string closure; // what the modules it imports were built from +}; + +class CrashLedger { +public: + // 2, 4, 8, 16 minutes, then an hour, then a day. + static constexpr std::array TERMS { std::chrono::minutes { 2 }, std::chrono::minutes { 4 }, std::chrono::minutes { 8 }, + std::chrono::minutes { 16 }, std::chrono::minutes { 60 }, std::chrono::minutes { 24 * 60 } }; + static constexpr std::chrono::hours KEPT { 24 * 30 }; + + enum class Verdict { + clean, // nothing is held against the file + held, // it stays with mcppls's engine + retry, // its term is over and something changed, or it has ended the engine only once: it goes back + }; + + // clangd ended on `key`. The term doubles with each exit. + const CrashRecord& exited(std::string_view key, std::string_view file, std::string_view engine, std::string_view condition, + std::string_view content, std::string_view closure, LedgerClock::time_point now); + // Whether the file may be given to clangd now. An edit, a rebuilt closure, a new session and a changed command + // each only make it eligible: it goes back when its term is over. After a second exit an unchanged file (same + // text, same closure) does not go back when the term ends; it waits for a change or for the person. + Verdict verdict(std::string_view key, std::string_view engine, std::string_view content, std::string_view closure, + LedgerClock::time_point now) const; + // clangd built the file and went on: nothing is held against it any more. + bool survived(std::string_view key); + // The person asked for it to be tried again: its term is over now, and "unchanged" does not hold it. + bool try_again(std::string_view key); + void try_all_again(); + bool erase(std::string_view key); + // Records of another engine, and those older than KEPT, go. True when any went. + bool prune(std::string_view engine, LedgerClock::time_point now); + const CrashRecord* find(std::string_view key) const; + bool empty() const { return records_.empty(); } + std::size_t size() const { return records_.size(); } + std::vector keys() const; + + Json to_json() const; + // Reads what to_json wrote; anything else leaves the ledger empty. + void from_json(const Json& value); + +private: + struct Entry { + CrashRecord record; + bool asked { false }; // try_again: the next verdict is `retry` whatever the term and the text + }; + std::map> records_; +}; + +// ---- S4: the start budget --------------------------------------------------------------------------- + +// The sandbox attenuates each read-only grant from a table of fixed size for the life of the server process, and +// (openkal-windows through 0.21.0) returns no slot. The engine counts what its starts spent, and keeps a reserve: +// automatic restarts stop while the person can still start clangd RESERVE times. +class StartBudget { +public: + static constexpr std::size_t RESERVE { 3 }; + static constexpr std::size_t CAPACITY { 64 }; + + // `capacity` zero: this system's sandbox counts nothing, and the budget never runs out. + explicit StartBudget(std::size_t capacity = 0) : capacity_ { capacity } {} + void set_capacity(std::size_t capacity) { capacity_ = capacity; } + // A start was made with `readOnlyGrants` of them. + void started(std::size_t readOnlyGrants); + std::size_t starts() const { return starts_; } + std::size_t spent() const { return spent_; } + bool counted() const { return capacity_ != 0; } + // How many more starts fit, each spending `readOnlyGrants`; nullopt where nothing is counted. + std::optional left(std::size_t readOnlyGrants) const; + // An automatic restart may be made: more than the reserve is left. + bool automatic(std::size_t readOnlyGrants) const; + // A start asked for by the person may be made: one fits. + bool by_person(std::size_t readOnlyGrants) const; + +private: + std::size_t capacity_ { 0 }; + std::size_t starts_ { 0 }; + std::size_t spent_ { 0 }; +}; + +} // namespace mcppls::engine::clangd diff --git a/src/engine/clangd/process.cpp b/src/engine/clangd/process.cpp index 54de6500..dbf4ff25 100644 --- a/src/engine/clangd/process.cpp +++ b/src/engine/clangd/process.cpp @@ -714,4 +714,11 @@ std::optional ClangdProcess::exit_code() { return connection_->exit_code(); } +std::optional ClangdProcess::exit_status() { + if (!connection_) return std::nullopt; + const auto status = connection_->exit_status(); + if (!status) return std::nullopt; + return ExitStatus { status->code, status->byEnvironment }; +} + } // namespace mcppls::engine::clangd diff --git a/src/engine/clangd/process.cppm b/src/engine/clangd/process.cppm index 015b5c61..c07d23f2 100644 --- a/src/engine/clangd/process.cppm +++ b/src/engine/clangd/process.cppm @@ -71,6 +71,17 @@ public: // How the process ended, once it has: its exit status, -1 when a signal ended it; nullopt while it // runs or where nothing can say (fix plan F3). Never blocks. virtual std::optional exit_code() { return std::nullopt; } + // The same, whole (issue 86): the status and whether the environment ended the process. A process that can say + // only the folded number answers through exit_code. + struct ExitStatus { + int code { 0 }; + bool byEnvironment { false }; + }; + virtual std::optional exit_status() { + const auto code = exit_code(); + if (!code) return std::nullopt; + return *code == -1 ? ExitStatus { 0, true } : ExitStatus { *code, false }; + } }; std::string format_fallback_style(std::string_view requested, bool supported, bool mcppProject); @@ -260,6 +271,7 @@ public: std::function()> cpu_reader() const override; std::optional pid() const override; std::optional exit_code() override; + std::optional exit_status() override; }; } // namespace mcppls::engine::clangd diff --git a/src/lsp/connection.cpp b/src/lsp/connection.cpp index 15036471..540a48f7 100644 --- a/src/lsp/connection.cpp +++ b/src/lsp/connection.cpp @@ -142,10 +142,16 @@ void Connection::stop(std::chrono::milliseconds grace) { } std::optional Connection::exit_code() { + const auto status = exit_status(); + if (!status) return std::nullopt; + return status->byEnvironment ? -1 : status->code; +} + +std::optional Connection::exit_status() { if (!process_.valid()) return std::nullopt; auto waited = process_.try_wait(); if (!waited || !waited->has_value()) return std::nullopt; - return (*waited)->by_environment ? -1 : (*waited)->code; + return ExitStatus { (*waited)->code, (*waited)->by_environment }; } } // namespace mcppls::lsp diff --git a/src/lsp/connection.cppm b/src/lsp/connection.cppm index 388841bc..ab224155 100644 --- a/src/lsp/connection.cppm +++ b/src/lsp/connection.cppm @@ -56,6 +56,13 @@ public: void stop(std::chrono::milliseconds grace); bool closed() const { return closed_.load(); } std::optional exit_code(); + // How the peer ended, as the system reported it: the status, and whether the environment ended it (a fault, a + // signal, a termination) -- the number is then the condition's, not a status the program chose. Never blocks. + struct ExitStatus { + int code { 0 }; + bool byEnvironment { false }; + }; + std::optional exit_status(); // The peer's process identifier (mcpplibs.os `process::id`), while it is started. std::optional native_pid() const { if (!process_.valid() || process_.id() == 0) return std::nullopt; diff --git a/src/orchestrator/client.cpp b/src/orchestrator/client.cpp index b1d68e0d..7fad150c 100644 --- a/src/orchestrator/client.cpp +++ b/src/orchestrator/client.cpp @@ -66,6 +66,10 @@ std::optional take_field(std::string_view& rest) { } // namespace +std::string status_progress_token(std::string_view rootKey) { return std::format("mcppls/{}", rootKey); } + +bool is_status_progress_token(std::string_view token) { return token.starts_with("mcppls/"); } + std::string make_engine_request_key(std::string_view rootKey, std::string_view engineId, int generation, const Json& engineRequestId) { std::string key { "e:" }; append_field(key, rootKey); diff --git a/src/orchestrator/client.cppm b/src/orchestrator/client.cppm index 8f3f2a6b..f5e493e3 100644 --- a/src/orchestrator/client.cppm +++ b/src/orchestrator/client.cppm @@ -49,4 +49,10 @@ struct EngineRequestKey { }; std::optional parse_engine_request_key(std::string_view key); +// The token a root's own report of a busy state travels under (`$/progress`: loading, preparing modules), and whether +// a token is one. An engine's reports carry the engine's own tokens (clangd: "backgroundIndexProgress"); who waits for +// an engine's work -- the kernel, for a query that needs the engine's index -- tells the two apart by this. +std::string status_progress_token(std::string_view rootKey); +bool is_status_progress_token(std::string_view token); + } // namespace mcppls::orchestrator diff --git a/src/orchestrator/kernel.cpp b/src/orchestrator/kernel.cpp index f28d67a6..6219b92c 100644 --- a/src/orchestrator/kernel.cpp +++ b/src/orchestrator/kernel.cpp @@ -87,8 +87,8 @@ class CaptureSink final : public ClientSink { std::map diagnostics; // latest list by URI std::map diagnosticsCounts; // publications by URI Json status; - std::set progress; // tokens with work begun and not ended - int progressBegun { 0 }; + std::set progress; // an engine's tokens with work begun and not ended + int progressBegun { 0 }; // an engine's reports begun so far std::deque engineRequests; // requests to the client, answered on the loop std::set abandoned; // requests that timed out: a late answer is dropped @@ -125,6 +125,11 @@ class CaptureSink final : public ClientSink { } else if (method == "$/progress") { const Json* kind { lsp::find_path(params, { "value", "kind" }) }; if (kind == nullptr || !kind->is_string()) return; + // The workspace reports its own busy states the same way (loading, preparing modules). That is no + // engine's work: counted here, a query that waits for the engine's index took a start slow enough to + // be reported for the index begun and ended, and asked an engine that had read no database yet + // (`query symbol` without the definition, `query refs`: "no symbol named"). + if (is_status_progress_token(token_of(params))) return; if (*kind == "begin") { progress.insert(token_of(params)); ++progressBegun; diff --git a/src/orchestrator/kernel.cppm b/src/orchestrator/kernel.cppm index 52941ebd..8845136b 100644 --- a/src/orchestrator/kernel.cppm +++ b/src/orchestrator/kernel.cppm @@ -62,7 +62,7 @@ public: std::optional diagnostics(std::string_view uri) const; // the latest published list int diagnostics_publishes(std::string_view uri) const; // how many lists were published bool indexing() const; // an engine reports work in progress - int progress_begun() const; // work reports begun so far + int progress_begun() const; // an engine's work reports begun so far // Processes events until `done` holds or `timeout` passes; true when it held. bool wait_until(const std::function& done, std::chrono::milliseconds timeout); diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index b7e1099a..1a1629f4 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -1601,16 +1601,21 @@ struct Workspace::Impl final : engine::Host { if (served->provenance.certainty == answer::Certainty::uncertain || served->doubted > 0) log::info("{}", line); else log::debug("{}", line); const bool coreAsked { !job.waitsForLate && job.next > 0 && job.answerers[job.next - 1] == coreEngine }; + const Json clientId = job.clientId; // `=`: braces make an array holding it (CWG 2137) + if (coreAsked) keep_in_background(jobId, *job.asked, *served, clientId); + job.answeredBy = "cache"; + job.coreAnswered = true; // the file's words are not added to a kept answer + // The job is finished first, as answer_without_core finishes its own (issue 86): an engine that has not sent + // the request yet -- clangd between an exit and its restart holds it -- answers the detachment at once, as + // cancelled, and that answer finished the job under this function: the client was told "cancelled" instead of + // the kept answer, and `job` was written to after it was gone. + finish_job(jobId, std::move(served->result), served->provenance); if (coreAsked) { - keep_in_background(jobId, *job.asked, *served, job.clientId); - coreEngine->detach(job.clientId); + coreEngine->detach(clientId); for (const auto& engine : engines) { - if (engine.get() != coreEngine) engine->cancel(job.clientId); + if (engine.get() != coreEngine) engine->cancel(clientId); } } - job.answeredBy = "cache"; - job.coreAnswered = true; // the file's words are not added to a kept answer - finish_job(jobId, std::move(served->result), served->provenance); return true; } @@ -3872,8 +3877,11 @@ struct Workspace::Impl final : engine::Host { const std::string after { kit ? std::format("{} {}", kit->name, kit->target) : std::string {} }; if (after == before) return; if (!kit) log::warning("no semantic kit serves {} ({})", kitTarget, root); - else log::info("semantic kit {} ({}) for {} at {}{} ({})", kit->name, kit->target, kitTarget, kit->root, - kitMissingTarget.empty() ? std::string {} : std::format(", standing in for mcppls-kit-{}", kitMissingTarget), root); + // Issue 86 (design 2026-11-12, 5): with no target asked for, this is the host's default, chosen before the project + // is described -- a report copied "for x86_64-w64-mingw32" from this line as the project's target, which was MSVC. + else log::info("semantic kit {} ({}) for {} at {}{} ({})", kit->name, kit->target, + targetSelection.requestedTarget.empty() ? std::format("this host's default target {}, the project naming none so far", kitTarget) : kitTarget, + kit->root, kitMissingTarget.empty() ? std::string {} : std::format(", standing in for mcppls-kit-{}", kitMissingTarget), root); } std::string kit_install_command() const { @@ -4092,7 +4100,7 @@ struct Workspace::Impl final : engine::Host { if (busy && !progressToken.empty() && message == lastProgressMessage) return; if (busy && progressToken.empty()) { - progressToken = std::format("mcppls/{}", key); + progressToken = status_progress_token(key); // A request, but its answer is not needed: the token's work begins with the first // $/progress either way, and a client that refuses simply ignores what follows. client.send(lsp::make_request(std::format("w:{}:p", key), "window/workDoneProgress/create", diff --git a/tests/cts/README.md b/tests/cts/README.md index ae5cc7cd..e3642e86 100644 --- a/tests/cts/README.md +++ b/tests/cts/README.md @@ -179,6 +179,7 @@ project at a fixed commit -- call the runner directly with those options. | `compdb-rejected-command` | Fix plan 2026-09-26 F6: a `compile_commands.json` whose commands carry a value the compiler rejects (`-std=c++99999`, `prepare compdb-rejected-command`), as issue #23's did: every module scan fails, and the status says the command was rejected, as an environment issue in the compiler's words | | `mcpp-emit-wait` | Fix plan 2026-09-26 F4 (D1): the producer hangs past the ten seconds after which scanned sources are planned, to its 20 s bound; mcppls's own engine answers meanwhile and clangd is given no plan until the producer answers or is given up | | `completion-keywords` | Fix plan 2026-09-26 F9 and F15, as VS Code (`client-info`): the space is a completion trigger character; a space after `import ` or `export import ` opens the module list from mcppls's own index, and one typed anywhere else, or after `import ` or `export module `, is answered at once with nothing. The module-syntax keywords are offered where each can begin a declaration and not inside a function body, merged with clangd's answer, and on their own within 1.5 s for a file clangd cannot answer for yet (a new file waiting for clangd to read a database that has it); the report counts what the space trigger cost (S3-6.2-1 to S3-6.2-5) | +| `code-actions` | issue 89, as a client that declared code action literals (`client-capabilities`), which VS Code, Neovim, Zed and CLion do: clangd's refactorings are offered on a class's name ("Define constructor", "Declare implicit copy/move members"), on a selected expression and on `auto`, in a source file and in a module's interface unit, and each one chosen is carried out -- its command goes to clangd, and the edit clangd asks the client to apply holds the text (`code-action`). The engine's image carried no refactoring through `23.1.0-mcppls.11` | | `completion-templates` | S3-6.4, as a client that declared `snippetSupport` (`client-capabilities`): the code templates `auto main() -> int { }` and `int main() { }` where a declaration begins at file scope and what was typed is a prefix of `main`, alone or after `int` or `auto`, merged with clangd's answer; an item is a snippet with the cursor in the body, its edit replacing the declaration typed so far, its `filterText` that text with `main` in full, sorted after clangd's items (`completion-item`); none inside a function's body, in a file that declares `main`, or in a named module unit, where the module-syntax keywords still are (S3-6.4-1 to S3-6.4-5) | | `workaround-canaries` | Import-hang plan §9, 0.0.9 plan M-3: one check per registered workaround with a canary, run against the payload's clangd. WA-CLANGD-001 is a `clangd-check`; WA-CLANGD-009 (completion with `--experimental-modules-support` takes over 1.8 times as long on a file including 120 generated headers, `prepare heavy-headers`) and WA-CLANGD-010 (misc-const-correctness on a filter view and a view over one) are `clangd-lsp` checks, on Linux only. A failure here means a clangd update fixed that defect and the workaround it names can be removed. WA-CLANGD-001 and 010 are `retired-by` capabilities mcppls-clangd declares, so with the bundled engine they report themselves retired | | `no-modules` | 0.0.9 plan M-3, WA-CLANGD-009: sources and headers with no module in the plan: clangd runs without `--experimental-modules-support`, hover and definition work, and the report splits request time into clangd's and the server's (M-1); a module interface written into the workspace restarts clangd with the flag, the event's reason naming WA-CLANGD-009 | @@ -302,6 +303,7 @@ between an adopted target and a pending request while a producer is still runnin | `completion-contains` | a completion label starts with `expect` (with a list, every one does; with `"exact": true`, a label is it), and none is one of `"absent"`; `insert: [line, text]` adds a line first, `edit` changes another open buffer without saving it; `"trigger"` sends the request as typing that character asked for it (`context.triggerKind` 2) | | `completion-empty` | the completion (with `"trigger"` as above) has no items, within `"within-ms"` when given (fix plan 2026-09-26 F9: a space off an import line) | | `completion-item` | the completion item labelled `"label"` (trimmed) meets `"expect"`, expectations as for `report` on the item, retried within the check's time: what a label does not show, such as `insertTextFormat`, the edit and `filterText` (S3-6.4) | +| `code-action` | the code actions at `"at"` (through `"to"`, when a range is selected) hold one titled `"title"`, retried within the check's time, and none titled as `"absent"` lists; with `"edit-contains"`, the action is carried out as an editor carries it out -- its own edit, or its command sent as `workspace/executeCommand` and the `workspace/applyEdit` the server then asks for, which the runner answers as applied -- and the text it gives this file holds that string (issue 89) | | `capabilities` | the server capabilities `initialize` answered with meet `"expect"` (expectations as for `report`) | | `references-span` | the references include every path in `expect` | | `document-symbol-contains` | the outline has a top-level symbol named `expect` | diff --git a/tests/cts/cmake.cpp b/tests/cts/cmake.cpp index b7b3b39f..668bb048 100644 --- a/tests/cts/cmake.cpp +++ b/tests/cts/cmake.cpp @@ -15,7 +15,9 @@ Fixture cmake_clangxx_msvc{"cmake-clangxx-msvc", on{"win32-x64"}}; Fixture cmake_fetchcontent_offline{"cmake-fetchcontent-offline", on{"linux-x64"}}; Fixture cmake_msvc{"cmake-msvc", on{"win32-x64"}}; Fixture cmake_msvc_bdb{"cmake-msvc-bdb", on{"win32-x64"}}; -Fixture cmake_msvc_std{"cmake-msvc-std", on{"win32-x64"}}; +// Its project opens CMake 3.31.6's experimental `import std` (the gate's UUID, and how the standard module is +// linked): a row whose image has another CMake leaves it out by this tag (ci.yml, the build-26100 row). +Fixture cmake_msvc_std{"cmake-msvc-std", on{"win32-x64"}, tag{"cmake-3.31-import-std"}}; Fixture compdb_clang_cl_std{"compdb-clang-cl-std", on{"win32-x64"}}; Fixture compdb_clangxx_msvc_std{"compdb-clangxx-msvc-std", on{"win32-x64"}}; Fixture compdb_lto_msvc{"compdb-lto-msvc", on{"linux-x64"}}; diff --git a/tests/cts/editing.cpp b/tests/cts/editing.cpp index 3ae652bc..fca518e4 100644 --- a/tests/cts/editing.cpp +++ b/tests/cts/editing.cpp @@ -1,5 +1,5 @@ -// What a person does in an editor: typing an import or a body, saving, completion, formatting and -// tidying, verifying an edit, the watch of files, and the experience of a real project. +// What a person does in an editor: typing an import or a body, saving, completion, code actions, +// formatting and tidying, verifying an edit, the watch of files, and the experience of a real project. import mcppls.cts; using namespace mcpplibs::mtest; @@ -7,6 +7,7 @@ using namespace mcppls::cts; namespace { +Fixture code_actions{"code-actions", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}}; Fixture completion_keywords{"completion-keywords", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}, resource{"machine"}}; Fixture completion_templates{"completion-templates", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}}; Fixture format_explicit{"format-explicit", on{"linux-x64", "darwin-arm64", "win32-x64"}}; diff --git a/tests/cts/engine.cpp b/tests/cts/engine.cpp index 9b432a1a..b130ed9f 100644 --- a/tests/cts/engine.cpp +++ b/tests/cts/engine.cpp @@ -15,6 +15,7 @@ Fixture clangd_crash_context{"clangd-crash-context", on{"linux-x64", "linux-arm6 Fixture clangd_exit_strike{"clangd-exit-strike", on{"linux-x64", "linux-arm64", "darwin-arm64"}, resource{"machine"}}; Fixture diagnostic_bundle{"diagnostic-bundle", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}}; Fixture engine_restarts{"engine-restarts", on{"linux-x64", "linux-arm64", "win32-x64"}, resource{"machine"}}; +Fixture engine_trap{"engine-trap", on{"linux-x64", "linux-arm64", "darwin-arm64"}, resource{"machine"}}; Fixture engine_none{"engine-none", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}, resource{"machine"}}; Fixture failure_at_base{"failure-at-base", on{"linux-x64", "linux-arm64", "darwin-arm64", "win32-x64"}, resource{"machine"}}; Fixture failure_at_base_neovim{"failure-at-base@neovim", on{"linux-x64"}, resource{"machine"}}; diff --git a/tests/cts/fixtures/clangd-crash-closure/scenario.json b/tests/cts/fixtures/clangd-crash-closure/scenario.json index b4734b88..837cf9e5 100644 --- a/tests/cts/fixtures/clangd-crash-closure/scenario.json +++ b/tests/cts/fixtures/clangd-crash-closure/scenario.json @@ -1,6 +1,6 @@ { "name": "clangd-crash-closure", - "description": "0.0.17 W5 (#59 U7a), Linux and macOS: the stand-in clangd (`prepare clangd-crash-closure`) crashes 25 seconds in with a crash context naming src/main.cpp, which the editor is typing in: main.cpp is set aside, as a file clangd crashed on is. Then another importer of hello.greet is edited and the module's partition changes, so clangd builds the modules main.cpp imports again: main.cpp goes back to clangd once, well before its two-minute term is up. Up to 0.0.16 it waited the whole term.", + "description": "0.0.17 W5 (#59 U7a) under the crash ledger's rule (issue 86, design 2026-11-12 S3), Linux and macOS: the stand-in clangd (`prepare clangd-crash-closure`) crashes 25 seconds in with a crash context naming src/main.cpp, which the editor is typing in: main.cpp is set aside, as a file clangd crashed on is. Then another importer of hello.greet is edited and the module's partition changes, so clangd builds the modules main.cpp imports again. From 0.0.17 to 0.22.0 that sent main.cpp back at once -- and so did a restart, which builds the same modules again, each return an engine start (issue 86: five returns of one file in a session, an exit after each). Now a rebuilt closure makes the file eligible and it goes back when its term is over: it is still aside right after the rebuild (R2), and back with clangd once the two minutes from the crash have passed (R3), where it builds and nothing is held against it any more.", "prepare": [ [ "{conformance}", @@ -148,9 +148,24 @@ "answer-within": 60 }, { - "id": "R2-handed-back-once-rebuilt", + "id": "R2-eligible-not-yet-back", "kind": "report", - "timeout": 60, + "timeout": 10, + "expect": [ + { + "path": "/roots/0/engines/*/details/filesSetAside", + "min-items": 1 + }, + { + "path": "/roots/0/engines/*/details/crashLedger/*/exits", + "equals": 1 + } + ] + }, + { + "id": "R3-handed-back-when-its-term-is-over", + "kind": "report", + "timeout": 170, "expect": [ { "path": "/roots/0/events/*/kind", @@ -161,6 +176,32 @@ "max-items": 0 } ] + }, + { + "id": "T3-clangd-answers-for-it-again", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 60 + }, + { + "id": "R4-nothing-held-against-it", + "kind": "report", + "timeout": 30, + "expect": [ + { + "path": "/roots/0/events/*/kind", + "equals": "crash-ledger-cleared" + }, + { + "path": "/roots/0/engines/*/details/filesSetAside", + "max-items": 0 + } + ] } ] } diff --git a/tests/cts/fixtures/code-actions/scenario.json b/tests/cts/fixtures/code-actions/scenario.json new file mode 100644 index 00000000..6ca2344a --- /dev/null +++ b/tests/cts/fixtures/code-actions/scenario.json @@ -0,0 +1,23 @@ +{ + "name": "code-actions", + "description": "Issue 89: clangd's refactorings reach the editor, and one chosen is carried out. The engine's image was linked without them through 23.1.0-mcppls.11 and every such request was answered with an empty list. The client declares code action literals, as VS Code, Neovim, Zed and CLion do: an action is then a literal that holds a command, the server routes the command to clangd, and clangd's workspace/applyEdit comes back through the server to the client.", + "server-arguments": ["--no-discover"], + "client-capabilities": { + "textDocument": { "codeAction": { "codeActionLiteralSupport": { "codeActionKind": { "valueSet": ["", "quickfix", "refactor", "refactor.extract", "refactor.inline", "refactor.rewrite", "source"] } } } }, + "workspace": { "applyEdit": true } + }, + "checks": [ + { "id": "S1", "kind": "status", "source": "inferred", "state": "ready", "engine-name": "clangd" }, + { "id": "C1", "kind": "diagnostics-empty", "file": "src/pass.cpp" }, + { "id": "A1-class-constructor", "kind": "code-action", "file": "src/pass.cpp", "at": [2, 8], "title": "Define constructor", + "edit-contains": "FinalPass(int width, int height)" }, + { "id": "A2-class-special-members", "kind": "code-action", "file": "src/pass.cpp", "at": [2, 8], "title": "Declare implicit copy/move members", + "edit-contains": "FinalPass(const FinalPass &) = default;" }, + { "id": "A3-expression", "kind": "code-action", "file": "src/pass.cpp", "at": [15, 19], "to": [15, 24], "title": "Extract subexpression to variable", + "edit-contains": "auto placeholder = w * 2;" }, + { "id": "A4-deduced-type", "kind": "code-action", "file": "src/pass.cpp", "at": [15, 5], "title": "Replace with deduced type", + "edit-contains": "int" }, + { "id": "A5-module-unit", "kind": "code-action", "file": "src/pass.cppm", "at": [4, 8], "title": "Define constructor", + "edit-contains": "ModulePass(int width, int height)" } + ] +} diff --git a/tests/cts/fixtures/code-actions/src/pass.cpp b/tests/cts/fixtures/code-actions/src/pass.cpp new file mode 100644 index 00000000..3d7ab955 --- /dev/null +++ b/tests/cts/fixtures/code-actions/src/pass.cpp @@ -0,0 +1,18 @@ +namespace render { + +class FinalPass +{ + public: + static int create(int a); + + private: + int width_; + int height_; +}; + +} + +int area(int w, int h) { + auto doubled = w * 2; + return doubled * h; +} diff --git a/tests/cts/fixtures/code-actions/src/pass.cppm b/tests/cts/fixtures/code-actions/src/pass.cppm new file mode 100644 index 00000000..0563c528 --- /dev/null +++ b/tests/cts/fixtures/code-actions/src/pass.cppm @@ -0,0 +1,15 @@ +export module render.pass; + +export namespace render { + +class ModulePass +{ + public: + static int create(int a); + + private: + int width_; + int height_; +}; + +} diff --git a/tests/cts/fixtures/engine-trap/scenario.json b/tests/cts/fixtures/engine-trap/scenario.json new file mode 100644 index 00000000..dc1ef5f8 --- /dev/null +++ b/tests/cts/fixtures/engine-trap/scenario.json @@ -0,0 +1,204 @@ +{ + "name": "engine-trap", + "description": "Issue 86 (design 2026-11-12 S1-S4), Linux and macOS: a file that makes clangd end by a fault of its own, every time it is built -- the reporter's module unit, on which the ok image trapped in a hardened library check and printed nothing. The stand-in clangd (`prepare clangd-trap`) is the payload's clangd with a watch on its log: when clangd begins building src/trap.cppm (`ASTWorker building file`), the watch ends it with SIGTRAP. Up to 0.22.0 every such exit was recorded as -1, the files set aside were the ones last asked about (main.cpp, which cannot reach the fault), and the file went back to clangd on an edit, on a rebuilt module and in every new session -- one engine start each time, until the sandbox had none left. Now the exit keeps its condition, the one file whose build was in flight is set aside after the first exit, main.cpp keeps clangd, an edit inside the term does not send the file back, a second session does not hand it to clangd at all and says so in its status, and the whole scenario costs three starts in the first session and one in the second.", + "prepare": [ + [ + "{conformance}", + "prepare", + "clangd-trap", + "{payload}" + ] + ], + "server-arguments": [ + "--clangd", + "{workspace}/stand-in/clangd", + "--no-discover" + ], + "checks": [ + { + "id": "S1", + "kind": "status", + "state": "ready", + "engine-name": "clangd" + }, + { + "id": "W1-warmed-up", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 90 + }, + { + "id": "O1-the-file-that-ends-it", + "kind": "open", + "file": "src/trap.cppm" + }, + { + "id": "R1-named-and-set-aside-at-the-first-exit", + "kind": "report", + "timeout": 30, + "expect": [ + { + "path": "/roots/0/engines/*/details/lastExit/exitCondition", + "contains": "SIGTRAP" + }, + { + "path": "/roots/0/engines/*/details/lastExit/exitCode", + "equals": 5 + }, + { + "path": "/roots/0/engines/*/details/lastExit/endedByEnvironment", + "equals": true + }, + { + "path": "/roots/0/engines/*/details/lastExit/suspectRule", + "equals": "in-flight" + }, + { + "path": "/roots/0/engines/*/details/lastExit/suspects", + "max-items": 1 + }, + { + "path": "/roots/0/engines/*/details/lastExit/suspects/*", + "each-contains": "trap.cppm" + }, + { + "path": "/roots/0/engines/*/details/lastExit/reproduce/command/*", + "contains": "--check=" + }, + { + "path": "/roots/0/engines/*/details/filesSetAside", + "min-items": 1 + }, + { + "path": "/roots/0/engines/*/details/filesSetAside", + "max-items": 1 + }, + { + "path": "/roots/0/engines/*/details/filesSetAside/*", + "each-contains": "trap.cppm" + }, + { + "path": "/roots/0/events/*/kind", + "equals": "crash-ledger" + } + ] + }, + { + "id": "R2-status-names-file-and-condition", + "kind": "status", + "issue-code": "file-ended-engine", + "issue-message": "trap.cppm" + }, + { + "id": "T1-main-keeps-clangd", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 60 + }, + { + "id": "E1-an-edit-inside-the-term", + "kind": "type-text", + "file": "src/trap.cppm", + "line": 3, + "steps": [ + "i", + "in", + "int", + "int y", + "int y;", + "int y = 2;", + "" + ], + "interval-ms": 400, + "answer-within": 15 + }, + { + "id": "R3-it-stays-aside-and-nothing-restarted", + "kind": "report", + "timeout": 20, + "expect": [ + { + "path": "/roots/0/engines/*/details/filesSetAside", + "min-items": 1 + }, + { + "path": "/roots/0/engines/*/details/filesSetAside/*", + "each-contains": "trap.cppm" + }, + { + "path": "/roots/0/engines/*/details/startBudget/starts", + "at-most": 2 + }, + { + "path": "/roots/0/engines/*/details/crashLedger/*/exits", + "equals": 1 + } + ] + }, + { + "id": "N1-a-new-session", + "kind": "restart-server" + }, + { + "id": "O2-the-editor-opens-it-again", + "kind": "open", + "file": "src/trap.cppm" + }, + { + "id": "S2-the-new-session-says-so", + "kind": "status", + "engine-name": "clangd", + "issue-code": "file-ended-engine", + "issue-message": "trap.cppm", + "timeout": 90 + }, + { + "id": "T2-main-is-answered-by-clangd", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 90 + }, + { + "id": "R4-not-handed-to-clangd", + "kind": "report", + "timeout": 20, + "expect": [ + { + "path": "/roots/0/engines/*/details/filesSetAside", + "min-items": 1 + }, + { + "path": "/roots/0/engines/*/details/filesSetAside/*", + "each-contains": "trap.cppm" + }, + { + "path": "/roots/0/engines/*/details/startBudget/starts", + "at-most": 1 + }, + { + "path": "/roots/0/engines/*/details/lastExit", + "equals": null + }, + { + "path": "/roots/0/events/*/kind", + "equals": "file-kept-aside" + } + ] + } + ] +} diff --git a/tests/cts/fixtures/engine-trap/src/greet/detail.cppm b/tests/cts/fixtures/engine-trap/src/greet/detail.cppm new file mode 100644 index 00000000..56cb7801 --- /dev/null +++ b/tests/cts/fixtures/engine-trap/src/greet/detail.cppm @@ -0,0 +1,5 @@ +export module hello.greet:detail; +import std; +export namespace hello::detail { + std::string prefix() { return "Hello, "; } +} diff --git a/tests/cts/fixtures/engine-trap/src/greet/greet.cppm b/tests/cts/fixtures/engine-trap/src/greet/greet.cppm new file mode 100644 index 00000000..21a6536d --- /dev/null +++ b/tests/cts/fixtures/engine-trap/src/greet/greet.cppm @@ -0,0 +1,7 @@ +export module hello.greet; +export import :detail; +import std; + +export namespace hello { + std::string greet(std::string_view who) { return detail::prefix() + std::string(who); } +} diff --git a/tests/cts/fixtures/engine-trap/src/main.cpp b/tests/cts/fixtures/engine-trap/src/main.cpp new file mode 100644 index 00000000..71169dbf --- /dev/null +++ b/tests/cts/fixtures/engine-trap/src/main.cpp @@ -0,0 +1,7 @@ +import std; +import hello.greet; + +int main(int argc, char* argv[]) { + std::println("{}", hello::greet("mcpp")); + return 0; +} diff --git a/tests/cts/fixtures/engine-trap/src/trap.cppm b/tests/cts/fixtures/engine-trap/src/trap.cppm new file mode 100644 index 00000000..6b222196 --- /dev/null +++ b/tests/cts/fixtures/engine-trap/src/trap.cppm @@ -0,0 +1,4 @@ +export module hello.trap; +import hello.greet; + +export int trapped() { return 1; } diff --git a/tests/unit/test_exits.cpp b/tests/unit/test_exits.cpp new file mode 100644 index 00000000..9dd947d2 --- /dev/null +++ b/tests/unit/test_exits.cpp @@ -0,0 +1,199 @@ +// Issue 86 (design 2026-11-12): how an exit of clangd is named, whose it was, what the ledger holds against a file +// that ended it, and how many starts a session's sandbox has left. +import std; +import nlohmann.json; +import mcpplibs.mtest; +import mcppls.engine.clangd.exits; + +namespace cld = mcppls::engine::clangd; +using Json = nlohmann::json; + +using namespace mcpplibs::mtest; +using namespace std::chrono_literals; + +UTest exits_tests{"exits", [] { + + "S1: an end by the system keeps the system's number, with the top bit Windows' platform clears put back"_test = [] { + // 0xC000001D arrives as 0x4000001D, ended by the environment (openkal-windows' kal_process_wait). + const auto illegal = cld::exit_condition(0x4000001D, true, cld::ExitHost::windows); + expect(illegal.number == 0xC000001Du); + expect(illegal.kind == cld::ExitKind::fault); + expect(illegal.name == "illegal instruction"); + expect(illegal.text() == "an illegal instruction (0xC000001D)") << illegal.text(); + expect(illegal.number_text() == "0xC000001D"); + expect(cld::exit_condition(0x40000005, true, cld::ExitHost::windows).text() == "an access violation (0xC0000005)"); + expect(cld::exit_condition(0x400000FD, true, cld::ExitHost::windows).text() == "a stack overflow (0xC00000FD)"); + expect(cld::exit_condition(0x40000409, true, cld::ExitHost::windows).name == "stack buffer overrun"); + // The console's interrupt and a termination by the starter are ends from outside. + expect(cld::exit_condition(0x4000013A, true, cld::ExitHost::windows).kind == cld::ExitKind::outside); + const auto terminated = cld::exit_condition(0x7FFFFFFF, true, cld::ExitHost::windows); + expect(terminated.number == 0xFFFFFFFFu && terminated.kind == cld::ExitKind::outside); + // A status nobody named is printed as it is, and is still a fault. + const auto unknown = cld::exit_condition(0x40000420, true, cld::ExitHost::windows); + expect(unknown.kind == cld::ExitKind::fault && unknown.text() == "status 0xC0000420") << unknown.text(); + // A runner that returns its image's status: the number is the system's all the same. + const auto returned = cld::exit_condition(static_cast(0xC000001Du), false, cld::ExitHost::windows); + expect(returned.kind == cld::ExitKind::fault && returned.name == "illegal instruction"); + }; + + "S1: a signal is named, and SIGKILL, SIGTERM and SIGINT are ends from outside"_test = [] { + const auto illegal = cld::exit_condition(4, true, cld::ExitHost::posix); + expect(illegal.kind == cld::ExitKind::fault && illegal.text() == "SIGILL (signal 4)") << illegal.text(); + expect(cld::exit_condition(11, true, cld::ExitHost::posix).name == "SIGSEGV"); + expect(cld::exit_condition(6, true, cld::ExitHost::posix).name == "SIGABRT"); + for (const int outside : { 2, 9, 15 }) expect(cld::exit_condition(outside, true, cld::ExitHost::posix).kind == cld::ExitKind::outside) << outside; + expect(cld::exit_condition(63, true, cld::ExitHost::posix).text() == "signal 63"); + const auto status = cld::exit_condition(3, false, cld::ExitHost::posix); + expect(status.kind == cld::ExitKind::returned && status.text() == "status 3" && status.number_text() == "3"); + }; + + "E2: the runner's report of a fault gives the condition and the frames as image offsets"_test = [] { + const auto what = cld::runner_fault_line("openkal: fault: an instruction the program may not execute (SIGILL) at 0x3e8002928f9"); + expect(fatal(what.has_value())); + expect(what->what == "an instruction the program may not execute (SIGILL)" && what->frame.empty()) << what->what; + const auto frame = cld::runner_fault_line("openkal: #1 0x1a3db4dafb30 /tmp/payload/clangd/openkal/bin/clangd.okb+0x5b3fb2f\r"); + expect(fatal(frame.has_value())); + expect(frame->what.empty() && frame->frame == "clangd.okb+0x5b3fb2f") << frame->frame; + const auto runner = cld::runner_fault_line("I[10:00:00.000] openkal: #0 0x2a238d C:\\payload\\libexec\\okrun\\windows-x86_64\\okrun.exe+0xa138d"); + expect(runner.has_value() && runner->frame == "okrun.exe+0xa138d"); + expect(!cld::runner_fault_line("I[18:31:41.396] Built module std to /cache/std.pcm").has_value()); + expect(!cld::runner_fault_line("openkal: #3 unknown").has_value()); + expect(!cld::runner_fault_line("openkal: fault: ").has_value()); + }; + + "S2: the builds in flight are the ones that began and did not end"_test = [] { + cld::BuildsInFlight building; + building.began("/p/a.cppm", "/p/a.cppm"); + building.began("/p/b.cpp", "/p/b.cpp"); + building.began("/p/a.cppm", "/p/A.cppm"); // another version of the same file: one build, the later spelling + expect(building.files() == std::vector { "/p/A.cppm", "/p/b.cpp" }); + building.ended("/p/b.cpp"); + expect(building.files() == std::vector { "/p/A.cppm" } && building.contains("/p/a.cppm") && !building.contains("/p/b.cpp")); + building.clear(); + expect(building.size() == 0u); + }; + + "S2: the suspect is the file being built; a guess gets a strike, and an end from outside names nobody"_test = [] { + const auto fault = cld::exit_condition(0x4000001D, true, cld::ExitHost::windows); + // The report's exit of 01:21:32: one file building, three files visible. 0.22.0 recorded main.cpp. + const auto one = cld::exit_suspects(fault, std::nullopt, { "src/ui/monitor_view.cppm" }, { "src/main.cpp", "src/ui/monitor_view.cppm" }); + expect(one.rule == cld::ExitSuspects::Rule::in_flight && one.certain); + expect(one.files == std::vector { "src/ui/monitor_view.cppm" }); + // Several in flight: each a strike, the next exit separates them. + const auto several = cld::exit_suspects(fault, std::nullopt, { "a.cppm", "b.cpp" }, { "c.cpp" }); + expect(several.rule == cld::ExitSuspects::Rule::in_flight && !several.certain && several.files.size() == 2u); + // Nothing in flight: what was asked about or edited, as before, a strike each. + const auto guessed = cld::exit_suspects(fault, std::nullopt, {}, { "c.cpp" }); + expect(guessed.rule == cld::ExitSuspects::Rule::guessed && !guessed.certain && guessed.files == std::vector { "c.cpp" }); + // clangd's own crash context comes first. + const auto named = cld::exit_suspects(fault, std::string { "crasher.cpp" }, { "a.cppm" }, { "c.cpp" }); + expect(named.rule == cld::ExitSuspects::Rule::named && named.certain && named.files == std::vector { "crasher.cpp" }); + // Killed: whatever it was building keeps clangd. + const auto killed = cld::exit_suspects(cld::exit_condition(9, true, cld::ExitHost::posix), std::nullopt, { "a.cppm" }, { "c.cpp" }); + expect(killed.rule == cld::ExitSuspects::Rule::nobody && killed.files.empty()); + // A status the program returned says nothing of a fault: its one build in flight gets a strike, not the verdict. + const auto status = cld::exit_suspects(cld::exit_condition(1, false, cld::ExitHost::posix), std::nullopt, { "a.cppm" }, {}); + expect(status.rule == cld::ExitSuspects::Rule::in_flight && !status.certain); + }; + + "S3: one retry per term, whatever the reason; unchanged is not eligible after the second exit"_test = [] { + cld::CrashLedger ledger; + using Verdict = cld::CrashLedger::Verdict; + const auto t0 = cld::LedgerClock::time_point { std::chrono::hours { 500000 } }; + expect(ledger.verdict("k", "engine-1", "text-1", "closure-1", t0) == Verdict::clean); + const auto& first = ledger.exited("k", "/p/monitor_view.cppm", "engine-1", "an illegal instruction (0xC000001D)", "text-1", "closure-1", t0); + expect(first.exits == 1 && first.term == 2min && first.retryAt == t0 + 2min); + // Within the term nothing sends it back: not an edit, not a rebuilt closure. + expect(ledger.verdict("k", "engine-1", "text-2", "closure-2", t0 + 1min) == Verdict::held); + // The term over, a file that ended the engine once goes back once, changed or not. + expect(ledger.verdict("k", "engine-1", "text-1", "closure-1", t0 + 2min) == Verdict::retry); + // It ends the engine again: the term doubles. + const auto& second = ledger.exited("k", "/p/monitor_view.cppm", "engine-1", "an illegal instruction (0xC000001D)", "text-1", "closure-1", t0 + 3min); + expect(second.exits == 2 && second.term == 4min && second.firstAt == t0 && second.lastAt == t0 + 3min); + expect(ledger.verdict("k", "engine-1", "text-2", "closure-1", t0 + 5min) == Verdict::held) << "an edit inside the term"; + // The term over and nothing changed: nothing could change the outcome, so it waits. + expect(ledger.verdict("k", "engine-1", "text-1", "closure-1", t0 + 8min) == Verdict::held); + expect(ledger.verdict("k", "engine-1", "text-1", "closure-1", t0 + 24h) == Verdict::held); + // A change -- the text, or what its modules were built from -- and it goes back. + expect(ledger.verdict("k", "engine-1", "text-2", "closure-1", t0 + 8min) == Verdict::retry); + expect(ledger.verdict("k", "engine-1", "text-1", "closure-2", t0 + 8min) == Verdict::retry); + // The terms: 2, 4, 8, 16 minutes, an hour, a day, and a day from then on. + std::vector terms { 2min, 4min }; + for (int i { 0 }; i < 6; ++i) terms.push_back(ledger.exited("k", "/p/monitor_view.cppm", "engine-1", "x", "t", "c", t0 + 10min).term); + expect(terms == std::vector { 2min, 4min, 8min, 16min, 60min, 1440min, 1440min, 1440min }); + }; + + "S3: another engine starts clean; the person's try again and a build that did not end clangd clear the hold"_test = [] { + cld::CrashLedger ledger; + using Verdict = cld::CrashLedger::Verdict; + const auto t0 = cld::LedgerClock::time_point { std::chrono::hours { 500000 } }; + ledger.exited("k", "/p/a.cppm", "engine-1", "SIGILL (signal 4)", "text", "closure", t0); + ledger.exited("k", "/p/a.cppm", "engine-1", "SIGILL (signal 4)", "text", "closure", t0); + expect(ledger.verdict("k", "engine-2", "text", "closure", t0) == Verdict::clean) << "the image's sha256 changed"; + expect(ledger.verdict("k", "engine-1", "text", "closure", t0) == Verdict::held); + expect(ledger.try_again("k")); + expect(ledger.verdict("k", "engine-1", "text", "closure", t0) == Verdict::retry) << "inside the term and unchanged: the person decided"; + // It ended the engine again: the request is spent. + expect(ledger.exited("k", "/p/a.cppm", "engine-1", "SIGILL (signal 4)", "text", "closure", t0 + 1min).exits == 3); + expect(ledger.verdict("k", "engine-1", "text", "closure", t0 + 2min) == Verdict::held); + expect(ledger.survived("k")); + expect(ledger.verdict("k", "engine-1", "text", "closure", t0 + 2min) == Verdict::clean && ledger.empty()); + expect(!ledger.try_again("k") && !ledger.survived("k")); + // A record of another engine is replaced, not continued. + ledger.exited("k", "/p/a.cppm", "engine-1", "x", "t", "c", t0); + expect(ledger.exited("k", "/p/a.cppm", "engine-2", "x", "t", "c", t0).exits == 1); + }; + + "S3: the ledger is a file: what was written is what a new session reads, and old or foreign records are pruned"_test = [] { + cld::CrashLedger ledger; + const auto t0 = cld::LedgerClock::time_point { std::chrono::hours { 500000 } }; + ledger.exited("/p/a.cppm", "/p/A.cppm", "engine-1", "an illegal instruction (0xC000001D)", "text", "closure", t0); + ledger.exited("/p/a.cppm", "/p/A.cppm", "engine-1", "an illegal instruction (0xC000001D)", "text", "closure", t0 + 1min); + ledger.exited("/p/b.cpp", "/p/b.cpp", "engine-0", "SIGSEGV (signal 11)", "text", "closure", t0); + const Json written = Json::parse(ledger.to_json().dump()); + expect(written["version"] == 1 && written["files"].size() == 2u); + expect(written["files"]["/p/a.cppm"]["exits"] == 2 && written["files"]["/p/a.cppm"]["term"] == 4); + cld::CrashLedger read; + read.from_json(written); + expect(fatal(read.size() == 2u)); + const cld::CrashRecord* record { read.find("/p/a.cppm") }; + expect(fatal(record != nullptr)); + expect(record->file == "/p/A.cppm" && record->exits == 2 && record->term == 4min && record->retryAt == t0 + 5min + && record->condition == "an illegal instruction (0xC000001D)" && record->content == "text" && record->closure == "closure"); + using Verdict = cld::CrashLedger::Verdict; + expect(read.verdict("/p/a.cppm", "engine-1", "text", "closure", t0 + 2min) == Verdict::held) << "a new session does not hand it to clangd"; + expect(read.prune("engine-1", t0 + 2min)) << "another engine's record goes"; + expect(read.size() == 1u && read.find("/p/b.cpp") == nullptr); + expect(!read.prune("engine-1", t0 + 24h * 29)); + expect(read.prune("engine-1", t0 + 24h * 31) && read.empty()) << "thirty days"; + // Anything else is an empty ledger, not an error. + for (const auto& bad : { Json(nullptr), Json::array(), Json { { "version", 2 }, { "files", Json::object() } }, Json { { "version", 1 }, { "files", 3 } }, + Json { { "version", 1 }, { "files", Json { { "k", Json { { "file", "" }, { "engine", "e" }, { "exits", 1 } } } } } } }) { + cld::CrashLedger empty; + empty.from_json(bad); + expect(empty.empty()) << bad.dump(); + } + }; + + "S4: the reserve is three starts; the counts are the starts made and the slots they spent"_test = [] { + // The report: four read-only grants a start, a table of 64, the 17th start refused. + cld::StartBudget budget { cld::StartBudget::CAPACITY }; + expect(budget.left(4) == std::optional { 16 }); + for (int start { 1 }; start <= 12; ++start) budget.started(4); + expect(budget.starts() == 12u && budget.spent() == 48u && budget.left(4) == std::optional { 4 }); + expect(budget.automatic(4)) << "four are left: one more than the reserve"; + budget.started(4); + expect(budget.left(4) == std::optional { 3 } && !budget.automatic(4)) << "the reserve is the person's"; + expect(budget.by_person(4)); + for (int start { 0 }; start < 3; ++start) budget.started(4); + expect(budget.starts() == 16u && budget.left(4) == std::optional { 0 } && !budget.by_person(4)) << "the 17th is the one refused"; + // CI's runners: three a start, 21 fit and the 22nd is refused. + cld::StartBudget three { cld::StartBudget::CAPACITY }; + for (int start { 0 }; start < 21; ++start) three.started(3); + expect(three.left(3) == std::optional { 0 }); + // A system that counts nothing never runs out. + cld::StartBudget uncounted; + for (int start { 0 }; start < 100; ++start) uncounted.started(9); + expect(!uncounted.counted() && !uncounted.left(9).has_value() && uncounted.automatic(9) && uncounted.by_person(9) && uncounted.starts() == 100u); + }; +}}; diff --git a/tests/unit/test_server.cpp b/tests/unit/test_server.cpp index 0eccb728..3bee73f5 100644 --- a/tests/unit/test_server.cpp +++ b/tests/unit/test_server.cpp @@ -223,6 +223,55 @@ class BuildingProcess : public cld::Process { bool running_ { false }; }; +// Issue 86: a clangd that ends by a fault of its own -- an ok image trapping in a hardened library check prints +// nothing -- the moment it begins building a file whose name contains `trap_view`. Every start does, so a file that +// is handed back ends the next one too. Other files build as BuildingProcess builds them. +class TrappingProcess : public BuildingProcess { +public: + struct Traps { + std::atomic starts { 0 }; + std::atomic trapped { 0 }; + // How the system reports the end: an illegal instruction, as this host's platform tells it (a signal's number, + // or an NTSTATUS with its top bit cleared). + ExitStatus status { 4, true }; + }; + TrappingProcess(std::shared_ptr shared, std::shared_ptr traps) : BuildingProcess(shared), shared_ { std::move(shared) }, traps_ { std::move(traps) } {} + + mcppls::base::Result start(const cld::ProcessConfig& config, MessageHandler onMessage, ClosedHandler onClosed, LogHandler onLog) override { + ++traps_->starts; + ended_.reset(); + onClosed_ = std::move(onClosed); + return BuildingProcess::start(config, std::move(onMessage), onClosed_, std::move(onLog)); + } + mcppls::base::Result send(const Json& message) override { + if (ended_) return {}; + if (message.value("method", std::string {}) == "textDocument/didOpen") { + const std::string uri { message["params"]["textDocument"]["uri"].get() }; + if (uri.contains("trap_view")) { + { + const std::lock_guard lock { shared_->mutex }; + shared_->sent.push_back(message); + } + shared_->log(std::format("I[00:00:00.000] ASTWorker building file {} version 1 with command clang++ -c", + mcppls::base::uri_to_path(uri).value_or(std::string {}))); + ended_ = traps_->status; + ++traps_->trapped; + onClosed_(); + return {}; + } + } + return BuildingProcess::send(message); + } + bool running() const override { return !ended_ && BuildingProcess::running(); } + std::optional exit_status() override { return ended_; } + +private: + std::shared_ptr shared_; + std::shared_ptr traps_; + ClosedHandler onClosed_; + std::optional ended_; +}; + // A native process double whose retiring workers keep reading the same CDB until released. class DatabaseReadingProcess : public BuildingProcess { public: @@ -2656,6 +2705,242 @@ UTest server_tests{"server", [] { } }; + "issue 86: the file being built when clangd ends is set aside at the first exit, and stays aside across an edit and a new session"_test = [] { + std::error_code ignored; + using namespace std::chrono_literals; + const std::string root { mcppls::base::join_path(base::spelled(os::dirs::temp()), + std::format("mcppls-test-ends-{}", std::chrono::steady_clock::now().time_since_epoch().count())) }; + (void)std::filesystem::create_directories(mcppls::base::join_path(root, "src"), ignored); + const std::string executable { mcppls::base::join_path(root, "clangd") }; + (void)os::fs::write(executable, "pretend-clangd"); + const std::string main { mcppls::base::join_path(root, "src/main.cpp") }; + const std::string trap { mcppls::base::join_path(root, "src/trap_view.cppm") }; + const std::string mainText { "int main() { return 0; }\n" }; + const std::string trapText { "module;\n#include \"widget.h\"\nexport module view;\nexport int measure(const Widget&);\n" }; + (void)os::fs::write(main, mainText); + (void)os::fs::write(trap, trapText); + mcppls::normalize::EnginePlan plan; + plan.entries.push_back(mcppls::normalize::EngineEntry { root, main, { "clang++", "-std=c++23", "-c", main }, "", "", {}, {} }); + plan.entries.push_back(mcppls::normalize::EngineEntry { root, trap, { "clang++", "-std=c++23", "-c", trap }, "view", "view", {}, {} }); + const eng::DocumentView mainView { mcppls::base::path_to_uri(main), main, "cpp", 1, mainText }; + const eng::DocumentView trapView { mcppls::base::path_to_uri(trap), trap, "cpp", 1, trapText }; + auto traps = std::make_shared(); + // The same fault as each system reports it, and as the engine then names it. + const bool windows { os::host() == os::platform::windows }; + if (windows) traps->status = { 0x4000001D, true }; + const std::string condition { windows ? "an illegal instruction (0xC000001D)" : "SIGILL (signal 4)" }; + const auto session = [&](EditorHost& host, std::shared_ptr shared) { + cld::Options options; + options.executable = executable; + options.version = "23.1.0"; + options.primeImplementationUnits = false; + options.processFactory = [shared, traps] { return std::make_unique(shared, traps); }; + auto engine = cld::make_engine(std::move(options)); + engine->start(host); + engine->apply(&plan); + for (int turn { 0 }; turn < 4; ++turn) { + engine->handle_timers(); + host.pump(*engine); + } + return engine; + }; + const auto run = [&](eng::Engine& engine, EditorHost& host, std::chrono::milliseconds time, const std::function& until) { + const auto end = eng::Clock::now() + time; + while (eng::Clock::now() < end && !until()) { + std::this_thread::sleep_for(20ms); + engine.handle_timers(); + host.pump(engine); + } + return until(); + }; + const auto count = [](BuildingProcess::Shared& shared, std::string_view method, const std::string& path) { + return std::ranges::count(BuildingProcess::files(shared, method), path); + }; + const std::string ledger { mcppls::base::join_path(mcppls::base::join_path(root, "cache"), "crash-ledger.json") }; + { + EditorHost host { root }; + auto shared = std::make_shared(); + auto engine = session(host, shared); + expect(fatal(traps->starts == 1)); + host.open.push_back(mainView); + engine->document(eng::DocumentEvent { eng::DocumentChange::opened, mainView }); + host.pump(*engine); + host.open.push_back(trapView); + engine->document(eng::DocumentEvent { eng::DocumentChange::opened, trapView }); + // The exit is settled half a second later, and clangd comes back a second after it ended. + expect(run(*engine, host, 6s, [&] { return traps->starts >= 2 && engine->status().accepting; })) << "clangd is started again"; + expect(traps->trapped == 1); + const Json report = engine->report(); + const Json& exit = report["lastExit"]; + expect(exit["exitCode"] == (windows ? 0xC000001Du : 4u) && exit["endedByEnvironment"] == true && exit["endedFromOutside"] == false) << exit.dump(); + expect(exit["exitCondition"] == condition) << exit["exitCondition"].dump(); + expect(exit["suspectRule"] == "in-flight" && exit["suspects"] == Json::array({ trap })) << exit.dump(); + expect(exit["buildsInFlight"] == Json::array({ trap })); + expect(exit["reproduce"]["file"] == trap && exit["reproduce"]["command"].dump().contains("--check=")) << exit["reproduce"].dump(); + expect(report["filesSetAside"].size() == 1u) << "after the first exit, and only the file that was being built: " << report["filesSetAside"].dump(); + expect(report["crashLedger"].size() == 1u && report["startBudget"]["starts"] == 2); + const auto status = engine->status(); + const auto issue = std::ranges::find_if(status.issues, [](const auto& item) { return item.code == "file-ended-engine"; }); + expect(fatal(issue != status.issues.end())) << "the status names the file and how clangd ended"; + expect(issue->message.contains("trap_view.cppm") && issue->message.contains(condition) && issue->command == "mcppls.restartClangd") << issue->message; + expect(std::ranges::none_of(status.issues, [](const auto& item) { return item.code == "file-quarantined"; })); + // The other file is clangd's again, and the one that ended it is not given to the new clangd. + expect(run(*engine, host, 3s, [&] { return count(*shared, "textDocument/didOpen", main) >= 2; })) << "main.cpp is opened in the new clangd"; + expect(count(*shared, "textDocument/didOpen", trap) == 1); + // An edit inside the term -- a new line, a new import -- does not send it back: one start a term, not one a keystroke. + const std::string edited { "module;\n#include \"widget.h\"\nexport module view;\nimport other;\nexport int measure(const Widget&);\n" }; + const eng::DocumentView editedView { trapView.uri, trap, "cpp", 2, edited }; + host.open.back() = editedView; + engine->document(eng::DocumentEvent { eng::DocumentChange::changed, editedView }); + run(*engine, host, 300ms, [] { return false; }); + expect(traps->trapped == 1 && traps->starts == 2 && count(*shared, "textDocument/didOpen", trap) == 1) << "it stays aside"; + expect(std::filesystem::exists(ledger, ignored)) << "the ledger is in the workspace's cache"; + engine->shut_down(); + } + // A new session, the same engine and the same file: the editor opens it, and clangd is not given it. + { + EditorHost host { root }; + auto shared = std::make_shared(); + auto engine = session(host, shared); + const int startsBefore { traps->starts.load() }; + host.open.push_back(trapView); + engine->document(eng::DocumentEvent { eng::DocumentChange::opened, trapView }); + host.open.push_back(mainView); + engine->document(eng::DocumentEvent { eng::DocumentChange::opened, mainView }); + run(*engine, host, 400ms, [] { return false; }); + expect(traps->trapped == 1 && traps->starts == startsBefore) << "no exit, no restart"; + expect(count(*shared, "textDocument/didOpen", trap) == 0 && count(*shared, "textDocument/didOpen", main) == 1); + expect(engine->report()["filesSetAside"].size() == 1u); + // The person's Restart clangd is "try again": it goes back, ends clangd again, and the term doubles. + expect(engine->restart_on_request()); + expect(run(*engine, host, 8s, [&] { return traps->trapped == 2 && traps->starts >= startsBefore + 2 && engine->status().accepting; })); + const Json report = engine->report(); + expect(report["filesSetAside"].size() == 1u); + const auto text = os::fs::read(ledger); + expect(fatal(text.has_value())); + const Json written = Json::parse(*text, nullptr, false); + expect(fatal(written.is_object() && written["files"].size() == 1u)) << *text; + const Json& record = *written["files"].begin(); + expect(record["exits"] == 2 && record["term"] == 4 && record["condition"] == condition) << record.dump(); + engine->shut_down(); + } + std::filesystem::remove_all(root, ignored); + }; + + "issue 86: automatic restarts stop while the sandbox still has the reserve of starts, and the refused start says how many were made"_test = [] { + std::error_code ignored; + using namespace std::chrono_literals; + const std::string root { mcppls::base::join_path(base::spelled(os::dirs::temp()), + std::format("mcppls-test-budget-{}", std::chrono::steady_clock::now().time_since_epoch().count())) }; + (void)std::filesystem::create_directories(mcppls::base::join_path(root, "src"), ignored); + const std::string executable { mcppls::base::join_path(root, "clangd") }; + (void)os::fs::write(executable, "pretend-clangd"); + const std::string main { mcppls::base::join_path(root, "src/main.cpp") }; + (void)os::fs::write(main, "int main() { return 0; }\n"); + const std::string unit { mcppls::base::join_path(root, "src/hello.cppm") }; + (void)os::fs::write(unit, "export module hello;\n"); + mcppls::normalize::EnginePlan plan; + plan.entries.push_back(mcppls::normalize::EngineEntry { root, main, { "clang++", "-std=c++23", "-c", main }, "", "", {}, {} }); + plan.entries.push_back(mcppls::normalize::EngineEntry { root, unit, { "clang++", "-std=c++23", "-c", unit }, "hello", "hello", {}, {} }); + // A clangd with a runner is granted its directories; the double counts its starts and can end on request. + struct Controls { + std::atomic starts { 0 }; + std::mutex mutex; + std::function end; + std::vector configs; + }; + class EndableProcess : public BuildingProcess { + public: + EndableProcess(std::shared_ptr shared, std::shared_ptr controls) : BuildingProcess(std::move(shared)), controls_ { std::move(controls) } {} + mcppls::base::Result start(const cld::ProcessConfig& config, MessageHandler message, ClosedHandler closed, LogHandler log) override { + ++controls_->starts; + ended_.reset(); + { + const std::lock_guard lock { controls_->mutex }; + controls_->configs.push_back(config); + controls_->end = [this, closed] { + ended_ = ExitStatus { 11, true }; // SIGSEGV, nothing in flight + closed(); + }; + } + return BuildingProcess::start(config, std::move(message), std::move(closed), std::move(log)); + } + bool running() const override { return !ended_ && BuildingProcess::running(); } + std::optional exit_status() override { return ended_; } + + private: + std::shared_ptr controls_; + std::optional ended_; + }; + auto shared = std::make_shared(); + auto controls = std::make_shared(); + cld::Options options; + options.executable = executable; + options.runner = executable; + options.version = "23.1.0"; + options.primeImplementationUnits = false; + options.processFactory = [shared, controls] { return std::make_unique(shared, controls); }; + EditorHost host { root }; + auto engine = cld::make_engine(std::move(options)); + engine->start(host); + engine->apply(&plan); + const auto run = [&](std::chrono::milliseconds time, const std::function& until) { + const auto end = eng::Clock::now() + time; + while (eng::Clock::now() < end && !until()) { + std::this_thread::sleep_for(10ms); + engine->handle_timers(); + host.pump(*engine); + } + return until(); + }; + expect(fatal(run(3s, [&] { return controls->starts == 1 && engine->status().accepting; }))); + const Json first = engine->report()["startBudget"]; + const std::size_t each { first["readOnlyGrantsEach"].get() }; + expect(fatal(each >= 1u)) << "a start grants at least one read-only directory: " << first.dump(); + expect(first["starts"] == 1 && first["slotsSpent"] == each && first["capacity"] == 64 && first["paused"] == false) << first.dump(); + const std::size_t fits { 64 / each }; + // The person restarts until four starts are left: each is made. + while (static_cast(controls->starts.load()) + 4 < fits) { + const int before { controls->starts.load() }; + expect(engine->restart_on_request()); + expect(fatal(run(3s, [&] { return controls->starts == before + 1 && engine->status().accepting; }))) << "start " << before + 1; + } + expect(engine->report()["startBudget"]["startsLeft"] == 4) << engine->report()["startBudget"].dump(); + // clangd ends: four are left, one more than the reserve, so it is restarted by itself. + const auto end_clangd = [&] { + std::function end; + { + const std::lock_guard lock { controls->mutex }; + end = controls->end; + } + end(); + }; + int before { controls->starts.load() }; + end_clangd(); + expect(fatal(run(6s, [&] { return controls->starts == before + 1 && engine->status().accepting; }))) << "an automatic restart above the reserve"; + expect(engine->report()["startBudget"]["startsLeft"] == 3); + // It ends again: three are left, and they are the person's. clangd is paused, and the status says what is left. + before = controls->starts.load(); + end_clangd(); + expect(run(6s, [&] { return engine->report()["startBudget"]["paused"] == true; })) << engine->report()["startBudget"].dump(); + run(1500ms, [] { return false; }); + expect(controls->starts == before) << "no automatic restart inside the reserve"; + auto status = engine->status(); + auto paused = std::ranges::find_if(status.issues, [](const auto& item) { return item.code == "engine-paused"; }); + expect(fatal(paused != status.issues.end())); + expect(paused->message.contains(std::format("started {} times", before)) && paused->message.contains("3 starts are left") + && paused->command == "mcppls.restartClangd") << paused->message; + expect(status.state == "unavailable" && !status.failed); + // The person's restart is made inside the reserve. + expect(engine->restart_on_request()); + expect(fatal(run(3s, [&] { return controls->starts == before + 1 && engine->status().accepting; }))); + status = engine->status(); + expect(std::ranges::none_of(status.issues, [](const auto& item) { return item.code == "engine-paused"; })); + expect(engine->report()["startBudget"]["paused"] == false && engine->report()["startBudget"]["startsLeft"] == 2); + engine->shut_down(); + std::filesystem::remove_all(root, ignored); + }; + "a clangd that exits before its handshake twice still lets the server initialize"_test = [] { std::error_code ignored; using namespace std::chrono_literals; @@ -2940,6 +3225,16 @@ UTest server_tests{"server", [] { expect(second->rootKey == "C:/work/a:b" && second->engineId == "c:d" && second->generation == 1 && second->engineRequestId == stringId); }; + "a root's own progress token is told from an engine's"_test = [] { + // The kernel waits for the engine's index by the engine's reports; the root's own report of a busy state + // (loading, preparing modules) is none of them, whatever the root's key holds. + expect(orch::is_status_progress_token(orch::status_progress_token("/work/root-a"))); + expect(orch::is_status_progress_token(orch::status_progress_token("C:/work/a:b"))); + expect(orch::is_status_progress_token(orch::status_progress_token(""))); + expect(!orch::is_status_progress_token("backgroundIndexProgress")); + expect(!orch::is_status_progress_token("")); + }; + "build files, interactive methods and state names"_test = [] { expect(orch::is_build_file("mcpp.toml") && orch::is_build_file("CMakeLists.txt") && orch::is_build_file("x.cmake")); expect(!orch::is_build_file("main.cpp") && !orch::is_build_file("greet.cppm")); diff --git a/tools/devtools/src/packaging.cpp b/tools/devtools/src/packaging.cpp index 58f64229..4487a8e0 100644 --- a/tools/devtools/src/packaging.cpp +++ b/tools/devtools/src/packaging.cpp @@ -89,6 +89,12 @@ std::optional payload_inputs(const std::string& root, const cmdli } inputs.clangd[item.substr(0, equals)] = item.substr(equals + 1); } + // `--candidate` says the engines given are an unreleased engine's. With none given the payload would be the + // lock's engine under a candidate's name: CI assembled it so from 0.0.21 on, its candidate never tested. + if (inputs.candidate && inputs.clangd.empty()) { + std::println(std::cerr, "mcppls-devtools: --candidate goes with the engine it is (--clangd =, or openkal=); none was given"); + return std::nullopt; + } const auto lockData = load_lock(root); if (!lockData) return std::nullopt; const auto known = pack::lock::platform_names(*lockData);