From 9bafb6ae34e0d0544d1931bf9d28bf469fcfe6fe Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sat, 31 Oct 2026 17:11:29 +0000 Subject: [PATCH 1/8] Simplify agent instructions and review policy - Make tests plus whole-SOW self-review the review gate; drop external reviewer requirements. - Remove the project-agent-orchestration skill; move its unique rules (cache redirection, read-only helpers, audit verdict) into AGENTS.md. - Merge project-release-tagging into project-release. - Replace mandatory SOW gate field lists with outcome-based requirements. - Rewrite project-journal-compatibility as invariants plus a test-matrix map; move detailed per-surface controls verbatim to references/validation-recipes.md. - Remove stale rules (completed program freezes, old activation order) and duplicated text; tighten AGENTS.md wording. --- .../project-agent-orchestration/SKILL.md | 133 ---- .../skills/project-docs-authoring/SKILL.md | 4 +- .../project-journal-compatibility/SKILL.md | 598 ++++------------- .../references/validation-recipes.md | 354 ++++++++++ .../skills/project-release-tagging/SKILL.md | 206 ------ .agents/skills/project-release/SKILL.md | 33 +- .agents/sow/SOW.template.md | 6 +- AGENTS.md | 608 ++++++------------ 8 files changed, 723 insertions(+), 1219 deletions(-) delete mode 100644 .agents/skills/project-agent-orchestration/SKILL.md create mode 100644 .agents/skills/project-journal-compatibility/references/validation-recipes.md delete mode 100644 .agents/skills/project-release-tagging/SKILL.md diff --git a/.agents/skills/project-agent-orchestration/SKILL.md b/.agents/skills/project-agent-orchestration/SKILL.md deleted file mode 100644 index 8fc91e44..00000000 --- a/.agents/skills/project-agent-orchestration/SKILL.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -name: project-agent-orchestration -description: "Mandatory workflow when planning, implementing, validating, reviewing, or closing SOW-driven work in this repository." ---- -# Project Agent Orchestration - -## Purpose - -Keep implementation and review reproducible, evidence-based, and bounded to -this repository. - -## Scope - -Use this skill when: - -- creating or updating implementation SOWs; -- writing prompts for external reviewers; -- deciding whether a phase can advance; -- recording review findings, fixes, validation, or production-grade status. - -Do not use this skill for: - -- trivial wording-only changes that do not require a SOW; -- end-user SDK behavior decisions covered by the journal compatibility skill. - -## Mandatory Knowledge - -- The project manager implements the active SOW, performs local validation, and - owns the outcome. Evidence: `AGENTS.md`. -- External review is the default, user-waivable technical review gate. Before - submitting code to GitHub, recommend external review when the current change - set has not already been reviewed. Evidence: `AGENTS.md`. -- Do not run external reviewers until the user authorizes them. Once - authorized, follow the standing authorization and any limits for the current - conversation. Evidence: the system-wide `external-reviewers` skill. -- The system-wide `external-reviewers` skill is the sole source of truth for - available reviewers, model selection, harnesses, prompts, authorization - scope, retries, and failure handling. Do not copy those details here. -- Current review cadence is whole-SOW batching: finish the complete active SOW, - run local validation, update SOW evidence, then review the entire SOW and - changed surface as one meaningful batch when authorized. Do not review - trivial edits or partial churn. Evidence: `AGENTS.md`. -- CRITICAL: Do not make changes outside this repository. This applies to all assistants and external reviewers. -- The only write exception outside the repository is `/tmp`; prefer `.local/` inside this repository for scratch files. -- After each implementation chunk is implemented, reviewed, and verified, prefer committing that chunk before starting the next chunk. Stage explicit files only; never use `git add -A` or `git add .`. -- Reviewer agents must run read-only. Use the system-wide - `external-reviewers` skill rather than repository-local harness instructions. -- Read-only dependency metadata commands can still write package caches. Prompts that allow dependency inspection must either forbid dependency-fetching commands or require cache/output variables under `.local/` or `/tmp`, including `GOMODCACHE`, `GOCACHE`, `GOPATH`, `npm_config_cache`, `PIP_CACHE_DIR`, `CARGO_HOME`, and equivalent tool caches. -- Journal work must not probe the live host journal. External-agent prompts for journal compatibility work must forbid `systemd-cat`, `logger`, live `journalctl` without `--file` or a repository-local `--directory`, writes to `/var/log/journal` or `/run/log/journal`, and any systemd command that changes host journal state. -- Core SDK runtime work must preserve the four-layer runtime-purity split from `AGENTS.md`: core file-format SDK, systemd/journald compatibility layer, optional identity helper, and optional writer-lock helper. Prompts must not ask agents to put host identity discovery or cooperating-writer locking back into core reader/writer paths. -- Core reader/writer runtime prompts must forbid external programs and host-observation sources in core code, including `/proc`, `/host/proc`, `/etc/machine-id`, platform registries, `sysctl`, `system_profiler`, `ps`, shell commands, subprocess APIs, and equivalent mechanisms. These are allowed only in explicitly named optional helper code and tests for those helpers. - -Canonical external-agent prompt block: - -```text -CRITICAL REPOSITORY BOUNDARY: -- Do not make changes outside this repository for any reason. -- Repository path: current repository root. -- You may inspect external references read-only when the task requires it. -- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. -- The only write exception outside the repository is /tmp. -- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. -``` - -## Best Practices - -- Split work into small SOWs with one concrete deliverable and clear acceptance gates. -- Work on exactly one active SOW at a time. -- Record external-review authorization or waiver in the active SOW. -- Use the system-wide `external-reviewers` skill at meaningful review - boundaries after whole-SOW implementation and local validation. -- Keep reviewer prompts neutral: include the original request, SOW filename, changed scope, validation commands, the canonical repository-boundary block, and ask for unwanted side effects and security issues. -- For SOWs touching runtime purity, ask reviewers to verify that core SDK code has no implicit host identity discovery, subprocess execution, or automatic writer locking, and that optional helpers are opt-in and documented separately. -- For dependency research or package metadata checks, include explicit cache redirection instructions before allowing commands such as `go get`, `go list`, `npm view`, `npm pack`, `pip download`, `pip index`, `cargo metadata`, or `cargo doc`. -- Under standing authorization, repeat review after material fixes using the - same whole-SOW scope until no blocking issues remain. - -## Bad Practices - -- Do not let any assistant or external agent edit outside this repository. -- Do not run external reviewers before user authorization. -- Do not hard-code reviewer names, model identifiers, or harness commands in - repository instructions. -- Do not let reviewers make changes; reviewers must be read-only. -- Do not let external agents run package-manager commands with default caches, because they can write under home directories even when the visible command output is read-only. -- Do not narrow follow-up reviewer prompts to only the last fix; keep the original review scope and add fix notes. -- Do not advance a SOW on "mostly ok" or unresolved production-grade doubts. - -## Workflow Checklist - -1. Confirm the active SOW has a completed pre-implementation gate. -2. Implement the approved SOW plan and record changed files and evidence. -3. Complete local validation and update the SOW. -4. If external review is not authorized, recommend it at the whole-SOW boundary - and before GitHub submission. If the user waives it, record the waiver. -5. When external review is authorized, use the system-wide - `external-reviewers` skill and record every finding with its disposition. -6. Fix verified blocking findings and, under standing authorization, re-review - after material fixes until phase gates are satisfied. -7. Run the project-local audit and record results before closing. - Inspect its full verdict: the audit can exit zero while reporting partial - state. Require the complete-and-clean verdict before advancing or closing. -8. If the audit fails, repair the issue inside this repository, rerun the audit, and record the clean result before closing. -9. Prefer committing the verified chunk before starting the next work chunk. - -## Validation Checklist - -Before claiming a phase is ready to advance: - -- Active SOW records implementation ownership, validation, review - authorization or waiver, reviewer runs when authorized, findings, and - dispositions. -- External review has either completed with blocking findings resolved and - re-reviewed, or the user waiver is recorded. -- `.agents/sow/audit.sh` has been run and relevant findings are resolved or recorded. -- No durable artifact contains raw sensitive data. -- No changed file sits outside this repository. -- Verified chunks are committed with explicit path staging before the next chunk starts unless the SOW records why commit was skipped. - -## Evidence - -- `AGENTS.md`: project roles, SOW gates, repository boundary, and review policy. -- System-wide `external-reviewers` skill: current reviewer availability, - selection, harness, prompt, authorization, retry, and failure rules. - -## Update Rules - -Update this skill when: - -- the user changes repository review policy or cadence; -- the system-wide reviewer skill changes its repository-facing contract; -- a review cycle exposes a missed orchestration failure mode; -- repository boundary or scratch-space policy changes. diff --git a/.agents/skills/project-docs-authoring/SKILL.md b/.agents/skills/project-docs-authoring/SKILL.md index fe39bec9..0c187ab2 100644 --- a/.agents/skills/project-docs-authoring/SKILL.md +++ b/.agents/skills/project-docs-authoring/SKILL.md @@ -63,8 +63,8 @@ Do not use this skill for `documentation/` (internal notes, not published). `.github/workflows/docs-examples.yml` compiles and runs examples). - When validation tooling runs `cargo`, `go`, or Python test scripts locally, export cache redirection (`CARGO_HOME`, `CARGO_TARGET_DIR`, `GOMODCACHE`, - `GOCACHE` under repo-local `.local/` paths) per the orchestration skill - cache rules. Python is used only as repository tooling here. + `GOCACHE` under repo-local `.local/` paths) per the `AGENTS.md` repository + boundary cache rules. Python is used only as repository tooling here. - Wiki pages document marker syntax inside ```markdown fences (`docs/Wiki-Publishing.md`); the validator and harness are fence-aware and ignore marker text inside fenced blocks. Preserve that property when diff --git a/.agents/skills/project-journal-compatibility/SKILL.md b/.agents/skills/project-journal-compatibility/SKILL.md index ab3a49c1..5c1cb422 100644 --- a/.agents/skills/project-journal-compatibility/SKILL.md +++ b/.agents/skills/project-journal-compatibility/SKILL.md @@ -1,476 +1,132 @@ --- name: project-journal-compatibility -description: "Mandatory compatibility rules when changing journal file readers, writers, fixtures, conformance tests, interoperability tests, or journalctl rewrites." +description: "Use when changing journal file readers, writers, verifiers, fixtures, conformance or interoperability tests, or the Rust/Go journalctl rewrites." --- # Project Journal Compatibility -## Purpose - -Keep all implementations aligned with the systemd journal file format, Netdata Rust reader/writer sources, and the project scope decisions. - -## Scope - -Use this skill when: - -- importing or adapting Rust journal code; -- implementing journal readers or writers in Rust or Go; -- porting systemd tests or fixtures; -- building shared interoperability tests; -- implementing journalctl rewrites. - -Do not use this skill for: - -- pure repository bootstrap work; -- generic SOW maintenance unrelated to journal behavior. - -## Mandatory Knowledge - -- SDK journal-format compatibility targets `systemd/systemd` tag `v260.1`. -- The portable file-backed journalctl contract targets tag `v262` - (8cc40e0c5e9234bf45084751ac53b1fbfe70b492). Run the pinned full-feature - oracle; host versions and feature-limited builds cannot certify this target. - Query-matrix qualification requires `--journalctl PATH`; every helper must use - that selected oracle, including version reporting and empty-result cases. -- Native PCRE2-8 >=10.42 is CLI-private. Enable CGO only for Go CLI builds - and tests; retain CGO-disabled SDK checks. Declare the C compiler, pkg-config - and PCRE2 development prerequisites in command-inclusive coverage/CI. -- File-backed CLIs preserve native Unix path bytes through selection, facade - opening and discovery. File patterns use platform globbing with a temporary - thread-local environment locale; literal paths bypass expansion. Windows - uses Unicode paths and brace/wildcard selection. Keep these native calls in - CLI modules and retain pure SDK checks. Test replacement-named decoys and - locale-sensitive patterns against the pinned oracle, with portable Unicode - controls in native OS CI. -- Color parity includes policy/environment selection, priority and audit - styles, PCRE2 first-match offsets, multiline preparation, verbose fields, - JSON primitives, separators and table headers. Compare ANSI bytes as well - as visible content; normalize JSON key order without discarding its styles. - Retain buffered match offsets without a second regex evaluation and keep - color-only work off the plain-output path. -- Parser recognition is insufficient. Compare encounter order and terminal - actions, raw required values, empty-versus-absent options, final-action scope, - timestamps/DST/TZif state, binary MESSAGE behavior, status and complete output. - Check calendar rendering independently from bound parsing: POSIX TZ, custom - TZDIR, leap records, transition-free footers and malformed-rule fallback must - agree in entry output, headers and tables. Omitted DST dates must exercise - the configured posixrules file, absent-file defaults and transition flags; - provider defaults alone do not establish native parity. Preserve raw epoch widths and - lazily resolve rendering zones once, without per-row file reads or zone allocation. - Do not let intentional follow termination hide a child failure or a blocked - stdout pipe. Retain native exit status and drain large backlogs to files. -- Use native OS jobs for commands with native dependencies. A CGO-disabled - cross-build of the SDK does not establish native CLI portability. -- Audit complete CLI selector components after a verified mismatch. Unit show - groups and invocation-discovery groups differ; test naming/glob boundaries, - raw field bytes, current-user conversion, ID precedence, global bounds, - directional limits, secondary-ID offsets and frozen follow selection. - Compare supplied oracle binaries explicitly, including live command builders. - Exercise platform fallback matchers on the primary test host. FreeBSD unit - brackets need the shared CLI compatibility matcher: native fnmatch lacks - named classes and collating elements. Validate malformed brackets as well - as positive matches, multibyte-to-byte retry, and native OS execution. Keep - ordinary patterns on native fnmatch without compatibility allocations. - Use indexed - eligibility and metadata getters; do not expand unrelated message payloads. -- The test scope is SDK conformance plus file-backed journalctl behavior. -- Do not implement daemon-only journalctl commands such as daemon sync, flush, rotate, or relinquish-var operations. -- journalctl already treats repeated matches for the same field as OR alternatives and different fields as AND. -- The `+` separator is a systemd journalctl disjunction feature to replicate for file-backed journalctl behavior; it is not a new extension. -- Each product language must provide two API layers: idiomatic SDK API plus a libsystemd-compatible reader facade. The facade is required unless a SOW records concrete evidence that it would require native bindings, violate the pure-language policy, or create an unsafe/unrepresentable API in that language. -- Each product-language writer must expose two compatible append shapes: a systemd-compatible raw full-field `KEY=value` byte payload layer and a structured binary-safe `{name, value}` SDK layer. Structured is the SDK hot path for already-structured producers; raw full payloads are the systemd-compatible low-level layer. -- Each product-language writer must expose the same three field-name policy layers: `RAW`, `JOURNALD`, and `JOURNAL-APP`. `JOURNALD` is the default and preserves trusted protected fields such as `_HOSTNAME`; `JOURNAL-APP` emulates untrusted application-facing journald restrictions and drops invalid caller fields; `RAW` allows every field name the DATA structure can represent directly, currently non-empty and no `=` in the field name. -- The SDK must not perform producer-specific field-name remapping. Do not add or keep SDK behavior that emits project-specific mapping marker fields or project-specific mapped field prefixes. Consumers that need naming transformations must perform them before calling the SDK. -- Core journal readers and writers are file-format implementations only. They must not execute external programs, discover host identity, read host identity sources, or enforce cooperating-writer locks by default. They must operate on explicit caller-provided paths, bytes, timestamps, machine IDs, boot IDs, seqnum IDs, and options. -- Systemd/journald compatibility is a policy/API layer above the file-format core. It may require caller-provided machine and boot identity, but it must not silently probe host identity. Callers that want automatic identity discovery must explicitly call the optional identity helper and pass the result in. -- Cooperating-writer locking is an optional helper/wrapper, independent from systemd compatibility. The journal format has a one-writer operational contract, but the lock protocol is not part of the systemd journal file format and must not be described as systemd compatibility. Core writer constructors must not expose lock-enable options; callers acquire and release the optional lock helper separately around writer use. -- Host-observation mechanisms such as `/proc`, `/host/proc`, `/etc/machine-id`, platform registries, `sysctl`, `system_profiler`, `ps`, shell commands, and subprocess APIs are forbidden in core reader/writer runtime paths. They are allowed only in explicitly named optional helper code and tests for that helper. -- Low-level writers must sort ENTRY DATA references by on-disk DATA object offset and remove duplicate DATA references by default, matching systemd. A trusted unique-payload option may skip only duplicate elimination when the caller guarantees no duplicate full payloads in one entry; it must not skip offset sorting unless a later SOW records measured evidence, compatibility validation, and a user decision for a non-byte-identity mode. -- Jenkins lookup3 hashing must match systemd `jenkins_hashlittle2()`, including the empty payload value `0xdeadbeefdeadbeef`. -- The final writer target includes compression and Forward Secure Sealing, but implementation may be phased. -- Live concurrency compatibility is mandatory for every writer and reader. A writer is not production-compatible unless stock `journalctl --file` and stock libsystemd readers can safely read the file while that writer is appending. A reader is not production-compatible unless it can safely read files while they are being appended by each repository writer and, where testable without violating repository-boundary rules, stock systemd writers. -- The live concurrency contract is one writer plus multiple readers on the same journal file. Tests must cover online state, append publication windows, tail metadata changes, entry-array growth, reader follow/tail behavior, clean close verification, and interruption/reopen scenarios for the claimed feature slice. -- Distinguish live mapping/access bounds from stable-file integrity checks. A - writer may publish rounded arena capacity before physical allocation reaches - it. Ordinary readers must still read committed entries safely; excluded-writer - snapshots, strict verification and append-open retain declared-arena checks. - Exercise the actual DATA-preparation window and physical growth boundary, not - only completed appends. `tests/interoperability/run_live_growth.py` supplies - deterministic regular/compact SDK writer-reader coverage; it supplements the - stock/live feature matrix rather than replacing it. -- Writers expose a cross-language `live_publish_every_entries` publication - cadence. The default `1` is the stock live-reader compatibility mode. `0` - disables explicit SDK live publication, and `N > 1` publishes after every - `N` appended entries for latency-tolerant consumers. Non-default modes must - not be described as stock `journalctl --follow` compatible unless their exact - mode is covered by live matrix evidence; they still must pass clean-close - verify and final reads. -- The reusable live-concurrency harness is under `tests/conformance/live/`. Writer tests should use the configured monotonically increasing sequence field, default `LIVE_SEQ`, so stock readers prove complete ordered visibility. -- Stock reader harness adapters may retry transient active-writer `ENODATA` open/read failures or partial snapshots only while the writer is active. After the writer exits, final ordered reads and `journalctl --verify --file` must pass. -- High-level directory writers must apply configured retention once when an active writer is opened or created. Existing-active reopen and eager open enforce during construction; lazy archived-only construction remains side-effect-free until the first append opens the active file, then retention runs before the first entry is written. Active/current files must remain protected and normal retention deletion lifecycle events must be reused. -- Apply the **Go/Rust Feature Parity** development rule in `AGENTS.md`; - consumer language alone does not justify a feature gap. -- Go's explicit `LogConfig.RootRetention` and Rust's - `Config::with_root_retention(true)` are exceptions to the default - machine-local lifecycle rules above: archived-only startup also maintains - the owned root, file lengths and tail times govern deletion, and live files - may be finalized for idle expiry or allocation-policy changes. Recovered - active files require caller verification/exclusion. Use header-native - `InspectRootRetention` before open and `Log.InspectRootRetention` for live - query/status inventory (Rust exposes equivalent snake_case APIs); do not - substitute Reader entry-vector expansion. - Live inventory must reject missing/replaced active files instead of reporting - false-empty success, and maintenance must validate it before mutation. - Root close, disposal and rotation MUST share live-path/header ownership - preflight before mutation or ownership detachment. Archive and lazy-creation - destinations MUST be absent. Reject safely, preserve bytes and permit healthy - rotation/creation retry after path restoration; rejected close releases only. - Test changes completed between SDK calls, with caller exclusion during calls, - and cover empty actives after policy setters as well as lazy successors. - Preserve separate safe-maintenance errors and fatal archive-mutation errors: - failed retired-file open/validation before mutation must not poison a healthy - current writer. Detach outgoing ownership before archive mutations so even - a post-rename failure leaves evidence inspectable. New root files, successors - and allocation checks must share configured options rather than inheriting - stale recovered layout. Exercise regular/compact transitions through both - append APIs, pre/post-mutation failures, and archive-before-delete events. - Fixed-header inventory MUST validate every header-addressed object extent - against the declared arena and allocated object tail, including ENTRY-array - and tail-entry hints. Reuse stable-header validation with append recovery and - excluded-writer snapshots; corruption in array contents remains outside the - header-only inventory contract. Exercise malformed header offsets in both - languages and require failed preflight to preserve journal bytes. - Stable validation MUST also enforce fixed-header object/category population - bounds and cached offset/count agreement, using complete on-disk field gates - for historical headers. The aggregate of present disjoint object categories - MUST fit within the total, including one object for each declared DATA/FIELD - hash table; those two object types have no dedicated count fields. Use checked - addition or a remaining-object budget. Test exact and insufficient budgets - with zero, one and two table declarations, including historical headers. - A present ENTRY_ARRAY counter MUST cover the distinct nonzero first/tail - array locations declared by complete header fields. Equal pointers count once; - additional arrays are permitted. Test historical counters before cached-tail - fields exist and preserve bytes on inventory/startup rejection. - Test small in-bounds contradictions, not only overflowing offsets. Do not impose stable counter consistency on live mapping validation. -- For deterministic regular uncompressed writer output, the layout target is byte-for-byte identity with the systemd v260.1 reference ingester for the accepted corpus. Writers must match systemd object order, alignment, initial allocation envelope, v260 header fields, entry-array growth, tail metadata, and hash-chain header behavior for that slice. -- Deterministic byte-identity validation must cover systemd final-state variants: online/plain close, offline close, and archived close. -- Header readers must use the on-disk `header_size` when validating object and hash-table locations. Do not compare historical file offsets against the current in-memory v260 `JournalHeader` struct size, and do not expose bytes beyond the on-disk header as newer header fields. -- Readers must not require `HEADER_INCOMPATIBLE_KEYED_HASH` for opening existing historical files. If that flag is absent, readers must use the unkeyed Jenkins lookup3 hash path and still support applicable compression flags such as LZ4. Writers may remain conservative when appending and reject unkeyed historical files unless a SOW explicitly proves safe mutation of that historical format. -- Writer append-open rejection for historical unkeyed files must be controlled and must occur before entry mutation. Do not leave assertion panics or partial state changes as the user-facing failure mode. -- High-level directory writers own reliable active replacement. If a low-level - append-open rejects an existing active file as unsupported, the directory - writer must follow systemd's reliable-open shape: use readable header - metadata to continue sequence identity where possible, move the old active - file out of the normal `.journal` set with a collision-safe `*.journal~` - disposed name, and create a fresh active file. Do not archive unsupported - active files into normal readable history. -- Journal-native API performance is part of compatibility. If the format provides a hash table, FIELD/DATA chain, DATA entry array, ENTRY array, offset, or mmap-backed path for an operation, implementations must use that path instead of scanning and expanding all entries unless a SOW records measured evidence and an explicit accepted reason. -- Field-name enumeration should use FIELD hash-table traversal on valid indexed files. A compatibility fallback to entry scanning is acceptable only when a historical or damaged FIELD table cannot be traversed safely, and that fallback must be documented in the active SOW. -- Unfiltered unique value enumeration must match systemd's algorithmic shape: find the FIELD object and walk its DATA chain, then de-duplicate across files. It must not scan every entry or expand unrelated fields. -- Changes to `journal-engine` index semantics, `FileIndexKey`, serialized index - contents, cache schema, or consumer-visible cache partitioning must explicitly - decide whether `CACHE_VERSION` changes. If bumped, record the one-time - reindex/invalidation impact in the active SOW, product scope, and release - notes. If not bumped, record why old cache entries remain semantically safe. -- Avoid unnecessary DATA decompression, repeated `FIELD=value` parsing, allocation, sorting, hashing, or syscalls in reader/writer hot paths. Treat these as regressions unless tests and benchmark evidence prove they are required. -- Relocated baseline/candidate source copies must include the tests' referenced - fixture and helper files at their original relative paths. Check those - dependencies before running the suite and verify copied bytes. Select Cargo - packages from their manifest names, not directory names or dependency aliases. -- Explorer column catalogs must be built from FIELD indexes. The Rust - `ExplorerQuery::debug_collect_column_fields_by_row_traversal` switch is a - debug-only discrepancy tool; production reports, benchmarks, or compatibility - claims with that switch enabled are invalid and must be treated as an - explorer defect. -- Indexed aggregation changes must force temporary DATA reuse during nested - posting and ENTRY reads. From `go/`, run - `go test -count=1 ./journal -run '^TestExplorerIndexedWindow'` for both layouts, - all DATA compression modes, - mmap/auto eviction and ReadAt scratch reuse. Keep possible mmap faults in child - processes. Compare full results and indexed counters, including empty/binary - values, singleton DATA, unset/repeated values, time filters, bucket integer - boundaries and partial read-error counters. A small fixture or - a successful zero-row query does not establish the default-window lifetime. - Benchmark matching and unmatched low/high-cardinality queries with - `BenchmarkExplorerIndexedValueOwnership`; record allocations separately from - elapsed time. Use `BenchmarkExplorerIndexedCompressedValueOwnership` with an - explicit `JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS` for prebuilt codec controls; - check indexed/traversal outputs, actual decompression and update totals before - timing. Freeze the same complete test helper for baseline and candidate, and - exclude fixture construction and reader teardown from timing. An incorrect or - crashing baseline cannot certify a speedup. -- Inspect CLI-helper timer boundaries before interpreting profiles or batching. - The Go reader helper opens/closes a fresh reader and checksums/sorts results - on every `--loops` iteration; its final ExplorerStats describe the last loop. - Pilot one iteration before selecting a batch count for each workload. Keep - CPU-profile post-timer activity and heap-profile forced GC separate from - query-cost claims; sampled cumulative allocation is not peak live memory. -- For Go ENTRY-slot header acquisition, run `TestEntrySlotHeader` controls on - both backends, default/tiny/non-power-of-two windows, boundary headers, - partial errors, closed readers, row pins, eviction and live array growth. - Measure `BenchmarkReaderOpen`, `BenchmarkReaderEntryHeader` and - `BenchmarkReaderAllocation` with the same complete header benchmark helper - in both builds. Record actual allocation classes with - `TestReaderAllocationShape`; raw sizeof does not establish heap cost, and - 64-bit layout preservation does not establish 32-bit layout preservation. - Exclude candidate-only direct-helper correctness tests from both timing - binaries, document that exclusion and run those tests separately. Preserve - the baseline validator's policy when isolating acquisition changes from - other paused work, and verify matching pure-build flags and source hashes. - For uncertain small-open timings, compare the same immutable five-entry - corpus using `JOURNAL_SDK_READER_OPEN_SMALL_CORPUS` and retain a generated-file - control with those same binaries. Preserve earlier results: a changed helper - can alter test linking even when production source is byte-identical. -- For Go native-file opener changes, run `TestNativeReaderFile` uncached on - amd64 and 386. Verify read-only/blocking/close-on-exec flags, typed path - errors, symlink identity, descriptor closure and sparse reads above 2 GiB. - Preserve interrupted-open retry, native malformed-file cleanup and existing - `.zst` source/staging behavior; build non-Linux delegation targets. The direct - Linux opener bypasses Go's internal test-cache input hook, so use `-count=1` - for fixture-dependent tests and retain explicit immutable fixture hashes. - Exclude candidate-only opener contract tests from both timing builds, as for - direct header tests. Primitive syscall savings do not establish full-reader - performance; repeat small, large and higher-level reader/query controls. -- For cached constructor statistics, run `TestReaderAccessorInitial` for - empty/nonempty files and initial mmap failure. Verify explicit counters, - selected/requested modes and fallback reason against a fresh statistics - query. Windows zero-byte raw files fail explicit mmap construction and Auto - falls back to ReadAt; preserve that platform behavior in test expectations. - Exclude candidate-only statistics tests from both timing builds. -- Smoke tests are not sufficient evidence for production compatibility. SOW validation must record exact stock systemd version, commands/helpers, stress duration, entry counts, reader counts, and failure criteria. -- Common compression-library dependencies are allowed after dependency review. Journal parsing/writing must not depend on systemd/libjournal; CGO and linking to system journal libraries remain disallowed unless the user explicitly changes those separate constraints. -- Every external-agent prompt must include the canonical repository-boundary block verbatim from `AGENTS.md` or `.agents/skills/project-agent-orchestration/SKILL.md`. -- Compatibility probing must not use the workstation's live journal. Do not run `systemd-cat`, `logger`, live `journalctl`, or write under `/var/log/journal` or `/run/log/journal`; stock-reader validation must use `journalctl --file` or repository-local `--directory` fixtures only. - -## Best Practices - -- Treat systemd source and tests as the compatibility authority when documentation and implementation disagree. -- Keep shared tests language-neutral and run them against both product SDKs. -- Add shared live-concurrency tests before accepting a writer or reader as production-compatible. -- Prove cross-language interoperability with files written by each implementation and read by every implementation. -- Prove stock-reader interoperability while repository writers are actively appending, not only after close. -- For live writer/reader compatibility changes, run - `tests/interoperability/run_live_matrix.py` and require the feature matrix to - cover regular, zstd/xz/lz4 DATA compression, compact, compact plus DATA - compression, and sealed/FSS files. The matrix must include stock - `journalctl --file`, stock libsystemd, all repository readers, final - `journalctl --verify --file`, and `--verify-key` for sealed files. -- For compressed-DATA writer changes, run `tests/interoperability/run_compression_matrix.py` - and require the structural oracle to validate object order, offsets, flags, - counters, hash-chain consistency, tail metadata, references, and compression - flags plus stock journalctl, stock libsystemd, and all repository reader - checks. -- For compact writer changes, run `tests/interoperability/run_compact_matrix.py` - and require the structural oracle to validate compact object layout, - 32-bit compact offset constraints, optional compression flags, stock - journalctl, stock libsystemd, and all repository reader checks. -- For deterministic writer layout changes, run - `tests/interoperability/run_byte_identity.py --final-state all` and require - `all_equal: true` for the accepted uncompressed corpus unless the active SOW - records exact byte deltas and a user decision changes the pass condition. - The accepted corpus must keep deliberate DATA hash-bucket collisions, and the - byte-identity runner must fail if `data_hash_chain_depth` does not match the - systemd reference value for every language and final state. -- For writer lock changes, run `tests/interoperability/run_lock_matrix.py` - and require all SDK writer pairs to reject a second active writer before the - contender publishes a ready file, plus stale-lock cleanup after crashed - writers. - This matrix applies to optional lock helpers/wrappers, not to the core writer - constructors after SOW-0071. -- For directory reader or file-backed `journalctl --directory` changes, run - `tests/interoperability/run_directory_matrix.py` and require stock - journalctl plus Rust and Go rewrites to agree on traversal, - ordering, filtering, boot listing, corrupt-file skipping, empty directories, - and repository `.journal.zst` directory discovery. -- For mixed directory feature changes, run - `tests/interoperability/run_mixed_directory_matrix.py` and require stock - journalctl plus Rust and Go rewrites to agree on mixed - regular/compact files, uncompressed and zstd/xz/lz4 DATA-compressed files, - sealed/unsealed reads, directory verification with and without keys, and the - repository whole-file `.journal.zst` directory extension. -- For file-backed journalctl query or follow changes, run - `tests/interoperability/run_journalctl_query_matrix.py` and require stock - journalctl plus Rust and Go rewrites to agree on - `--since`/`--until` realtime ranges, `--boot` descriptors, and live - `--follow` reads from actively appended file and directory inputs, including - no-tail, default-tail, and boot-plus-since cases. -- For journalctl parser or option-surface changes, run - `tests/parser-parity/check_v260_manifest.py` and - `tests/parser-parity/run_parser_parity.py`. Parser parity must cover option - names, short aliases, output modes, parser-required conflicts, recognized - options interspersed after show-action match arguments, stock short-option - attached values and clusters such as `-n2`, `-ball`, and `-rn2`, and - optional-argument edge cases where bare and explicit-empty forms differ, such - as `--boot` versus `--boot=`, bare `--lines` versus `--lines=`, and bare - `--case-sensitive` versus `--case-sensitive=`. -- CLI integer controls must exercise native base prefixes/octal, leading and - trailing whitespace, signed 32-bit descriptor limits and explicit ID-offset - signs. Preserve raw Unix match operands separately from raw grep patterns. -- Short-output controls must include multiline and distinct-suffix messages, - width/line/byte thresholds and repeated MESSAGE values; grep/cat select the - first value while short output selects the last. Prove bounded reads stop - before further matching, boot observation precedes grep, and follow-tail - selection/resume state includes nonmatching native rows. -- Check short MESSAGE preparation before printability/truncation: tabs, - completed and malformed CSI/OSC sequences, CR boundaries and show-all raw - controls. JSON duplicate controls must include a binary first occurrence; - a byte array must not become the repeated-values collection. -- Cursor controls must compare final native position as well as selected - output: grep misses, bound crossings, zero counts and after-terminal seeks. - Verify follow cursor-file contents after a skipped suffix or no matches, - rather than accepting file existence alone. Size parsing needs compound - suffix, fraction, unit-order/case and overflow controls, including terminal - help encounter order. Preserve raw source-option presence on Unix. -- For verifier changes, run `tests/interoperability/run_verify_matrix.py` and - require stock `journalctl --verify --file` plus Rust and Go - verification paths to agree on positive regular, zstd/xz/lz4 DATA-compressed, - compact, compact plus DATA-compressed, and sealed files, and on negative - object type, object size, payload hash, hash-chain, entry-array, - header-counter, seqnum, monotonic, and TAG/FSS corruption classes. -- For ordinary traversal or compressed-reader changes, build both conformance - adapters and run `tests/interoperability/run_reader_hardening.py` against an - explicitly selected file-backed stock oracle. Use a new `.local/` output - directory. The paired corpus covers cycles, missing required posting links, - acyclic compatibility controls, facade/directory callers, complete Zstandard - streams and whole-file staging. `--only chains --only streams` does not - validate public crash recovery. Strict verification is a separate gate; - ordinary stock corruption handling can differ from the SDK's completeness - errors, so record the actual expected contract for each fixture. -- For recovery changes, test fresh exact queries as well as queries after a - full unfiltered drain. Earlier traversal can hide a missing-pointer bug by - activating recovery first. Include sparse postings, same-size unpublished - tails, partial arrays, forward/reverse/time positioning, directory refresh, - and strict capture/append rejection. Release DATA guards before cold recovery - reads. Do not replay controlled Explorer callbacks or sampling state to retry - a failed indexed operation. -- Reader recovery must not mutate source journals. Separate static damaged-file - extraction, actual writer publication, and synthetic fault injection in test - evidence. Restoring previously committed bytes is not evidence of journald - publication or a required repair lifecycle. Before adding state or repeated - scans for a restore/retry scenario, establish its supported upstream trigger. -- Recovery EOF controls must separate repeated same-direction/count polls from - reversal, explicit seeks and actual append count changes. Check bounded read - work on an unchanged damaged file and verify source bytes remain unchanged. - Cover raw as well as filtered readers; Go explicit Refresh is a separate path. -- Count-bounded ENTRY-array traversal must reject the first required revisit; - a lazy unbounded-cycle guard alone can miss it before the count completes. - Cover short cycles, descending-prefix/ascending revisits, public Rust Node - iteration and writer tail selection before mutation. Preserve acyclic - backward links and ignore reader links beyond the declared population. - Prefix checks may inspect only already visited arrays, not future headers. -- Requalification must use one stable observed recovery view across every - Boolean branch. Test a failure after an earlier branch was clipped; do not - install a partially qualified expression. Synthetic ENTRY/DATA restoration - can exercise rollback but does not establish a production repair contract. - Pending builders and installed expressions - must retain retry state and known DATA identities, including empty lists. - A failed backward preparation must retain captured populations rather than - reread DATA counts after a later append. Check public expression composition - where exposed. Clearing matches must rewind file/directory/facade readers - without an explicit seek; test this after multiple sparse matches. -- Filter position controls must cover installing at a resolved row, reversal - after EOF and early AND/OR exhaustion, duplicate realtime bounds, and failed - selected ENTRY loads followed by getters and transient I/O retry. Test both SDK and - facade APIs, regular/compact layouts and Go access backends. Directory - controls must include sequential file transitions, partial merge prefetch, - initial unset errors and pending realtime seeks; preserve candidate order - through complete retry/EOF traversal. Explicit filtered realtime search checks - visited posting slots; separately prove ordinary traversal still encounters - damaged skipped slots. Use probe counts plus isolated release comparisons to - establish search work and elapsed cost respectively. - Test recovery first discovered while sequential traversal crosses files, - followed by a later merge-probe error. Internal merge restarts - must reuse the inner routine; one public movement owns rollback of its - selected child without discarding other prefetched candidates. - Install and replace filters at a committed directory row, including group - operators. Preserve pending seeks after an initial failed load. Prove native - sequential discovery with an unvisited later-file error: nearer rows must - remain readable before that file is reached. Raw/filtered mode changes must - reuse proven sequential positions as well as filter replacements. - Audit strict index consumers for suppressed new traversal errors and cached - partial results. Exercise requested facet/source posting chains, positive - counts with missing pointers and actual cache omission with a healthy sibling; - retain documented value tolerance and propagate required traversal failures. -- Exercise recovery through public default constructors, not only a private - enablement switch. Cover file/directory/facade and whole-file compressed input, - normal EOF plus observed status, unchanged source bytes and declared counters, - and strict verification/capture/append rejection. Test live publication with - actual writers separately from synthetic restoration. A zero status is not a - full-index integrity result. Preserve errors for unknown secondary posting - tails and malformed compressed streams. - Run `TestRecoveryEqualCountFacetWithoutPostings` and measure - `BenchmarkRecoveryUnmatchedFacet` when changing Go facet shortcuts: incomplete - unused array capacity can leave visible and declared counts equal while - suppressing postings. Keep the large orphan value out of unmatched key copies; - record allocation evidence rather than inferring it from correct results. -- Before reader performance acceptance, freeze matching helper sources against - baseline and candidate SDKs. Keep immutable fixtures and complete semantic - counters. `tests/benchmarks/run_reader_regression_comparison.py` alternates - order and retains per-process output, measured runs and comparisons even when - a later case fails. An unsuccessful baseline is not a valid timing reference. -- Select representative performance controls from the actual changed work. - Careful code review and high-level tests can suffice for unchanged hot work - or cold-only checks. Use targeted timing/profiling for credible added costs - and observed regressions; do not mechanically expand every API, layout and - backend combination or build a large framework for small changes. Retain - adverse evidence, and distinguish uncertainty from established equality. -- Separate reader support for existing historical files from writer feature milestones. -- Record excluded upstream tests with a reason and extract file-level behavior where practical. - -## Bad Practices - -- Do not implement daemon/service behavior just to satisfy journalctl daemon-control options. -- Do not rely on one language's tests as sufficient for all languages. -- Do not claim compatibility from closed-file `journalctl --verify` alone. -- Do not treat live-reader smoke tests as sufficient for production compatibility. -- Do not load or link system journal libraries at runtime in any SDK implementation to pass tests. -- Do not silently skip corrupted fixture behavior; record expected errors and recovery behavior. - -## Workflow Checklist - -1. Confirm the active SOW names the exact compatibility surface being changed. -2. Identify relevant systemd tests, fixtures, and Netdata Rust source paths. -3. Add or update shared tests before accepting implementation as complete. -4. Add or update live stock-reader and cross-language concurrency tests before accepting writer/reader compatibility. -5. Run the same conformance suite across every affected language. -6. Run interoperability tests across every writer/reader pair affected by the SOW. -7. Run live one-writer/multiple-reader tests for every affected writer and reader. -8. Record benchmark or profiling evidence when the SOW includes performance claims. - -## Vacuum Ordering Validation - -- Exercise valid cross-history clock rollback, reversed creation order, ties - and unsequenced archives. Native sorting can have contradictory comparisons; - the approved deterministic history-head merge is an explicit parity exception. - Compare ordinary compatible histories with the pinned oracle and ambiguous - histories with the documented policy, including actual retained files. -- Retain count, size and age controls and single/many-history cost evidence. - Do not catch Rust sort panics or blindly substitute another sorting algorithm. - -## Validation Checklist - -Before claiming production-grade compatibility: - -- Shared conformance tests pass for every targeted language. -- Cross-language writer/reader matrix passes for every targeted file variant. -- journalctl behavior is tested against file-backed fixtures. -- Stock `journalctl --file` reads each targeted writer's files while the writer is appending. -- Stock libsystemd reader APIs read each targeted writer's files while the writer is appending. -- Each targeted reader reads live files produced by each targeted writer. -- Reader follow/tail behavior is compared with stock `journalctl` file-backed behavior. -- Daemon-only journalctl commands are not implemented and have documented behavior. -- Dependency audit confirms no CGO or system journal library linkage in the SDK runtime path unless a SOW records an explicit user policy change. - -## Evidence - -- `AGENTS.md`: project goals and scope decisions. -- `.agents/sow/specs/product-scope.md`: product scope and compatibility contracts. -- `.agents/sow/done/SOW-0001-20260523-project-bootstrap-and-orchestration.md`: initial decisions and evidence ledger. - -## Update Rules - -Update this skill when: - -- compatibility baseline changes; -- a new journal file feature becomes in scope; -- reviewer findings expose a missed compatibility or validation requirement; -- a phase adds a durable implementation workflow that future agents must repeat. +Files written by any SDK must be readable by systemd and by every other SDK, +and the SDKs must read what systemd writes, including while a writer is +appending. This skill names what must be protected and which tests prove it. +Project-wide rules (targets, runtime purity, performance contract, Go/Rust +parity, CLI exceptions) live in `AGENTS.md`; they are not repeated here. + +Detailed per-surface controls from past findings are in +[references/validation-recipes.md](references/validation-recipes.md). Read the +section for the surface you are changing; it is not a checklist for every change. + +## Authority + +- systemd source and tests are the authority when docs and implementation + disagree: SDK format behavior targets `v260.1`, the journalctl rewrites + target `v262` (see `AGENTS.md`). +- journalctl parity claims need the pinned full-feature `v262` oracle passed + with `--journalctl PATH` to every helper. Host builds differ in version and + features, so they cannot certify parity. +- When porting systemd tests, record each excluded upstream test with its + reason, and extract its file-level behavior where practical. +- Never probe the workstation's live journal: no `systemd-cat`, `logger`, + bare `journalctl`, or writes under `/var/log/journal` or `/run/log/journal`. + Use `--file` or repository-local `--directory` fixtures only. Live probing + mutates or depends on host state and makes results unreproducible. + +## Format invariants + +Each of these was a real compatibility requirement; breaking one produces +files systemd rejects or reads differently. + +- Jenkins lookup3 must match systemd `jenkins_hashlittle2()`, including + `0xdeadbeefdeadbeef` for the empty payload. +- Writers sort ENTRY DATA references by DATA offset and remove duplicates by + default, as systemd does. A trusted-unique option may skip only duplicate + removal, never sorting. +- Writers expose a raw `KEY=value` payload layer and a structured binary-safe + `{name, value}` layer (the hot path), with the same three field-name + policies: `JOURNALD` (default, keeps trusted fields such as `_HOSTNAME`), + `JOURNAL-APP` (application-facing journald restrictions, drops invalid + fields) and `RAW` (anything DATA can represent: non-empty, no `=`). The SDK + never remaps producer-specific names; consumers do that before calling. +- Readers use the on-disk `header_size`, never the in-memory v260 struct size, + and never expose bytes past it as newer fields. +- Historical unkeyed files (no `HEADER_INCOMPATIBLE_KEYED_HASH`) must open with + the unkeyed hash path and their compression flags. Writers may refuse to + append to them, but must refuse cleanly before any mutation. +- Directory writers replace an unsupported active file the way systemd does: + keep sequence identity from readable header metadata, move the old file to a + collision-safe `*.journal~` name, and create a fresh active file. Never + archive it into readable history. +- Regular uncompressed deterministic output is byte-identical to the systemd + `v260.1` reference ingester for the accepted corpus, across online, offline + and archived final states. +- Stable-file checks (verification, append-open, excluded-writer snapshots) + validate declared arenas and header counters; live readers must not, because + a writer may publish capacity before allocating it. +- Changes to `journal-engine` index semantics, `FileIndexKey`, serialized + index contents or cache partitioning must decide whether `CACHE_VERSION` + changes, and record the reindex impact or why old entries stay safe. +- Writer feature target: compression and Forward Secure Sealing. Reader + support for historical files is separate from writer milestones. + +## Live concurrency + +The contract is one writer plus many readers on the same file. A writer is +production-compatible only if stock `journalctl --file` and stock libsystemd +read it safely while it appends; a reader only if it safely reads files every +repository writer (and, where testable, systemd) is appending. Closed-file +`--verify` and smoke tests do not establish this. + +- Writer tests use a monotonically increasing sequence field (default + `LIVE_SEQ`) so stock readers prove complete ordered visibility; the harness + is under `tests/conformance/live/`. +- Stock-reader adapters may retry transient `ENODATA` or partial snapshots + only while the writer is active; after it exits, final ordered reads and + `journalctl --verify --file` must pass. +- `live_publish_every_entries` defaults to `1` (stock-follow compatible). + Other values are not stock-follow compatible unless the live matrix covers + that exact mode. + +## Recovery + +Reader recovery never mutates source journals. Keep static damaged-file +extraction, real writer publication and synthetic fault injection separate in +evidence: restoring bytes by hand does not prove what journald does. Exercise +recovery through public default constructors and with fresh queries, because +an earlier full drain can activate recovery and hide a missing-pointer bug. +Record expected errors for corrupted fixtures; never skip them silently. + +## Performance evidence + +The performance contract is in `AGENTS.md`. When timing a change, run the same +frozen helper sources against baseline and candidate, keep allocation separate +from elapsed time, and remember sampled cumulative allocation is not peak +memory. A failing or crashing baseline cannot certify a speedup. + +## Test matrices + +Choose the matrices that cover what changed, and run shared tests against both +SDKs; one language's tests never stand in for the other's. + +| Change | Run | +|---|---| +| Live writer/reader behavior | `tests/interoperability/run_live_matrix.py` (regular, zstd/xz/lz4, compact, compact+compression, sealed; stock journalctl and libsystemd, all SDK readers, final verify) | +| Arena growth / publication window | `tests/interoperability/run_live_growth.py` (supplements the live matrix) | +| Compressed DATA writing | `tests/interoperability/run_compression_matrix.py` | +| Compact writing | `tests/interoperability/run_compact_matrix.py` | +| Deterministic layout | `tests/interoperability/run_byte_identity.py --final-state all` (requires `all_equal: true`, including hash-chain depth) | +| Optional writer-lock helper | `tests/interoperability/run_lock_matrix.py` | +| Directory reading / `--directory` | `tests/interoperability/run_directory_matrix.py`, `run_mixed_directory_matrix.py` | +| journalctl query / follow | `tests/interoperability/run_journalctl_query_matrix.py` | +| journalctl options / parser | `tests/parser-parity/check_v260_manifest.py`, `run_parser_parity.py` | +| Verifier | `tests/interoperability/run_verify_matrix.py` | +| Traversal, compressed readers, recovery | `tests/interoperability/run_reader_hardening.py` with both adapters built and an explicit stock oracle | +| Reader performance | `tests/benchmarks/run_reader_regression_comparison.py` | + +Commands with native dependencies (the CLIs) need native OS CI jobs; a +CGO-disabled SDK cross-build says nothing about CLI portability. + +## Done + +A compatibility change is done when the matrices covering it pass for every +affected language and file variant, cross-language files round-trip, stock +readers read live files from each affected writer, and the SOW records the +stock systemd version, commands, durations, entry and reader counts, and +failure criteria used. diff --git a/.agents/skills/project-journal-compatibility/references/validation-recipes.md b/.agents/skills/project-journal-compatibility/references/validation-recipes.md new file mode 100644 index 00000000..2ef8907b --- /dev/null +++ b/.agents/skills/project-journal-compatibility/references/validation-recipes.md @@ -0,0 +1,354 @@ +# Journal Compatibility Validation Recipes + +Detailed controls accumulated from past SOW findings. They are not a checklist +for every change: read only the section matching the surface you are changing, +and use it to choose tests that cover the actual risk. Each control exists +because a past change got that case wrong. + +## journalctl CLI parity controls + +Use when changing the Rust or Go journalctl commands. + +- The portable file-backed journalctl contract targets tag `v262` + (8cc40e0c5e9234bf45084751ac53b1fbfe70b492). Run the pinned full-feature + oracle; host versions and feature-limited builds cannot certify this target. + Query-matrix qualification requires `--journalctl PATH`; every helper must use + that selected oracle, including version reporting and empty-result cases. +- Native PCRE2-8 >=10.42 is CLI-private. Enable CGO only for Go CLI builds + and tests; retain CGO-disabled SDK checks. Declare the C compiler, pkg-config + and PCRE2 development prerequisites in command-inclusive coverage/CI. +- File-backed CLIs preserve native Unix path bytes through selection, facade + opening and discovery. File patterns use platform globbing with a temporary + thread-local environment locale; literal paths bypass expansion. Windows + uses Unicode paths and brace/wildcard selection. Keep these native calls in + CLI modules and retain pure SDK checks. Test replacement-named decoys and + locale-sensitive patterns against the pinned oracle, with portable Unicode + controls in native OS CI. +- Color parity includes policy/environment selection, priority and audit + styles, PCRE2 first-match offsets, multiline preparation, verbose fields, + JSON primitives, separators and table headers. Compare ANSI bytes as well + as visible content; normalize JSON key order without discarding its styles. + Retain buffered match offsets without a second regex evaluation and keep + color-only work off the plain-output path. +- Parser recognition is insufficient. Compare encounter order and terminal + actions, raw required values, empty-versus-absent options, final-action scope, + timestamps/DST/TZif state, binary MESSAGE behavior, status and complete output. + Check calendar rendering independently from bound parsing: POSIX TZ, custom + TZDIR, leap records, transition-free footers and malformed-rule fallback must + agree in entry output, headers and tables. Omitted DST dates must exercise + the configured posixrules file, absent-file defaults and transition flags; + provider defaults alone do not establish native parity. Preserve raw epoch widths and + lazily resolve rendering zones once, without per-row file reads or zone allocation. + Do not let intentional follow termination hide a child failure or a blocked + stdout pipe. Retain native exit status and drain large backlogs to files. +- Use native OS jobs for commands with native dependencies. A CGO-disabled + cross-build of the SDK does not establish native CLI portability. +- Audit complete CLI selector components after a verified mismatch. Unit show + groups and invocation-discovery groups differ; test naming/glob boundaries, + raw field bytes, current-user conversion, ID precedence, global bounds, + directional limits, secondary-ID offsets and frozen follow selection. + Compare supplied oracle binaries explicitly, including live command builders. + Exercise platform fallback matchers on the primary test host. FreeBSD unit + brackets need the shared CLI compatibility matcher: native fnmatch lacks + named classes and collating elements. Validate malformed brackets as well + as positive matches, multibyte-to-byte retry, and native OS execution. Keep + ordinary patterns on native fnmatch without compatibility allocations. + Use indexed + eligibility and metadata getters; do not expand unrelated message payloads. +- For journalctl parser or option-surface changes, run + `tests/parser-parity/check_v260_manifest.py` and + `tests/parser-parity/run_parser_parity.py`. Parser parity must cover option + names, short aliases, output modes, parser-required conflicts, recognized + options interspersed after show-action match arguments, stock short-option + attached values and clusters such as `-n2`, `-ball`, and `-rn2`, and + optional-argument edge cases where bare and explicit-empty forms differ, such + as `--boot` versus `--boot=`, bare `--lines` versus `--lines=`, and bare + `--case-sensitive` versus `--case-sensitive=`. +- CLI integer controls must exercise native base prefixes/octal, leading and + trailing whitespace, signed 32-bit descriptor limits and explicit ID-offset + signs. Preserve raw Unix match operands separately from raw grep patterns. +- Short-output controls must include multiline and distinct-suffix messages, + width/line/byte thresholds and repeated MESSAGE values; grep/cat select the + first value while short output selects the last. Prove bounded reads stop + before further matching, boot observation precedes grep, and follow-tail + selection/resume state includes nonmatching native rows. +- Check short MESSAGE preparation before printability/truncation: tabs, + completed and malformed CSI/OSC sequences, CR boundaries and show-all raw + controls. JSON duplicate controls must include a binary first occurrence; + a byte array must not become the repeated-values collection. +- Cursor controls must compare final native position as well as selected + output: grep misses, bound crossings, zero counts and after-terminal seeks. + Verify follow cursor-file contents after a skipped suffix or no matches, + rather than accepting file existence alone. Size parsing needs compound + suffix, fraction, unit-order/case and overflow controls, including terminal + help encounter order. Preserve raw source-option presence on Unix. + +## Live growth and publication cadence + +Use when changing writer publication, arena growth or live readers. + +- Distinguish live mapping/access bounds from stable-file integrity checks. A + writer may publish rounded arena capacity before physical allocation reaches + it. Ordinary readers must still read committed entries safely; excluded-writer + snapshots, strict verification and append-open retain declared-arena checks. + Exercise the actual DATA-preparation window and physical growth boundary, not + only completed appends. `tests/interoperability/run_live_growth.py` supplies + deterministic regular/compact SDK writer-reader coverage; it supplements the + stock/live feature matrix rather than replacing it. +- Writers expose a cross-language `live_publish_every_entries` publication + cadence. The default `1` is the stock live-reader compatibility mode. `0` + disables explicit SDK live publication, and `N > 1` publishes after every + `N` appended entries for latency-tolerant consumers. Non-default modes must + not be described as stock `journalctl --follow` compatible unless their exact + mode is covered by live matrix evidence; they still must pass clean-close + verify and final reads. +- The reusable live-concurrency harness is under `tests/conformance/live/`. Writer tests should use the configured monotonically increasing sequence field, default `LIVE_SEQ`, so stock readers prove complete ordered visibility. +- Stock reader harness adapters may retry transient active-writer `ENODATA` open/read failures or partial snapshots only while the writer is active. After the writer exits, final ordered reads and `journalctl --verify --file` must pass. + +## Root retention and fixed-header validation + +Use when changing directory writer retention, rotation, root inventory or header validation. + +- High-level directory writers must apply configured retention once when an active writer is opened or created. Existing-active reopen and eager open enforce during construction; lazy archived-only construction remains side-effect-free until the first append opens the active file, then retention runs before the first entry is written. Active/current files must remain protected and normal retention deletion lifecycle events must be reused. +- Go's explicit `LogConfig.RootRetention` and Rust's + `Config::with_root_retention(true)` are exceptions to the default + machine-local lifecycle rules above: archived-only startup also maintains + the owned root, file lengths and tail times govern deletion, and live files + may be finalized for idle expiry or allocation-policy changes. Recovered + active files require caller verification/exclusion. Use header-native + `InspectRootRetention` before open and `Log.InspectRootRetention` for live + query/status inventory (Rust exposes equivalent snake_case APIs); do not + substitute Reader entry-vector expansion. + Live inventory must reject missing/replaced active files instead of reporting + false-empty success, and maintenance must validate it before mutation. + Root close, disposal and rotation MUST share live-path/header ownership + preflight before mutation or ownership detachment. Archive and lazy-creation + destinations MUST be absent. Reject safely, preserve bytes and permit healthy + rotation/creation retry after path restoration; rejected close releases only. + Test changes completed between SDK calls, with caller exclusion during calls, + and cover empty actives after policy setters as well as lazy successors. + Preserve separate safe-maintenance errors and fatal archive-mutation errors: + failed retired-file open/validation before mutation must not poison a healthy + current writer. Detach outgoing ownership before archive mutations so even + a post-rename failure leaves evidence inspectable. New root files, successors + and allocation checks must share configured options rather than inheriting + stale recovered layout. Exercise regular/compact transitions through both + append APIs, pre/post-mutation failures, and archive-before-delete events. + Fixed-header inventory MUST validate every header-addressed object extent + against the declared arena and allocated object tail, including ENTRY-array + and tail-entry hints. Reuse stable-header validation with append recovery and + excluded-writer snapshots; corruption in array contents remains outside the + header-only inventory contract. Exercise malformed header offsets in both + languages and require failed preflight to preserve journal bytes. + Stable validation MUST also enforce fixed-header object/category population + bounds and cached offset/count agreement, using complete on-disk field gates + for historical headers. The aggregate of present disjoint object categories + MUST fit within the total, including one object for each declared DATA/FIELD + hash table; those two object types have no dedicated count fields. Use checked + addition or a remaining-object budget. Test exact and insufficient budgets + with zero, one and two table declarations, including historical headers. + A present ENTRY_ARRAY counter MUST cover the distinct nonzero first/tail + array locations declared by complete header fields. Equal pointers count once; + additional arrays are permitted. Test historical counters before cached-tail + fields exist and preserve bytes on inventory/startup rejection. + Test small in-bounds contradictions, not only overflowing offsets. Do not impose stable counter consistency on live mapping validation. + +## Byte identity and historical files + +Use when changing deterministic writer layout, header parsing or historical-file support. + +- For deterministic regular uncompressed writer output, the layout target is byte-for-byte identity with the systemd v260.1 reference ingester for the accepted corpus. Writers must match systemd object order, alignment, initial allocation envelope, v260 header fields, entry-array growth, tail metadata, and hash-chain header behavior for that slice. +- Deterministic byte-identity validation must cover systemd final-state variants: online/plain close, offline close, and archived close. +- Header readers must use the on-disk `header_size` when validating object and hash-table locations. Do not compare historical file offsets against the current in-memory v260 `JournalHeader` struct size, and do not expose bytes beyond the on-disk header as newer header fields. +- Readers must not require `HEADER_INCOMPATIBLE_KEYED_HASH` for opening existing historical files. If that flag is absent, readers must use the unkeyed Jenkins lookup3 hash path and still support applicable compression flags such as LZ4. Writers may remain conservative when appending and reject unkeyed historical files unless a SOW explicitly proves safe mutation of that historical format. +- Writer append-open rejection for historical unkeyed files must be controlled and must occur before entry mutation. Do not leave assertion panics or partial state changes as the user-facing failure mode. +- High-level directory writers own reliable active replacement. If a low-level + append-open rejects an existing active file as unsupported, the directory + writer must follow systemd's reliable-open shape: use readable header + metadata to continue sequence identity where possible, move the old active + file out of the normal `.journal` set with a collision-safe `*.journal~` + disposed name, and create a fresh active file. Do not archive unsupported + active files into normal readable history. + +## Indexed aggregation and engine cache + +Use when changing Explorer/indexed aggregation or journal-engine index semantics. + +- Changes to `journal-engine` index semantics, `FileIndexKey`, serialized index + contents, cache schema, or consumer-visible cache partitioning must explicitly + decide whether `CACHE_VERSION` changes. If bumped, record the one-time + reindex/invalidation impact in the active SOW, product scope, and release + notes. If not bumped, record why old cache entries remain semantically safe. +- Indexed aggregation changes must force temporary DATA reuse during nested + posting and ENTRY reads. From `go/`, run + `go test -count=1 ./journal -run '^TestExplorerIndexedWindow'` for both layouts, + all DATA compression modes, + mmap/auto eviction and ReadAt scratch reuse. Keep possible mmap faults in child + processes. Compare full results and indexed counters, including empty/binary + values, singleton DATA, unset/repeated values, time filters, bucket integer + boundaries and partial read-error counters. A small fixture or + a successful zero-row query does not establish the default-window lifetime. + Benchmark matching and unmatched low/high-cardinality queries with + `BenchmarkExplorerIndexedValueOwnership`; record allocations separately from + elapsed time. Use `BenchmarkExplorerIndexedCompressedValueOwnership` with an + explicit `JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS` for prebuilt codec controls; + check indexed/traversal outputs, actual decompression and update totals before + timing. Freeze the same complete test helper for baseline and candidate, and + exclude fixture construction and reader teardown from timing. An incorrect or + crashing baseline cannot certify a speedup. + +## Go reader micro-path controls + +Use when changing Go ENTRY-slot headers, native file opening or constructor statistics. + +- Inspect CLI-helper timer boundaries before interpreting profiles or batching. + The Go reader helper opens/closes a fresh reader and checksums/sorts results + on every `--loops` iteration; its final ExplorerStats describe the last loop. + Pilot one iteration before selecting a batch count for each workload. Keep + CPU-profile post-timer activity and heap-profile forced GC separate from + query-cost claims; sampled cumulative allocation is not peak live memory. +- For Go ENTRY-slot header acquisition, run `TestEntrySlotHeader` controls on + both backends, default/tiny/non-power-of-two windows, boundary headers, + partial errors, closed readers, row pins, eviction and live array growth. + Measure `BenchmarkReaderOpen`, `BenchmarkReaderEntryHeader` and + `BenchmarkReaderAllocation` with the same complete header benchmark helper + in both builds. Record actual allocation classes with + `TestReaderAllocationShape`; raw sizeof does not establish heap cost, and + 64-bit layout preservation does not establish 32-bit layout preservation. + Exclude candidate-only direct-helper correctness tests from both timing + binaries, document that exclusion and run those tests separately. Preserve + the baseline validator's policy when isolating acquisition changes from + other paused work, and verify matching pure-build flags and source hashes. + For uncertain small-open timings, compare the same immutable five-entry + corpus using `JOURNAL_SDK_READER_OPEN_SMALL_CORPUS` and retain a generated-file + control with those same binaries. Preserve earlier results: a changed helper + can alter test linking even when production source is byte-identical. +- For Go native-file opener changes, run `TestNativeReaderFile` uncached on + amd64 and 386. Verify read-only/blocking/close-on-exec flags, typed path + errors, symlink identity, descriptor closure and sparse reads above 2 GiB. + Preserve interrupted-open retry, native malformed-file cleanup and existing + `.zst` source/staging behavior; build non-Linux delegation targets. The direct + Linux opener bypasses Go's internal test-cache input hook, so use `-count=1` + for fixture-dependent tests and retain explicit immutable fixture hashes. + Exclude candidate-only opener contract tests from both timing builds, as for + direct header tests. Primitive syscall savings do not establish full-reader + performance; repeat small, large and higher-level reader/query controls. +- For cached constructor statistics, run `TestReaderAccessorInitial` for + empty/nonempty files and initial mmap failure. Verify explicit counters, + selected/requested modes and fallback reason against a fresh statistics + query. Windows zero-byte raw files fail explicit mmap construction and Auto + falls back to ReadAt; preserve that platform behavior in test expectations. + Exclude candidate-only statistics tests from both timing builds. + +## Reader hardening, recovery and filter positioning + +Use when changing ordinary traversal, compressed readers, recovery, cursors or filters. + +- For ordinary traversal or compressed-reader changes, build both conformance + adapters and run `tests/interoperability/run_reader_hardening.py` against an + explicitly selected file-backed stock oracle. Use a new `.local/` output + directory. The paired corpus covers cycles, missing required posting links, + acyclic compatibility controls, facade/directory callers, complete Zstandard + streams and whole-file staging. `--only chains --only streams` does not + validate public crash recovery. Strict verification is a separate gate; + ordinary stock corruption handling can differ from the SDK's completeness + errors, so record the actual expected contract for each fixture. +- For recovery changes, test fresh exact queries as well as queries after a + full unfiltered drain. Earlier traversal can hide a missing-pointer bug by + activating recovery first. Include sparse postings, same-size unpublished + tails, partial arrays, forward/reverse/time positioning, directory refresh, + and strict capture/append rejection. Release DATA guards before cold recovery + reads. Do not replay controlled Explorer callbacks or sampling state to retry + a failed indexed operation. +- Reader recovery must not mutate source journals. Separate static damaged-file + extraction, actual writer publication, and synthetic fault injection in test + evidence. Restoring previously committed bytes is not evidence of journald + publication or a required repair lifecycle. Before adding state or repeated + scans for a restore/retry scenario, establish its supported upstream trigger. +- Recovery EOF controls must separate repeated same-direction/count polls from + reversal, explicit seeks and actual append count changes. Check bounded read + work on an unchanged damaged file and verify source bytes remain unchanged. + Cover raw as well as filtered readers; Go explicit Refresh is a separate path. +- Count-bounded ENTRY-array traversal must reject the first required revisit; + a lazy unbounded-cycle guard alone can miss it before the count completes. + Cover short cycles, descending-prefix/ascending revisits, public Rust Node + iteration and writer tail selection before mutation. Preserve acyclic + backward links and ignore reader links beyond the declared population. + Prefix checks may inspect only already visited arrays, not future headers. +- Requalification must use one stable observed recovery view across every + Boolean branch. Test a failure after an earlier branch was clipped; do not + install a partially qualified expression. Synthetic ENTRY/DATA restoration + can exercise rollback but does not establish a production repair contract. + Pending builders and installed expressions + must retain retry state and known DATA identities, including empty lists. + A failed backward preparation must retain captured populations rather than + reread DATA counts after a later append. Check public expression composition + where exposed. Clearing matches must rewind file/directory/facade readers + without an explicit seek; test this after multiple sparse matches. +- Filter position controls must cover installing at a resolved row, reversal + after EOF and early AND/OR exhaustion, duplicate realtime bounds, and failed + selected ENTRY loads followed by getters and transient I/O retry. Test both SDK and + facade APIs, regular/compact layouts and Go access backends. Directory + controls must include sequential file transitions, partial merge prefetch, + initial unset errors and pending realtime seeks; preserve candidate order + through complete retry/EOF traversal. Explicit filtered realtime search checks + visited posting slots; separately prove ordinary traversal still encounters + damaged skipped slots. Use probe counts plus isolated release comparisons to + establish search work and elapsed cost respectively. + Test recovery first discovered while sequential traversal crosses files, + followed by a later merge-probe error. Internal merge restarts + must reuse the inner routine; one public movement owns rollback of its + selected child without discarding other prefetched candidates. + Install and replace filters at a committed directory row, including group + operators. Preserve pending seeks after an initial failed load. Prove native + sequential discovery with an unvisited later-file error: nearer rows must + remain readable before that file is reached. Raw/filtered mode changes must + reuse proven sequential positions as well as filter replacements. + Audit strict index consumers for suppressed new traversal errors and cached + partial results. Exercise requested facet/source posting chains, positive + counts with missing pointers and actual cache omission with a healthy sibling; + retain documented value tolerance and propagate required traversal failures. +- Exercise recovery through public default constructors, not only a private + enablement switch. Cover file/directory/facade and whole-file compressed input, + normal EOF plus observed status, unchanged source bytes and declared counters, + and strict verification/capture/append rejection. Test live publication with + actual writers separately from synthetic restoration. A zero status is not a + full-index integrity result. Preserve errors for unknown secondary posting + tails and malformed compressed streams. + Run `TestRecoveryEqualCountFacetWithoutPostings` and measure + `BenchmarkRecoveryUnmatchedFacet` when changing Go facet shortcuts: incomplete + unused array capacity can leave visible and declared counts equal while + suppressing postings. Keep the large orphan value out of unmatched key copies; + record allocation evidence rather than inferring it from correct results. + +## Performance comparisons + +Use when making a reader or writer performance claim. + +- Relocated baseline/candidate source copies must include the tests' referenced + fixture and helper files at their original relative paths. Check those + dependencies before running the suite and verify copied bytes. Select Cargo + packages from their manifest names, not directory names or dependency aliases. +- Before reader performance acceptance, freeze matching helper sources against + baseline and candidate SDKs. Keep immutable fixtures and complete semantic + counters. `tests/benchmarks/run_reader_regression_comparison.py` alternates + order and retains per-process output, measured runs and comparisons even when + a later case fails. An unsuccessful baseline is not a valid timing reference. +- Select representative performance controls from the actual changed work. + Careful code review and high-level tests can suffice for unchanged hot work + or cold-only checks. Use targeted timing/profiling for credible added costs + and observed regressions; do not mechanically expand every API, layout and + backend combination or build a large framework for small changes. Retain + adverse evidence, and distinguish uncertainty from established equality. + +## Vacuum ordering + +Use when changing archive vacuum ordering. + +- Exercise valid cross-history clock rollback, reversed creation order, ties + and unsequenced archives. Native sorting can have contradictory comparisons; + the approved deterministic history-head merge is an explicit parity exception. + Compare ordinary compatible histories with the pinned oracle and ambiguous + histories with the documented policy, including actual retained files. +- Retain count, size and age controls and single/many-history cost evidence. + Do not catch Rust sort panics or blindly substitute another sorting algorithm. diff --git a/.agents/skills/project-release-tagging/SKILL.md b/.agents/skills/project-release-tagging/SKILL.md deleted file mode 100644 index 413f0e2f..00000000 --- a/.agents/skills/project-release-tagging/SKILL.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -name: project-release-tagging -description: "Use when creating, checking, or pushing release tags for this repository, especially Go module release tags." ---- -# Project Release Tagging - -## Purpose - -Ensure published releases are consumable by all SDK users, including Go users -whose module lives under the `go/` subdirectory. - -For the complete release process, use -[project-release](../project-release/SKILL.md). This skill supplies focused -package and tag rules; `RELEASING.md` maintains the CI and maintainer commands. - -## Scope - -Use this skill when: - -- creating a release tag; -- checking whether a release tag exists locally or remotely; -- pushing a release tag; -- preparing a release that should be consumed by `go get`. - -## Mandatory Knowledge - -- This repository has a root release tag, for example `v0.2.0`. -- The Go module path is declared in `go/go.mod` as - `github.com/netdata/systemd-journal-sdk/go`. -- Because the Go module is in the repository subdirectory `go/`, each Go - module release must also have a tag prefixed with that subdirectory, for - example `go/v0.2.0`. -- The root release tag and the Go submodule release tag must point to the same - commit unless a SOW records a deliberate split release decision. -- Rust crates.io publication uses project-prefixed package names. The public - SDK package is `systemd-journal-sdk`; lower-level packages are - `systemd-journal-sdk-common`, `systemd-journal-sdk-registry`, - `systemd-journal-sdk-core`, `systemd-journal-sdk-host`, - `systemd-journal-sdk-log-writer`, `systemd-journal-sdk-index`, and - `systemd-journal-sdk-engine`. -- Rust crates must be publish-dry-run and published in dependency order: - common, registry, core, host, log-writer, index, engine, public SDK. If a - SOW changes dependencies, update the order in that SOW and this skill. -- Go's module reference defines this rule: if a module is defined in a - repository subdirectory, the semantic version tag name is prefixed with the - module subdirectory followed by `/`. -- Once pushed, release tags must not be moved or deleted without explicit user - approval. Go module proxies and checksum databases may cache the old tag - target. - -## CI Release Workflow - -- Prefer the canonical repository's manual `Release` workflow for Rust - publication after preparation has merged. Read `RELEASING.md` for setup and - execution. The workflow file is `.github/workflows/release.yml`; helper and - recovery tests live under `tests/release/`. -- The user explicitly starts a run on master with version `X.Y.Z` and the full - merged release source SHA. Workflow code and release source are separate - checkouts; the source may precede the workflow's merge commit. -- Configure the GitHub `release` environment for exactly the master branch, - and configure all eight crates for Trusted Publishing with owner `netdata`, - repository `systemd-journal-sdk`, workflow `release.yml`, environment `release`. -- CI checks both language suites at source-declared compiler minimums before - obtaining publishing permission. Each new crate is dry-run, then receives - fresh temporary permission and is uploaded in the established order. -- API/index availability, non-yanked status, archive checksum and clean source - SHA must agree before a published version can be skipped. Preserve the - original version/source inputs when resuming a partial release. -- All eight Rust crates and an exact-version registry consumer pass before - a maintainer pushes paired annotated tags using existing GitHub access. - Copy the exact source from the successful CI summary, create missing tags - there, and push the new pair atomically to the canonical repository. - Correct existing siblings are preserved; conflicting tags stop the release. -- Any maintainer with repository write access can start the CI workflow. - CI uses crates.io Trusted Publishing; maintainers with tag-push permission - own GitHub tag creation. No additional GitHub App or account token is - configured for this workflow. -- After the tag push, run the guide's `verify-go` helper command to verify - both annotated targets, matching Rust archive sources and a fresh - exact-version Go consumer. It reads the - selected source's Go minimum, so verification supports historical source - while the working branch has advanced. The helper does not change tags. -- Actual version publication belongs to its execution SOW. Completing the - automation implementation SOW does not mean a version has been published. -- A new crate that does not already exist needs an explicitly authorized - initial publication before Trusted Publishing can be configured for it. - -## Manual Fallback Workflow - -1. Confirm the intended Rust crate and Go module versions. They are normally - the same, for example `0.2.0`, but may differ when a SOW records a - Rust-only or Go-only release. -2. Update and verify every active consumer install example for the intended - version. Scan the active documentation and specification roots - dynamically; do not limit the scan to a fixed file list or only the - immediately previous version: - - ```bash - grep -RInE \ - 'systemd-journal-sdk.*version = "[0-9]+\.[0-9]+\.[0-9]+"|systemd-journal-sdk/go@v[0-9]+\.[0-9]+\.[0-9]+' \ - --include='*.md' --include='*.toml' \ - README.md rust/README.md docs .agents/sow/specs - ``` - - Verify every returned Rust example identifies the intended Rust crate - version and every Go example identifies the intended Go module version. -3. Verify the worktree is clean: - - ```bash - git status --short --branch - ``` - -4. Verify the Go module path: - - ```bash - sed -n '1,20p' go/go.mod - ``` - -5. Check local and remote tags before creating anything: - - ```bash - git tag -l 'v0.2.0' 'go/v0.2.0' - git ls-remote --tags origin refs/tags/v0.2.0 refs/tags/v0.2.0^{} refs/tags/go/v0.2.0 refs/tags/go/v0.2.0^{} - ``` - -6. If either tag exists at a different commit, stop and ask the user. Do not - move, delete, force-push, or recreate tags without explicit approval. -7. Create annotated tags on the intended commit: - - ```bash - git tag -a v0.2.0 -m 'v0.2.0' - git tag -a go/v0.2.0 -m 'go/v0.2.0' - ``` - -8. Push the branch first, then push both tags: - - ```bash - git push origin - git push origin v0.2.0 go/v0.2.0 - ``` - -9. Verify remote tag targets: - - ```bash - git ls-remote --tags origin refs/tags/v0.2.0 refs/tags/v0.2.0^{} refs/tags/go/v0.2.0 refs/tags/go/v0.2.0^{} - ``` - -10. Report the peeled tag commit hashes to the user. - -## Rust crates.io Workflow - -Before publishing Rust crates: - -1. Verify `rust/Cargo.toml` workspace package version matches the intended - release. -2. Check public API compatibility since the previous release. Public Rust - struct field additions are source-breaking for downstream exhaustive struct - literals unless the struct is already `#[non_exhaustive]`; public method - additions are normally additive. Record the semver decision in the release - SOW before tagging. -3. Verify publishable internal dependencies include both `package = ...` and - `version = ...` so Cargo can replace path dependencies with registry - dependencies. -4. Run `cargo publish --dry-run` for each publishable package in dependency - order: - - ```bash - cargo publish --manifest-path rust/src/crates/journal-common/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-registry/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-core/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-host/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-log-writer/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-index/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/crates/journal-engine/Cargo.toml --dry-run - cargo publish --manifest-path rust/src/journal/Cargo.toml --dry-run - ``` - -5. Cargo verifies publishable path dependencies against crates.io. For a new - release version, dependent crate dry-runs may fail until the previous crate - in dependency order has been published and indexed. In that case, dry-run - and publish one crate at a time in dependency order: dry-run `common`, - publish `common`, then dry-run `registry`, publish `registry`, and continue. -6. Publish in the same dependency order only after the package's dry-run passes - and the SOW review gate is satisfied. -7. Never record crates.io tokens or credential details in durable artifacts. - -## Validation Checklist - -- Root tag exists locally and remotely when the release includes a root tag. -- Go submodule tag exists locally and remotely when the release includes a Go - module tag. -- Both peeled tag targets are the same commit when both tags are included. -- The branch containing that commit is pushed. -- Every active consumer install example identifies the intended Rust crate or - Go module version. -- `git status --short --branch` is clean after release work. -- Rust crates.io dry-runs pass for all publishable Rust packages when Rust - packages are part of the release. -- Rust package publication is recorded in the SOW with package names and - versions, without credential details. - -## Evidence - -- `go/go.mod`: Go module path. -- Go Modules Reference, "Mapping versions to commits": - `https://go.dev/ref/mod#vcs-version`. diff --git a/.agents/skills/project-release/SKILL.md b/.agents/skills/project-release/SKILL.md index 15c442ba..a57d8746 100644 --- a/.agents/skills/project-release/SKILL.md +++ b/.agents/skills/project-release/SKILL.md @@ -1,6 +1,6 @@ --- name: project-release -description: "Prepare, publish, verify, or resume a Rust/Go SDK release using this repository's CI and maintainer tag handoff." +description: "Use when preparing, publishing, tagging, verifying, or resuming a Rust/Go SDK release, including crates.io publication and go/ module tags." --- # Project Release @@ -9,11 +9,8 @@ and Go module tags at the same selected source commit. A paired release is complete after both languages can consume that exact version. Read [RELEASING.md](../../../RELEASING.md) for the maintained setup, commands -and recovery procedure. Use -[project-release-tagging](../project-release-tagging/SKILL.md) for focused -package, semver and tag rules, and -[project-agent-orchestration](../project-agent-orchestration/SKILL.md) for SOW -and review gates. The release contract is in +and recovery procedure, and `AGENTS.md` for SOW and review gates. The release +contract is in [product-scope.md](../../sow/specs/product-scope.md#paired-ci-releases). Follow the user's actual request and existing authorization. Preparing a PR @@ -21,6 +18,30 @@ or editing instructions does not authorize publication or tag pushes. An explicit release request can authorize those steps; do not ask again for actions already authorized. Loading this skill does not start a release. +Releases are the place where exact procedure is the point: published crates +and pushed tags cannot be taken back, so follow `RELEASING.md` exactly and stop +for the user on anything it does not cover. + +## Release invariants + +- The Go module lives in `go/` (`go/go.mod`: + `github.com/netdata/systemd-journal-sdk/go`), so Go only resolves a version + from a `go/vX.Y.Z` tag ([Go modules reference](https://go.dev/ref/mod#vcs-version)). + Every paired release has annotated `vX.Y.Z` and `go/vX.Y.Z` tags peeling to + the same commit, unless the SOW records a deliberate split release. +- Never move, delete or force-push a pushed release tag. Go proxies and the + checksum database cache the first target, so a moved tag breaks consumers. +- Rust packages are `systemd-journal-sdk` plus the `systemd-journal-sdk-*` + internal crates. Publishable internal dependencies need both `package =` and + `version =` so Cargo can replace path dependencies with registry ones. +- Semver: adding a field to a public Rust struct, or a variant to a public + enum, breaks downstream exhaustive literals and matches unless the type is + `#[non_exhaustive]`. Record the semver decision in the preparation SOW. +- Before tagging, scan `README.md`, `rust/README.md`, `docs/` and + `.agents/sow/specs/` for Rust `version = "…"` and Go `@v…` install examples; + every active one must name the new version. +- Never record crates.io tokens or credential details in durable artifacts. + ## Prepare and merge 1. Check the active/pending SOWs and the previous published APIs in **both** diff --git a/.agents/sow/SOW.template.md b/.agents/sow/SOW.template.md index 0355a08a..8cd2db6d 100644 --- a/.agents/sow/SOW.template.md +++ b/.agents/sow/SOW.template.md @@ -124,10 +124,10 @@ Implementation: Reviewers: -- [External-review authorization or waiver status. When authorized, use the - system-wide `external-reviewers` skill; do not copy model or harness details.] +- [Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used.] -Repository boundary block for every external-reviewer prompt: +Repository boundary block for every external-agent prompt: ```text CRITICAL REPOSITORY BOUNDARY: diff --git a/AGENTS.md b/AGENTS.md index 3e7be45d..53ac849e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,188 +3,130 @@ ## Glossary - CGO: Go's C interoperability mechanism. -- DATA: systemd journal file object type for a stored field/value payload. -- ENTRY: systemd journal file object type for a journal row. -- FIELD: systemd journal file object type for a field name. -- SOW: Statement of Work. +- DATA, ENTRY, FIELD: systemd journal file object types for a field/value payload, a journal row, and a field name. - FTS: full-text search. - IP: Internet Protocol. - SNMP: Simple Network Management Protocol. -- THIS: uppercase English pronoun used in repository-boundary prompt emphasis. -- DO: uppercase English verb used in repository-boundary prompt emphasis. -- FOR: uppercase English preposition used in repository-boundary prompt emphasis. -- MAKE: uppercase English verb used in repository-boundary prompt emphasis. -- WASM: WebAssembly. -- DO NOT: uppercase emphasis in agent prompt blocks. +- SOW: Statement of Work. ## Goals -This project produces pure SDKs for reading and writing systemd journal files in Rust and Go. +Pure Rust and Go SDKs for reading and writing systemd journal files, plus +file-backed journalctl rewrites in both languages. Success means: -Success means: - -- SDKs read existing journal files without linking to system journal libraries. -- SDKs write valid journal files without CGO or external journal libraries. -- Journal files written by one language can be read by every other language and by compatible systemd tooling where applicable. -- The same shared conformance suite, fixtures, interoperability tests, benchmarks, and profiling workflows apply to every implementation. -- SDK APIs exercise the journal format's native performance features. -- Correct output is not enough when the format has an index, hash table, object chain, offset array, mmap, or reusable object path. -- Use those structures to avoid row scans, repeated parsing, unnecessary decompression, unnecessary allocation, or avoidable syscalls. -- journalctl rewrites exist for Rust and Go for file-backed/query behavior. -- Daemon-only journalctl operations are not implemented in this project. +- Read and write valid journal files without CGO or system journal libraries. +- Files written by either language are readable by the other and by systemd + tooling. +- One shared conformance suite, fixture set, interoperability tests and + benchmarks apply to both implementations. +- APIs use the format's native performance features (see Performance Contract). +- Daemon-only journalctl operations are out of scope. Project SOW status: initialized ## Go/Rust Feature Parity -- Accepted SDK functionality MUST be implemented in both Go and Rust as part of - the same approved feature scope. APIs SHOULD follow each language's idioms - while providing equivalent behavior, failure semantics and operational guarantees. -- Feature plans and acceptance criteria MUST identify both language surfaces and - validate each implementation, including relevant cross-language file behavior. - One consumer using only one language does not justify a parity gap. -- A language-specific exception or staged delivery requires an explicit user - decision recorded in the active SOW. A remaining parity gap MUST be resolved - with the user before publication; release preparation is not the first parity check. +Accepted SDK functionality ships in both Go and Rust in the same feature scope, +idiomatic per language but with equivalent behavior, failure semantics and +guarantees. Plans and acceptance criteria name both surfaces and validate each, +including cross-language file behavior. A consumer using only one language +does not justify a gap. A language-specific exception or staged delivery needs +an explicit user decision recorded in the SOW, and any remaining gap is +resolved with the user before publication. ## Performance Contract -This project is not only a compatibility clone. It is a high-performance -journal file-format SDK. Design every reader, writer, query helper, and facade -to use the most efficient journal-native path that preserves the documented -contract. Any deviation needs active-SOW evidence and an explicit user decision. - -Mandatory rules: - -- Prefer journal-native structures over row scans. Use FIELD/DATA hash tables, - FIELD DATA chains, DATA entry arrays, ENTRY arrays, offsets, sequence/time - metadata, and mmap-backed slices when those structures answer the request. -- Do not implement a public API by expanding every entry when the journal format - can answer it through an index or object chain, unless active-SOW evidence - proves the indexed path is unsafe or slower. For example, unfiltered unique - field values walk the FIELD object's DATA chain, matching systemd, and do not - scan all entries. Field-name enumeration walks FIELD hash tables on valid - indexed files, with entry-scan fallback only for documented compatibility - cases where the FIELD table cannot be traversed safely. -- Avoid decoding, decompressing, copying, hashing, sorting, or allocating data - outside the requested scope. Compressed DATA must stay compressed unless the - caller needs that DATA's value for filtering, faceting, FTS, or returned-row - display. -- Explorer column catalogs must come from FIELD indexes, not row traversal. - `ExplorerQuery::debug_collect_column_fields_by_row_traversal` is a - discrepancy-debug tool only. Production use, benchmark success, or - compatibility acceptance with this flag enabled is invalid evidence and must - be treated as an explorer bug. -- Treat repeated parsing of reusable DATA objects as a performance bug unless a - SOW records evidence that the alternative is slower or unsafe. -- Choose performance evidence by tracing the actual changed work. Careful code - review and representative high-level tests can establish that a change does - not negatively affect performance. Use existing benchmarks and targeted - profiles for credible added costs or observed regressions; do not build or - expand a large measurement framework to prove every small detail. An - inconclusive timing interval alone is neither proof of equality nor evidence - of a regression. Preserve adverse observations and investigate them. - Compare equivalent benchmarks against systemd when practical and against the - fastest sibling implementation; explain unavailable comparisons. -- Record any intentionally slower path in the SOW, docs, and tests, with the - reason, measured cost, and the conditions where it is acceptable. +This is a high-performance journal SDK, not only a compatibility clone. Every +reader, writer, query helper and facade uses the most efficient journal-native +path that preserves the documented contract; any deviation needs SOW evidence +and a user decision. + +- Answer requests from FIELD/DATA hash tables, FIELD DATA chains, DATA and ENTRY + arrays, offsets, sequence/time metadata and mmap slices instead of expanding + every entry, unless SOW evidence shows the indexed path is unsafe or slower. + For example, unfiltered unique values walk the FIELD's DATA chain as systemd + does, and field-name enumeration walks FIELD hash tables, falling back to + entry scans only for documented cases where the table is unsafe to traverse. +- Do not decode, decompress, copy, hash, sort or allocate outside the requested + scope. Compressed DATA stays compressed unless its value is needed for + filtering, faceting, FTS or returned rows. Repeated parsing of reusable DATA + objects is a performance bug. +- Explorer column catalogs come from FIELD indexes. + `ExplorerQuery::debug_collect_column_fields_by_row_traversal` is a debug tool; + any production use, benchmark or compatibility claim with it enabled is + invalid and is an explorer bug. +- Choose evidence by tracing the changed work: careful review and + representative tests can show no regression; use existing benchmarks and + targeted profiles for credible added costs. Do not build large measurement + frameworks for small changes. An inconclusive timing proves neither equality + nor regression; keep and investigate adverse observations. Compare against + systemd and the faster sibling implementation where practical. +- Record any intentionally slower path in the SOW, docs and tests with its + reason, measured cost and acceptable conditions. ## Runtime Purity Architecture -This project has four separate layers. Keep them separate in every language: - -1. **Core journal file-format SDK** - reads and writes journal files only. Core - readers and writers do not execute external programs, probe host identity - sources, read OS-specific identity files or registries, or enforce writer - locks by default. They operate only on caller-provided paths, bytes, options, - timestamps, machine IDs, boot IDs, seqnum IDs, and related metadata. Any - exception belongs in an explicit optional helper or needs a user-approved SOW - contract change. -2. **Systemd/journald compatibility layer** - applies journald-compatible field - naming, metadata, and API conventions. It may require caller-provided machine - and boot identity. It does not silently discover host identity. Callers that - want automatic identity discovery explicitly invoke an optional identity - helper and pass the result in. -3. **Optional identity helper service** - discovers host machine/boot identity - when a caller explicitly opts in. This helper is outside the core - file-format SDK contract. -4. **Optional writer-lock helper service** - provides cooperating-writer - exclusion when a caller explicitly opts in. This helper is independent from - systemd compatibility. The journal file format requires one writer per file - as an operational contract, but systemd does not define or enforce a portable - lock protocol in the journal file format. Core writer constructors do not - expose lock-enable options; callers acquire and release the lock helper - separately around writer use. - -Core SDK runtime code reads host identity or process state only inside explicit -optional helper implementations. Core reader and writer paths do not read from -these inputs: - -- `/proc` -- `/host/proc` -- machine identity files: `/etc/machine-id` and `/var/lib/dbus/machine-id` -- platform registries -- `sysctl` -- `system_profiler` -- `ps` -- shell commands -- subprocess APIs - -Tests, docs, and examples must use synthetic identities unless explicitly -testing an optional helper. +Keep four layers separate in both languages: + +1. **Core file-format SDK** - reads and writes journal files using only + caller-provided paths, bytes, options, timestamps and machine/boot/seqnum + IDs. It never runs programs, probes host identity or locks writers by default. +2. **Systemd/journald compatibility layer** - journald field naming, metadata + and API conventions. It may require caller-provided identity but never + discovers it silently. +3. **Optional identity helper** - discovers machine/boot identity only when a + caller opts in and passes the result on. +4. **Optional writer-lock helper** - cooperating-writer exclusion on opt-in. + One writer per file is an operational contract, but systemd defines no + portable lock protocol, so this is not systemd compatibility. Core writer + constructors expose no lock options; callers wrap writer use themselves. + +Core reader/writer paths never read `/proc`, `/host/proc`, `/etc/machine-id`, +`/var/lib/dbus/machine-id`, platform registries, `sysctl`, `system_profiler` or +`ps`, and never use shell commands or subprocess APIs; only the optional helpers +may. Tests, docs and examples use synthetic identities unless testing a helper. ## SOW System -This project uses a local Statement of Work system. - -The SOW system is self-contained in this repository. Normal SOW work depends on -this `AGENTS.md`, project-local SOW files, project-local specs, project-local -skills, and the active SOW. Use global templates or scripts only when the user -explicitly asks to repair or upgrade the SOW framework. +This project uses a self-contained local Statement of Work system: this file, +the SOW files, specs and project skills. Use global SOW templates or scripts +only when the user asks to repair or upgrade the framework. ### Roles -- **User responsibilities:** purpose, scope decisions, design forks, risk acceptance, destructive approvals, and final product judgment. -- **Project manager responsibilities:** SOW creation, phase planning, code, tests, documentation, benchmark/profiling work, implementation evidence, reviewer coordination, evidence ledgers, status reporting, and gate enforcement. -- **Reviewer agent responsibilities:** independently review technical behavior and search for regressions. -- **Reviewer agents review security:** identify vulnerabilities and unwanted side effects. -- **Reviewer agents assess readiness:** decide whether the result is production-grade. - -External review is the default, user-waivable technical review gate for -implementation SOWs. Before submitting code to GitHub, recommend external -review when the current change set has not already been reviewed. Do not run -external reviewers before the user authorizes them. Once authorized, standing -authorization applies for the current conversation within any limits set by the -user. - -The system-wide `external-reviewers` skill is the sole source of truth for -available reviewers, model selection, harnesses, prompts, authorization scope, -retries, and failure handling. Do not copy those volatile details into -repository instructions. Review the complete active SOW after local validation -as one meaningful batch; do not review trivial edits or partial churn. - -Historical note: this repository previously required implementation -delegation. That mandate is neither required nor preferred now. Historical -records may retain exact model names and execution trails, but those records -must not be assumed to describe current practice. +- **User:** purpose, scope, design forks, risk acceptance, destructive approvals and final product judgment. +- **Project manager (the agent):** SOW creation and planning, code, tests, docs, benchmarks, evidence, self-review, status reporting and gate enforcement. -### Required First Checks +### Review Gate + +The review gate is the test suite plus the project manager's own review. Do +not run or recommend external reviewers. + +- Finish the whole active SOW locally, including tests and evidence, then + review the entire SOW and changed surface as one batch, not partial churn. +- Review for behavior, regressions, Go/Rust parity, security, unwanted side + effects and production readiness; record each finding and its disposition. +- After material fixes, rerun tests and re-review the whole scope. +- A phase advances only when tests pass and blocking findings are resolved, + never on "mostly ok". + +Done SOWs may record external reviewers or delegated implementation from +earlier practice; that does not describe current practice. -Before non-trivial work: +### Required First Checks -1. Read pending/current SOWs for overlap, contradictions, and existing decisions. -2. Read relevant specs under `.agents/sow/specs/`. -3. Inspect `.agents/skills/project-*/SKILL.md` and load every runtime project skill whose trigger matches the work. -4. Inspect code/docs/data as ground truth. -5. Ask the user only for irreducible product/design/risk decisions. +Before non-trivial work, read pending/current SOWs and relevant specs for +overlap and prior decisions, load every project skill whose trigger matches, +treat code and data as ground truth, and ask the user only for irreducible +product, design or risk decisions. ### Git Worktrees -Create a git worktree only when the user explicitly asks for it or approves it. +Create a git worktree only when the user asks for or approves it. ### Repository Boundary -CRITICAL: Do not make changes outside this repository. This applies to all assistants and external reviewers. +CRITICAL: Do not make changes outside this repository. This applies to all assistants and helper agents. Canonical external-agent prompt block: @@ -198,262 +140,143 @@ CRITICAL REPOSITORY BOUNDARY: - Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. ``` -Rules: - -- Read-only inspection outside this repository is allowed when required by a SOW. -- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. -- The only write exception is `/tmp`. -- Prefer `.local/` inside this repository for scratch work, generated temporary files, cloned references, logs, and external-agent working notes. -- Include the canonical external-agent prompt block verbatim in every external-agent prompt. - Any exception needs an explicit user decision recorded in the active SOW first. +- The block binds this session as well as helper agents. Include it verbatim + in every helper-agent prompt; exceptions need a user decision in the SOW. +- Investigation and review helpers are read-only; their prompts forbid + creating, modifying, deleting, moving, formatting, staging or committing files. +- Package metadata commands (`go get`, `go list`, `cargo metadata`, + `npm view`, `pip download`, …) write caches even when they look read-only: + point `GOMODCACHE`, `GOCACHE`, `GOPATH`, `CARGO_HOME`, `npm_config_cache`, + `PIP_CACHE_DIR` and equivalents under `.local/` or `/tmp`, or forbid fetching. ### Sensitive Data In Durable Artifacts -Treat SOWs, specs, documentation, project skills, agent instructions, and code comments as commit-ready artifacts. -Treat them as public by default. -A repository-specific policy may mark a listed artifact class private. +SOWs, specs, docs, skills, agent instructions and code comments are +commit-ready and public by default. CRITICAL: Never write raw sensitive data to durable artifacts. -Secure handling exception: if raw sensitive data is required to continue, stop and ask the user for a secure handling path before writing anything. -This includes passwords, API keys, bearer tokens, SNMP communities, private keys, and connection strings with embedded credentials. -It also includes session cookies, community member names, customer names, customer identifiers, personal data, customer-identifying public IP addresses, private endpoints, account IDs, and proprietary incident details. - -Write only sanitized evidence: - -- use placeholders such as `[REDACTED_SECRET]`, `[CUSTOMER]`, `[ACCOUNT]`, `[PRIVATE_ENDPOINT]`; -- use stable aliases such as `customer-a` only when the real mapping is not stored in the repository; -- cite file paths, line numbers, command names, schema fields, or error classes instead of copying sensitive values; -- summarize logs and traces; include only minimal redacted snippets. - -If sensitive data is required to continue, stop and ask the user for a secure handling path. If sensitive data is found in a durable artifact, sanitize it before any commit. If sensitive data was already committed, tell the user and do not rewrite history without explicit approval. +This covers credentials of any kind (passwords, API keys, tokens, SNMP +communities, private keys, credentialed connection strings, session cookies) +and identifying data (community member or customer names and identifiers, +personal data, customer-identifying public IPs, private endpoints, account IDs, +proprietary incident details). + +Use placeholders such as `[REDACTED_SECRET]` or `[CUSTOMER]`, aliases whose +mapping is not stored in the repository, and citations of paths, commands or +error classes instead of values; summarize logs. If raw sensitive data is needed +to continue, stop and ask the user for a secure handling path. Sanitize any +found before commit; if it was already committed, tell the user and do not +rewrite history without approval. ### Open-Source Reference Evidence -When a SOW uses external open-source repositories as evidence, record the upstream repository identity and checked commit, not the workstation mirror path. - -For local mirrored or cloned open-source repositories, cite evidence in this form: - -```text -owner/repo @ commit -relative/path/inside/repo:line -``` - -Rules: - -- Never use workstation absolute paths for external open-source evidence in SOWs. -- Resolve `owner/repo` from the repository remote, not only from the local directory name. -- Record the commit with `git -C rev-parse --short=12 HEAD` or the full hash when precision matters. -- Use paths relative to the upstream repository root after the `owner/repo @ commit` line. -- If multiple repositories were checked, list each repository and commit separately. +Cite external open-source evidence as `owner/repo @ commit` followed by +repository-relative `path:line`, with `owner/repo` taken from the remote. Never +cite workstation mirror paths: they are meaningless to other readers and leak +local layout. ### Pre-Implementation Gate -Implementation must not begin until the active SOW contains a concrete `## Pre-Implementation Gate` section. Before moving a SOW from `pending/open` to `current/in-progress`, or before continuing implementation in an existing current SOW that lacks this section, fill the gate. - -The gate must record: - -- Problem / root-cause model. -- Evidence reviewed. -- Affected contracts and surfaces. -- Existing patterns to reuse. -- Risk and blast radius. -- Sensitive data handling plan. -- Implementation plan. -- Validation plan. -- Artifact impact plan. -- Open decisions. - -Generic placeholders such as `TBD`, `N/A`, or "to be checked later" are invalid unless the SOW explains why the item truly does not apply. If the gate exposes an unknown that cannot be resolved by investigation, stop and ask the user before implementation. +Before implementation starts, the active SOW's `## Pre-Implementation Gate` +must show that the problem is understood: the root cause with evidence, the +affected contracts and both language surfaces, the risks, the plan, how it will +be validated, and any open decisions. Scale the depth to the change; mark +headings that do not apply with a one-line reason instead of padding them. If +investigation cannot resolve an unknown, stop and ask the user. ### When A SOW Is Required -Create or reuse a SOW for non-trivial work: - -- feature work; -- bug fixes with behavioral impact; -- refactors; -- migrations; -- documentation or content changes with product/business impact; -- process changes; -- regressions; -- spec hygiene; -- project skill changes; -- any work with unclear risk. - -Trivial work does not need a SOW: - -- typo fixes; -- formatting-only changes; -- mechanical rename with no behavior change; -- simple search/replace with low risk. - -When unsure, treat the work as non-trivial. +Non-trivial work needs a SOW: features, behavioral bug fixes, refactors, +migrations, product-impacting docs, process or skill changes, spec hygiene, +regressions, and anything of unclear risk. Typos, formatting, behavior-free +renames and low-risk search/replace do not. When unsure, treat it as non-trivial. ### SOW Locations -- Pending: `.agents/sow/pending/` -- Current: `.agents/sow/current/` -- Done: `.agents/sow/done/` -- Specs: `.agents/sow/specs/` -- Template for new SOWs: `.agents/sow/SOW.template.md` -- Local audit: `.agents/sow/audit.sh` - -Create new SOW files from `.agents/sow/SOW.template.md`. The template is project-local and may be customized for this repository. - -Empty SOW directories must contain `.gitkeep` or `.keep` so the committed repository preserves the full SOW layout after clone/checkout. - -Filename: - -```text -SOW-NNNN-YYYYMMDD-{slug}.md -``` - -Status and directory must agree: - -- `open` lives in `pending/` -- `in-progress` lives in `current/` -- `paused` lives in `current/` -- `completed` lives in `done/` -- `closed` lives in `done/` +SOWs live in `.agents/sow/{pending,current,done}/` (empty directories keep a +`.gitkeep`), specs in `.agents/sow/specs/`, and are created from +`.agents/sow/SOW.template.md` as `SOW-NNNN-YYYYMMDD-{slug}.md`. The audit is +`.agents/sow/audit.sh`. Status and directory must agree: `open` in `pending/`, +`in-progress` or `paused` in `current/`, `completed` or `closed` in `done/`. ### SOW Completion And Commit -The successful terminal SOW status is `completed`. `done` is a directory name, not a status value. Never write `Status: done` or `Status: complete`. - -After each implementation chunk/SOW is implemented, reviewed, and verified, prefer committing that chunk before starting the next chunk. This preserves rollback points before subsequent work. - -If `.agents/sow/audit.sh` fails, do not close the SOW and do not advance to the next chunk. Record the audit failure in the active SOW, repair it inside this repository, rerun the audit, and record the clean result. - -When a SOW's work is ready to close: - -1. Finish implementation, docs, specs, skills, validation, and follow-up mapping. -2. Update the SOW to `Status: completed`. -3. Move the SOW file to `.agents/sow/done/`. -4. Commit the work, artifact updates, SOW status change, and SOW move together as one commit, unless the user explicitly requested a different commit split. +The successful terminal status is `completed`; `done` is only a directory +name, so never write `Status: done` or `Status: complete`. -Do not create a separate commit just to mark or move the SOW. Do not claim a SOW is completed while the implementation and the SOW lifecycle change live in separate uncommitted or separately committed states. - -Git staging rule: never use `git add -A` or `git add .`. Stage explicit paths only. +- Close a SOW only after its implementation, docs, specs, skills, validation + and follow-ups are finished and `.agents/sow/audit.sh` reports + `SOW initialization complete and clean`. The audit can exit zero while + reporting partial state, so read the verdict. If it fails, record and repair + it before closing or advancing. +- Commit the work, artifact updates, status change and move to `done/` + together in one commit unless the user asks otherwise, so a completed SOW + never lives apart from its implementation. Prefer committing each verified + chunk before the next, to keep rollback points. +- Never use `git add -A` or `git add .`; stage explicit paths only. ### One SOW At A Time -Never execute multiple SOWs as one batch. - -If work overlaps: - -- merge or consolidate before implementation; or -- split into separate SOWs and complete one before starting the next. - -Progress reports are not stop points. Once a SOW is in progress, continue until it is delivered, failed with evidence, blocked on a real user decision/approval, or superseded by newer user instructions. +Never run multiple SOWs as one batch; merge overlapping work before +implementation or split it and finish one first. Read-only helper agents may +run in parallel within the active SOW. Progress reports are not stop points: +an in-progress SOW continues until delivered, failed with evidence, blocked on +a real user decision, or superseded by the user. ### User Decisions -When user decisions are needed: - -1. Present concrete evidence with files/lines or source references. -2. Provide numbered options. -3. Explain pros, cons, implications, and risks. -4. Recommend one option with reasoning. -5. Record the user's decision in the SOW before implementation. +Present concrete evidence (files/lines or sources), numbered options with pros, +cons and risks, and a recommendation with its reasoning. Record the decision in +the SOW before implementing it. ### Followup Discipline -"Deferred" is not a terminal outcome. - -Before a SOW can close, every valid deferred item must be: - -- implemented in the current SOW; or -- explicitly rejected as not worth doing, with evidence; or -- represented by a real pending/current SOW file. - -Pre-close, search the SOW for: - -```text -defer|later|follow-up|future|TODO|pending -``` - -Map every remaining item to implemented, rejected, or tracked. +"Deferred" is not a terminal outcome. Before a SOW closes, every valid leftover +item is implemented, rejected with evidence, or tracked by a real pending SOW, +so work does not silently disappear. ### Regressions -A regression is discovered after a SOW was considered completed or closed, later testing or use finds broken behavior, and the original SOW's claimed outcome is no longer true. - -When behavior that a completed SOW claimed working stops working: - -1. Find the original SOW in `done/`. -2. Move it back to `current/`. -3. Mark it `in-progress` with a regression note in `## Status`. -4. Append a new dated `## Regression - YYYY-MM-DD` section at the end of the file, after the original outcome, lessons, and follow-up content. -5. In that appended section, record what broke, evidence, why previous validation missed it, the repair plan, validation, and updates needed to specs, skills, docs, audits, or follow-up SOWs. -6. Fix and validate there. - -Never prepend regression content above the original SOW narrative. The original requirements, analysis, plan, validation, outcome, lessons, and follow-up must remain readable first. -Do not create a new SOW for a true regression. +When behavior a completed SOW claimed working breaks later, reopen that SOW +instead of creating a new one: move it back to `current/`, mark it +`in-progress`, and append a dated `## Regression - YYYY-MM-DD` section at the +end recording what broke, why validation missed it, the fix, and its +validation. Never prepend; the original narrative stays readable first. ### Validation Gate -A SOW cannot be completed until Validation records: - -- acceptance criteria evidence; -- tests or equivalent validation; -- real-use evidence when a runnable path exists; -- reviewer findings and how they were handled; -- same-failure search results; -- sensitive data gate; -- artifact maintenance gate; -- SOW status/directory consistency; -- spec update or specific reason no spec update was needed; -- project skill update or specific reason no skill update was needed; -- end-user/operator docs update or evidence-backed reason none were affected; -- end-user/operator skill update or evidence-backed reason none were affected by docs/spec changes; -- lessons extracted or specific reason there were none; -- follow-up mapping. - -Generic "N/A" is invalid. - -Reviewer agents must be read-only. Reviewer prompts must forbid creating, modifying, deleting, moving, formatting, staging, committing, or changing files. +A SOW is complete when its Validation section lets a reader trust the outcome: +acceptance criteria met with evidence, tests run and their results, real-use +evidence where a runnable path exists, self-review findings and dispositions, +a search for the same failure elsewhere, the sensitive-data check, and what +happened to follow-ups. ### Artifact Maintenance Gate -Every SOW close must explicitly record whether each durable artifact class was updated or why no update was needed: - -- `AGENTS.md` - workflow, responsibility, local framework, project-wide guardrails. -- Runtime project skills - `.agents/skills/project-*/SKILL.md` for HOW to work here. -- Specs - `.agents/sow/specs/` for WHAT the project does. -- End-user/operator docs - README, docs site, runbooks, published guides, help text, or other human-facing documentation. -- End-user/operator skills - output/reference skills copied or consumed outside normal repo work. -- SOW lifecycle - split, merge, status, directory, deferred work, regression reopening, and follow-up mapping. -- `.agents/sow/SOW-status.md` - human-readable project status summary, updated whenever SOW state changes. +At close, state which durable artifacts changed and why the others did not +need to: `AGENTS.md`, project skills, specs, end-user docs (README, `docs/`, +help text), and `.agents/sow/SOW-status.md`, which is updated whenever SOW state +changes. Stale specs and docs are how this project accumulates false guidance, +so check them against the change rather than by habit. ### Specs -Specs are memory of WHAT this project does. - -Update specs when shipped work changes: - -- product behavior; -- public contracts; -- data formats; -- UX rules; -- business logic; -- operational guarantees; -- known edge cases. - -Specs describe current reality, not aspiration. If specs and code disagree, record the discrepancy in the active SOW and resolve or track it. +Specs in `.agents/sow/specs/` record WHAT the project does today, not +aspirations. Update them when shipped work changes behavior, public contracts, +data formats, operational guarantees or known edge cases. If specs and code +disagree, record the discrepancy in the SOW and resolve or track it. ### Project Skills -Project skills are memory of HOW to work here. - -Runtime input project skills live under `.agents/skills/project-*/SKILL.md`. The `project-` prefix is the generic hook meaning agents working in this repo must consider this skill. Before non-trivial work, inspect those skill descriptions and load every matching runtime skill. +Project skills record HOW to work here. Skills under +`.agents/skills/project-*/SKILL.md` must be considered before non-trivial work; +load each one whose trigger matches. Runtime input skills: -- `.agents/skills/project-agent-orchestration/SKILL.md` - `.agents/skills/project-docs-authoring/SKILL.md` - `.agents/skills/project-journal-compatibility/SKILL.md` - `.agents/skills/project-release/SKILL.md` -- `.agents/skills/project-release-tagging/SKILL.md` Legacy runtime skills: @@ -465,47 +288,36 @@ Output/reference skills: ### Project-specific overrides -- The baseline systemd compatibility target is `systemd/systemd` tag `v260.1`. -- The file-backed Rust and Go `journalctl` commands target `systemd/systemd` - tag `v262` (8cc40e0c5e9234bf45084751ac53b1fbfe70b492). This CLI target - does not advance the SDK journal-format target or compiler minimums. -- Native PCRE2-8 >=10.42 is allowed only in the two journalctl commands. - The Go command requires CGO and a C compiler; the Go SDK remains CGO-free. - The commands may also use the platform C library for Unix file globbing, - thread-local locale selection, and terminal queries. Windows filenames use - the Unicode path contract. These CLI exceptions do not extend to SDKs. - Keep CLI build/test steps explicitly CGO-enabled and preserve independent - pure SDK checks. System journal library linkage remains prohibited. -- Netdata's consuming-module declarations are the authority for SDK compiler - minimums. The SDK's Go and Rust minimums may match or remain below Netdata's - corresponding declared minimums, but must never exceed them. Requiring a - newer compiler would make the SDK incompatible with Netdata. -- Inspect the current Netdata source before setting or changing a compiler - minimum. Use the consuming Go module's `go` directive and the consuming Rust - workspace's `rust-version`; do not substitute a newer compiler downloaded by - packaging, selected by CI, available locally, or preferred by a scanner. -- Never propose raising an SDK compiler minimum above Netdata's declaration as - a dependency-refresh, release-hygiene, vulnerability-remediation, or - scanner-cleanup option. Select dependency versions compatible with the - Netdata floor instead. Newer toolchains may be used as additional validation - targets, but they do not change release metadata or the supported minimum. -- If a scanner reports findings against Netdata's compiler floor, investigate - reachability and validate the actual Netdata build toolchain, then record an - explicit scanner disposition. Do not clear the finding by breaking Netdata - compatibility. -- The final writer target includes compression and Forward Secure Sealing where systemd journal files define it, but implementation may be phased. -- Readers must handle applicable historical journal file variants covered by the shared conformance fixtures. -- Provide two API layers per language: idiomatic SDK API plus a libsystemd-compatible reader facade. The facade is required unless a SOW records concrete evidence that it would require native bindings, violate the pure-language policy, or create an unsafe/unrepresentable API in that language. -- journalctl repeated matches for the same field already provide OR semantics. No new `KEY in [values]` syntax is required. -- The `+` separator is a systemd journalctl disjunction feature to replicate for file-backed journalctl behavior; it is not a new extension. -- Implement journalctl rewrites for file-backed/query behavior in Rust and Go. -- Do not implement daemon-only journalctl commands, including daemon sync, flush, rotate, and relinquish-var operations. -- Common compression-library dependencies are allowed after dependency review. Journal parsing/writing must not depend on systemd/libjournal. CGO remains disallowed in SDKs; the journalctl-only PCRE2 exception above does not permit linking system journal libraries. -- Before Netdata integration or stable release work advances, GitHub code scanning and Codacy analysis findings must be fixed or explicitly dispositioned under the active SOW policy. Raw SARIF, Codacy exports, and scanner logs must stay under `.local/`; durable artifacts may contain only sanitized aggregate summaries and evidence. -- Rust and Go sources are frozen for the 2026-06-11 docs-and-parity program (SOW-0103 through SOW-0106). Problems found in Rust or Go become pending SOWs; they are not fixed inside the program SOWs. -- Current review cadence: finish the complete active SOW locally first, including local validation and SOW evidence. -- After local validation, recommend external review if it has not been authorized or explicitly waived. When authorized, follow the system-wide `external-reviewers` skill and review the entire SOW and changed surface as one batch. -- A phase cannot advance until local implementation and validation have completed and external review has either completed with findings resolved or been explicitly waived by the user and recorded in the SOW. -- After each verified chunk, prefer committing the chunk before starting the next work chunk, using explicit path staging only. -- After SOW-0003 completes, SOW-0005 (Go writer first) activates before SOW-0004, SOW-0006, SOW-0007, SOW-0008, SOW-0009, or SOW-0010. After SOW-0005 completes, continue according to the active SOW dependency chain, but only one implementation SOW may be active at a time. -- Reviewer agents may run in parallel within the active SOW; implementation SOWs must not run in parallel unless the user explicitly changes the one-SOW-at-a-time rule. +- Targets: SDK journal-format compatibility follows `systemd/systemd` tag + `v260.1`. The file-backed journalctl commands follow tag `v262` + (8cc40e0c5e9234bf45084751ac53b1fbfe70b492); that does not move the SDK format + target or compiler minimums. +- CLI-only native code: the two journalctl commands may use native PCRE2-8 + >=10.42 (the Go command therefore needs CGO and a C compiler) and the platform + C library for Unix globbing, thread-local locale and terminal queries; Windows + uses the Unicode path contract. These exceptions never extend to the SDKs: + keep CLI builds explicitly CGO-enabled and pure SDK checks separate. SDKs stay + CGO-free, and nothing links system journal libraries or depends on + systemd/libjournal. Common compression libraries are allowed after review. +- Compiler minimums: SDK Go and Rust minimums never exceed Netdata's declared + ones (the consuming Go module's `go` directive and Rust workspace's + `rust-version`, read from current Netdata source), because a higher minimum + makes the SDK unusable by Netdata. This holds for dependency refreshes and + vulnerability or scanner cleanup too: pick compatible dependency versions and + disposition findings by reachability. Newer toolchains are extra validation + targets only. +- API layers: each language provides an idiomatic SDK API plus a + libsystemd-compatible reader facade, unless a SOW shows the facade would need + native bindings, break the pure-language policy, or be unsafe or + unrepresentable. +- Writer target includes compression and Forward Secure Sealing where systemd + defines them, possibly phased. Readers handle the historical variants covered + by the shared conformance fixtures. +- journalctl semantics: repeated matches on one field are already OR, so no + `KEY in [values]` syntax is needed; the `+` disjunction separator is systemd + behavior to replicate. Daemon-only commands (sync, flush, rotate, + relinquish-var) are out of scope. +- Before Netdata integration or stable release work advances, GitHub code + scanning and Codacy findings are fixed or dispositioned in the SOW. Raw SARIF, + Codacy exports and scanner logs stay under `.local/`; durable artifacts get + sanitized summaries only. From 55b13ec945e3452d8af29f6d05d9786218e4ed85 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sat, 31 Oct 2026 17:14:17 +0000 Subject: [PATCH 2/8] Reorganize SOW backlog after Netdata usage review - Close SOW-0048 as obsolete: Netdata's OTel backend no longer writes journal files. - Supersede SOW-0049 and SOW-0050 with SOW-0164 (Netdata SDK-exclusive adoption and vendored journal removal). - Add SOW-0162 (publishable C reader crate) and SOW-0163 (v0.11.0 release). - Merge SOW-0155 into SOW-0156. - Repoint SOW-0066 to the v1.0 API stability decision gated on SOW-0164. - Refresh stale blockers in SOW-0045 and SOW-0071 and record the backlog order in SOW-status.md. --- .agents/sow/SOW-status.md | 134 +++++----- ...260528-go-reader-alignment-optimization.md | 6 +- ...e-purity-and-optional-platform-services.md | 4 + ...528-netdata-otel-writer-sdk-integration.md | 7 +- ...8-netdata-reader-plugin-sdk-integration.md | 7 +- ...260528-netdata-vendored-journal-removal.md | 6 +- ...009-writer-tail-offset-representability.md | 5 +- ...530-v1-release-and-registry-publication.md | 10 +- ...20261010-fixed-header-consistency-audit.md | 5 + ...162-20261031-publishable-c-reader-crate.md | 246 +++++++++++++++++ .../SOW-0163-20261031-v0-11-0-release.md | 228 ++++++++++++++++ ...20261031-netdata-sdk-exclusive-adoption.md | 248 ++++++++++++++++++ 12 files changed, 822 insertions(+), 84 deletions(-) rename .agents/sow/{pending => done}/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md (91%) rename .agents/sow/{pending => done}/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md (90%) rename .agents/sow/{pending => done}/SOW-0050-20260528-netdata-vendored-journal-removal.md (95%) rename .agents/sow/{pending => done}/SOW-0155-20261009-writer-tail-offset-representability.md (97%) create mode 100644 .agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md create mode 100644 .agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md create mode 100644 .agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 63ec8597..a621b6ff 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -1,83 +1,82 @@ # SOW Status -Last updated: 2026-10-26 +Last updated: 2026-10-31 -## Current +## Backlog Order + +The user decided on 2026-10-31 to fix all open SOWs rather than close +valid-but-low-impact leads. Execution is one SOW at a time in this proposed +order; the user may reorder it. -- SOW-0045 - Go Reader Alignment Optimization: regression reopened and paused. - Unchanged HEAD reproduces skipped appends after healthy empty/EOF refresh, - including unique-triggered refresh. Separate positioning work remains queued; - a pre-existing directory realtime-seek/getter gap and unfiltered time-probe - error suppression are also recorded for that analysis. SOW-0108 is complete; - SDK-then-CLI execution remains unchanged. +1. SOW-0045 - Go reader regressions (refresh skips first append; directory + seek getters; swallowed time-search errors). +2. SOW-0071 - Runtime purity guard misses Go process-launch variants. +3. SOW-0123 - Rust fresh-journal realtime clock seed (event-time lanes). +4. SOW-0137 - Rust writer residual array-open performance. +5. SOW-0156 - Fixed-header consistency audit (includes merged SOW-0155). +6. SOW-0161 - File indexer source posting membership. +7. SOW-0160 - Go XZ reader dictionary allocation. +8. SOW-0122 - journalctl performance follow-ups. +9. SOW-0097 - Go duplication debt. +10. SOW-0098 - Rust legacy core / `jf` duplication debt (feeds SOW-0162). +11. SOW-0153 - Consumer docs release clarity. +12. SOW-0125 - Netdata filter operators (needs design decisions first). +13. SOW-0162 - Publishable C reader crate. +14. SOW-0163 - v0.11.0 release. +15. SOW-0164 - Netdata SDK-exclusive adoption and vendored journal removal. +16. SOW-0066 - v1.0.0 API stability decision. +## Current + +- SOW-0045 - Go Reader Alignment Optimization: regression reopened, paused. + Healthy Go `Refresh()`/`QueryUnique()` at EOF or on an empty file skips the + first newly appended entry (reproduced on HEAD and go/v0.10.0). Also tracks + directory-seek stale getters and suppressed time-probe errors. Netdata does + not call the affected APIs. Former SOW-0108/0159/0121 blockers are done. - SOW-0071 - Runtime Purity And Optional Platform Services: regression - reopened and paused. The static guard misses standard Go process-launch - spellings despite its original acceptance promise. No SDK runtime launch - is demonstrated; scanner design and implementation remain unapproved. - Tracking adds no prerequisite or change to the 0108 -> 0159 -> 0121 order. + reopened, paused. The static guard misses `exec.CommandContext`, + `os.StartProcess` and `syscall.ForkExec`; no SDK runtime process launch + exists. Former blockers are done. ## Pending -- SOW-0161 - File Indexer Source Posting Membership: open source-analysis - lead, not activated. Investigate inherited source-map target membership and - captured-tail handling before selecting any behavior/performance change. -- SOW-0160 - Go XZ Reader Dictionary Allocation: open, not activated. Tracks - unchanged cumulative dictionary allocation measured during SOW-0108, separate - from required regression prevention. No decoder redesign is authorized and - the approved current execution order is unchanged. -- SOW-0156 - Fixed-Header Consistency Audit: separate follow-up agreed in - PR #9. Define invariants, audit both SDKs, then implement/test missing rules. - Not activated by the current array-count repair. -- SOW-0155 - Writer Tail Offset Representability: source-analysis lead from PR #9. - Investigate regular writers' unchecked tail-offset narrowing; no implementation - or public behavior decision is authorized. -- SOW-0153 - Consumer Documentation Release Clarity: open. Tracks pre-existing - Go stability guidance naming `go/v0.3.0` and Rust writer-state method - discoverability. No implementation is authorized; documentation approach - requires a user decision. Neither issue blocks the paired release. -- SOW-0137 - Rust Writer Residual Array-Open Performance: open. Tracks measured - follow-up analysis for remaining global ENTRY-array and regular/fallback DATA - array reopens found during SOW-0135 review; no implementation is authorized. -- SOW-0048 - Netdata OTEL Writer SDK Integration: open. Component integration - for OTEL writer paths after inventory and writer gates. -- SOW-0049 - Netdata Reader Plugin SDK Integration: open. Component integration - for OTEL signal viewer, no-libsystemd systemd journal reading, and static - packaging after reader gates. -- SOW-0050 - Netdata Vendored Journal Removal: open. Final cleanup after all - Netdata component integrations are complete. -- SOW-0066 - V1 Release And Registry Publication: open. Final `v1.0.0` - Rust/Go release, Rust registry publication where applicable, Go module tags, - and clean consumer install - validation after compatibility, portability, corpus, integration, and parity - gates are complete. -- SOW-0097 - Go Codacy Metric Debt Refactor: open, parked by user decision on - 2026-06-21. Refreshed and narrowed to real duplicated Go journal filesystem - helper debt first, but not mandatory before integration/release work. -- SOW-0098 - Rust Legacy Core Duplication Debt: open, parked by user decision - on 2026-06-21. Still valid as Rust quality debt, but live-path work is - high-risk and not mandatory before integration/release work. -- SOW-0122 - Journalctl Performance Followups: open. Tracks non-blocking - SOW-0121 broader performance follow-up for large invocation inputs, portable - `--follow` scalability, and large-fixture performance evidence before any - large-archive performance claim. Mandatory no-regression proof for the new - SDK/journalctl repairs belongs directly to SOW-0159/SOW-0121. -- SOW-0123 - Event-Time Lane Fresh-Journal Realtime Clock Seed: open. Records - the Rust fresh-lazy realtime floor gap and Go parity/documentation gap for - event-time-ordered journals. Project-manager diagnosis and implementation - plan are complete; glm, minimax, kimi, mimo, deepseek, and qwen returned - ready-to-implement verification. Rust/Go implementation remains pending. -- SOW-0125 - Netdata Filter Operators Gap Analysis: open, parked by user - decision on 2026-06-25. Preserves the requested string/numeric operator - support matrix, current exact-positive filter gap analysis, and open semantic - decisions. Not executable until the user explicitly resumes it. +- SOW-0066 - V1 Release: now the v1.0.0 API stability decision; Netdata gate + is SOW-0164. +- SOW-0097 - Go Codacy Metric Debt Refactor: real duplicated helpers; cited + line numbers need refreshing at activation. +- SOW-0098 - Rust Legacy Core Duplication Debt: real `jf` duplication; feeds + SOW-0162. +- SOW-0122 - Journalctl Performance Followups: unmeasured large-input and + `--follow` scalability work for the CLIs. +- SOW-0123 - Event-Time Lane Fresh-Journal Realtime Clock Seed: Rust floor gap + plus Go API parity; plan already prepared. +- SOW-0125 - Netdata Filter Operators Gap Analysis: feature analysis; needs + semantic decisions before implementation. +- SOW-0137 - Rust Writer Residual Array-Open Performance: unmeasured double + array open per append. +- SOW-0153 - Consumer Documentation Release Clarity: stale `go/v0.3.0` text and + Rust writer-state discoverability. +- SOW-0156 - Fixed-Header Consistency Audit: now includes SOW-0155 tail-offset + representability. +- SOW-0160 - Go XZ Reader Dictionary Allocation: per-DATA decoder allocation. +- SOW-0161 - File Indexer Source Posting Membership: unproven indexer lead. +- SOW-0162 - Publishable C Reader Crate: new, 2026-10-31. +- SOW-0163 - v0.11.0 Release: new, 2026-10-31. +- SOW-0164 - Netdata SDK-Exclusive Adoption And Vendored Journal Removal: new, + 2026-10-31; supersedes SOW-0049 and SOW-0050. ## Recently Closed Or Completed +- SOW-0048 - Netdata OTEL Writer SDK Integration: closed 2026-10-31 as obsolete; + Netdata's OTel backend no longer writes journal files. +- SOW-0049 / SOW-0050 - Netdata reader integration and vendored removal: closed + 2026-10-31, superseded by SOW-0164. +- SOW-0155 - Writer Tail Offset Representability: closed 2026-10-31, merged into + SOW-0156. + - SOW-0121 - File-Backed Journalctl Full Parity And Ship Decision: completed. Directional cursor integration and PR12 fixes pass three internal reviews, - 3,514 native comparisons and146 portable controls. Pushed-head native CI is - pending; historical qualification follows. + 3,514 native comparisons and 146 portable controls. The vacuum regression is repaired under approved deterministic history-head ordering. Both languages pass independent 10,000-set models; native CI passes Linux/macOS/Windows/FreeBSD with 130 portable controls. Existing 78 utility @@ -85,7 +84,8 @@ Last updated: 2026-10-26 impact review and normal final-commit CI complete publication. Upstream Codacy configuration retry and coverage upload fail remotely; native tests, coverage generation and CodeQL pass. These CI limitations stay explicit. - Broader scalability remains SOW-0122; PR #12 is draft, not merged/released. + Broader scalability remains SOW-0122. PR #12 merged on 2026-10-27 (00d33d4); + unreleased until SOW-0163. - SOW-0159 - Rust/Go SDK Hardening And Reader Compatibility: completed. SDK directional cursor APIs pass three independent reviews; CLI integration is diff --git a/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md b/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md index b215d9a0..fa5b1a56 100644 --- a/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md +++ b/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md @@ -16,7 +16,11 @@ under the one-active-SOW rule; prerequisite SOW-0108 and SDK SOW-0159 are completed, with journalctl SOW-0121 next after the signed SDK commit. Do not insert this separate positioning repair into the approved SOW-0108 -> SOW-0159 -> SOW-0121 execution chain without a dependency finding -or a user scheduling decision. Original completion evidence remains below. +or a user scheduling decision. + +2026-10-31: SOW-0108, SOW-0159 and SOW-0121 are all completed, so that +ordering no longer blocks this SOW. It waits only for activation in the +user-approved backlog order recorded in `.agents/sow/SOW-status.md`. Original completion evidence remains below. ## Requirements diff --git a/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md index a0e5df53..1a164570 100644 --- a/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md +++ b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md @@ -12,6 +12,10 @@ Go process-launch API variants. Tracking is paused; no SDK runtime impurity is proved and no scanner implementation is authorized by this finding. Preserve the active SOW-0108, then SOW-0159, then SOW-0121 execution order. +2026-10-31: SOW-0108, SOW-0159 and SOW-0121 are all completed, so that +ordering no longer blocks this SOW. It waits only for activation in the +user-approved backlog order recorded in `.agents/sow/SOW-status.md`. + ## Requirements ### Purpose diff --git a/.agents/sow/pending/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md b/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md similarity index 91% rename from .agents/sow/pending/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md rename to .agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md index 550b6da8..9ea574fa 100644 --- a/.agents/sow/pending/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md +++ b/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md @@ -2,10 +2,9 @@ ## Status -Status: open +Status: closed -Sub-state: created on 2026-05-28 as a component integration SOW, blocked by -SOW-0026 inventory and writer performance gates. +Sub-state: closed on 2026-10-31 as obsolete. Netdata's OTel backend no longer writes journal files, so there is no OTel journal writer to migrate. ## Requirements @@ -232,7 +231,7 @@ Follow-up mapping: ## Outcome -Pending. +Closed as obsolete without implementation. Netdata master checked on 2026-10-31 (`netdata/netdata @ eaeb95f5c4`): `src/crates/otel-plugin/Cargo.toml` has no journal crate dependency; OTel logs go through ingestor, WAL and `otel-ledger`. Vendored `journal-log-writer` has no consumer. Removal of the remaining vendored crates is tracked by SOW-0164. ## Lessons Extracted diff --git a/.agents/sow/pending/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md similarity index 90% rename from .agents/sow/pending/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md rename to .agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md index 55161783..308727a7 100644 --- a/.agents/sow/pending/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md +++ b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md @@ -2,10 +2,9 @@ ## Status -Status: open +Status: closed -Sub-state: created on 2026-05-28 as the Netdata reader component integration -SOW, blocked by SOW-0026 inventory and reader performance gates. +Sub-state: closed; superseded on 2026-10-31 by SOW-0164 (Netdata SDK-exclusive adoption and vendored journal removal), which the user approved as one SOW for removing the old implementation and adopting the SDK. The `otel-signal-viewer.plugin` target no longer exists in Netdata. ## Requirements @@ -243,7 +242,7 @@ Follow-up mapping: ## Outcome -Pending. +Closed without implementation; scope moved to SOW-0164. Current Netdata reader consumers of vendored code: `systemd-journal.plugin` via `src/crates/jf` (`journal_reader_ffi`, Docker/static builds with `--internal-systemd-journal`) and `otel-legacy-logs` via `journal-function`/`journal-index`. The SDK-side publishable C reader crate is SOW-0162. ## Lessons Extracted diff --git a/.agents/sow/pending/SOW-0050-20260528-netdata-vendored-journal-removal.md b/.agents/sow/done/SOW-0050-20260528-netdata-vendored-journal-removal.md similarity index 95% rename from .agents/sow/pending/SOW-0050-20260528-netdata-vendored-journal-removal.md rename to .agents/sow/done/SOW-0050-20260528-netdata-vendored-journal-removal.md index 5a3f0df4..aa4fa09b 100644 --- a/.agents/sow/pending/SOW-0050-20260528-netdata-vendored-journal-removal.md +++ b/.agents/sow/done/SOW-0050-20260528-netdata-vendored-journal-removal.md @@ -2,9 +2,9 @@ ## Status -Status: open +Status: closed -Sub-state: created on 2026-05-28 as the final Netdata cleanup SOW. +Sub-state: closed; superseded on 2026-10-31 by SOW-0164 (Netdata SDK-exclusive adoption and vendored journal removal), which the user approved as one SOW for removing the old implementation and adopting the SDK. ## Requirements @@ -228,7 +228,7 @@ Follow-up mapping: ## Outcome -Pending. +Closed without implementation; scope moved to SOW-0164. ## Lessons Extracted diff --git a/.agents/sow/pending/SOW-0155-20261009-writer-tail-offset-representability.md b/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md similarity index 97% rename from .agents/sow/pending/SOW-0155-20261009-writer-tail-offset-representability.md rename to .agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md index acc1608b..2b53dc53 100644 --- a/.agents/sow/pending/SOW-0155-20261009-writer-tail-offset-representability.md +++ b/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md @@ -2,10 +2,9 @@ ## Status -Status: open +Status: closed -Sub-state: source-analysis lead from PR #9 cubic comment 4234196552; pending -activation. No implementation or product decision is authorized by this record. +Sub-state: closed on 2026-10-31; merged into SOW-0156 by user decision, since both concern writer/header offset and consistency hardening. ## Requirements diff --git a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md index 83f48eca..29db137c 100644 --- a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md +++ b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md @@ -8,6 +8,12 @@ Sub-state: pending final production release and Rust/Go publication after SDK compatibility, portability, corpus validation, and integration gates are complete. +2026-10-31: registry publication and Go module tags already exist through +v0.10.0 and Netdata consumes them, so this SOW is now the v1.0.0 API stability +decision. Its Netdata integration gate is SOW-0164 (SOW-0048 closed as obsolete; +SOW-0049 and SOW-0050 superseded by SOW-0164). The compatibility, portability and +corpus gates (SOW-0055, SOW-0063, SOW-0064, SOW-0065) are already done. + ## Requirements ### Purpose @@ -93,7 +99,7 @@ Current state: `v1.0.0` publication workflow recorded as a SOW. - Cross-platform portability and real-world corpus evaluation are pending. - Future language parity closure is now tracked by SOW-0065. -- Netdata integration SOWs are pending and depend on SDK readiness. +- Netdata integration is tracked by SOW-0164 (supersedes SOW-0048/0049/0050). Risks: @@ -120,7 +126,7 @@ Evidence reviewed: - User request for final `v1.0.0` and language registry publication. - Pending SOWs that still affect release readiness: SOW-0055, SOW-0063, - SOW-0064, SOW-0065, and Netdata integration SOWs. + SOW-0064, SOW-0065, and SOW-0164. - Project release-tagging skill exists and should be used when this SOW starts. Affected contracts and surfaces: diff --git a/.agents/sow/pending/SOW-0156-20261010-fixed-header-consistency-audit.md b/.agents/sow/pending/SOW-0156-20261010-fixed-header-consistency-audit.md index 3117aed8..f09df7ca 100644 --- a/.agents/sow/pending/SOW-0156-20261010-fixed-header-consistency-audit.md +++ b/.agents/sow/pending/SOW-0156-20261010-fixed-header-consistency-audit.md @@ -7,6 +7,11 @@ Status: open Sub-state: separate follow-up agreed by the user; not activated by PR #9's concrete array-count correction. No implementation is authorized here yet. +2026-10-31: SOW-0155 (writer tail offset representability) merged into this SOW +by user decision. Its scope, evidence and open questions in +`.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md` +are part of this audit. + ## Requirements ### Purpose diff --git a/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md b/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md new file mode 100644 index 00000000..bec4ae28 --- /dev/null +++ b/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md @@ -0,0 +1,246 @@ +# SOW-0162 - Publishable C Reader Crate + +## Status + +Status: open + +`completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`. + +Sub-state: created on 2026-10-31 by user decision. Not activated; runs after the +debt SOWs in the backlog order in `.agents/sow/SOW-status.md` and before the +v0.11.0 release (SOW-0163). + +## Requirements + +### Purpose + +Let Netdata's `systemd-journal.plugin` consume this SDK's C reader as a +published dependency, the same way `netflow-plugin` consumes the Rust crates, +so Netdata's vendored `src/crates/jf` copy can be deleted (SOW-0164). + +### User Request + +The user chose to "publish the C reader as an SDK crate, so Netdata uses it +like netflow uses the SDK", with the goal that the old Netdata crates vanish +and Netdata uses the SDK exclusively. + +### Assistant Understanding + +Facts: + +- Netdata's `systemd-journal.plugin` calls the `rsd_journal_*` C functions + exported by `journal_reader_ffi` from its vendored `src/crates/jf` + workspace (`packaging/cmake/Modules/NetdataPluginSystemdJournal.cmake:55-57`, + `NetdataRust.cmake:61-63`). Docker and static builds enable it with + `--internal-systemd-journal`. +- This repository has its own evolved `rust/src/crates/jf` with the same + `rsd_journal_*` entry points (spot-checked: `open_files`, `query_unique`, + `seek_realtime_usec`). It is `publish = false`. +- The two copies differ by roughly 4,000 diff lines; Netdata's lacks later SDK + fixes. + +Inferences: + +- The crate should be built on the SDK's published reader/facade crates + rather than kept as a separate reader implementation, so one reader exists. + Needs confirmation during analysis. + +Unknowns: + +- Whether every symbol and struct layout used by the C plugin is present and + ABI-identical in the SDK copy. +- Whether `jf` can be thinned onto the SDK facade without behavior change. + +### Acceptance Criteria + +- A publishable crate exposes every C function and type Netdata's + `systemd-journal.plugin` uses, verified against the plugin's call sites. +- The crate is part of the release workflow's publish order and dry-runs. +- C-level tests exercise the exported API against shared fixtures. +- SOW-0098 (`jf` duplication debt) is resolved or explicitly folded in. + +## Analysis + +Sources checked: + +- Netdata `src/crates/jf`, `src/collectors/systemd-journal.plugin`, CMake modules. +- `rust/src/crates/jf` in this repository. + +Current state: + +- See Facts above. + +Risks: + +- ABI mismatch breaks the C plugin at link or run time. +- Adding a published crate changes release order and Trusted Publishing setup + (a new crate needs an authorized initial publication). + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- Netdata vendors a drifting copy of the C reader because the SDK's copy is not published. + +Evidence reviewed: + +- Listed under Assistant Understanding. + +Affected contracts and surfaces: + +- New public crate and C ABI; release workflow; `project-release` skill crate list. + +Existing patterns to reuse: + +- Published `systemd-journal-sdk-*` crates and the release workflow. + +Risk and blast radius: + +- See Risks. Rust-only by nature (C ABI); the Go SDK is unaffected. Record this + as a parity exception requiring user confirmation. + +Sensitive data handling plan: + +- No sensitive data expected; synthetic fixtures only. + +Implementation plan: + +1. To be written at activation from the symbol inventory. + +Validation plan: + +- To be written at activation. + +Artifact impact plan: + +- AGENTS.md: likely unaffected. +- Runtime project skills: `project-release` crate list and order. +- Specs: product scope gains the C reader crate. +- End-user/operator docs: README and `docs/` for the C reader. +- End-user/operator skills: none exist. +- SOW lifecycle: prerequisite of SOW-0163 and SOW-0164. +- SOW-status.md: updated on activation and close. + +Open-source reference evidence: + +- Netdata repository (`netdata/netdata`) is the consumer; cite `owner/repo @ commit` at activation. + +Open decisions: + +1. Go parity exception for a C-ABI crate (expected: Rust-only). +2. Crate name and whether it wraps the SDK facade or keeps `jf` internals. + +## Implications And Decisions + +- 2026-10-31: user chose a published SDK crate over re-copying `jf` into Netdata. + +## Plan + +1. Activate after the debt SOWs; fill the gate from a symbol inventory. + +## Implementation And Review Plan + +Implementation: + +- Project manager implements and validates. + +Reviewers: + +- Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used. + +Repository boundary block for every external-agent prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Record failures in this SOW and stop for the user on design changes. + +## Execution Log + +### 2026-10-31 + +- Created from the SOW backlog review. + +## Validation + +Acceptance criteria evidence: + +- Pending. + +Tests or equivalent validation: + +- Pending. + +Real-use evidence: + +- Pending. + +Reviewer findings: + +- Pending. + +Same-failure scan: + +- Pending. + +Sensitive data gate: + +- Pending. + +Artifact maintenance gate: + +- Pending. + +Specs update: + +- Pending. + +Project skills update: + +- Pending. + +End-user/operator docs update: + +- Pending. + +End-user/operator skills update: + +- Pending. + +Lessons: + +- Pending. + +Follow-up mapping: + +- Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. + +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative. diff --git a/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md new file mode 100644 index 00000000..d502fe03 --- /dev/null +++ b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md @@ -0,0 +1,228 @@ +# SOW-0163 - v0.11.0 Release + +## Status + +Status: open + +`completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`. + +Sub-state: created on 2026-10-31 by user decision ("we need to release 0.11.0 +once we are ready"). Not activated; runs after the debt SOWs and SOW-0162. + +## Requirements + +### Purpose + +Ship the SDK fixes made since v0.10.0 and the backlog fixes, and publish the C +reader crate, so Netdata can consume them. + +### User Request + +Release 0.11.0 once the backlog work is ready. + +### Assistant Understanding + +Facts: + +- 15 commits after `v0.10.0` (2026-10-11) are unreleased. SDK-affecting ones + include `25accee` (reader recovery: crash-truncated files no longer silently + dropped by the Go Netdata logs function; cycle guards), `a1144dc`, `4c49205`, + `5224c25`, `f50676f`, `3ee6203`. +- Netdata uses Go `v0.10.0` and Rust `0.8.1` (`netflow-plugin`). +- `rust/Cargo.toml` still declares `0.10.0`. + +Inferences: + +- Additive APIs (e.g. `SeekCursorDirection`) and recovery-by-default behavior + suggest a minor release; confirm semver during preparation. + +Unknowns: + +- Final scope depends on which backlog SOWs complete first. + +### Acceptance Criteria + +- Preparation and publication follow `project-release` and `RELEASING.md`. +- All crates, including SOW-0162's, published; paired `v0.11.0` and + `go/v0.11.0` tags verified with fresh consumers. + +## Analysis + +Sources checked: + +- `git log v0.10.0..HEAD`; Netdata `src/go/go.mod`, `src/crates/Cargo.toml`. + +Current state: + +- See Facts. + +Risks: + +- New crate needs an authorized initial publication before Trusted Publishing. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- Fixes Netdata would benefit from are unreleased. + +Evidence reviewed: + +- Listed above. + +Affected contracts and surfaces: + +- Rust crates, Go module tags, install examples, release notes. + +Existing patterns to reuse: + +- SOW-0157 / SOW-0158 (v0.10.0 preparation and publication). + +Risk and blast radius: + +- Published artifacts and tags are irreversible. + +Sensitive data handling plan: + +- No credentials in durable artifacts. + +Implementation plan: + +1. To be written at activation per `project-release`. + +Validation plan: + +- To be written at activation per `project-release`. + +Artifact impact plan: + +- AGENTS.md: likely unaffected. +- Runtime project skills: `project-release` if the crate list changes. +- Specs: product scope version references. +- End-user/operator docs: install examples and migration notes. +- End-user/operator skills: none exist. +- SOW lifecycle: publication may be tracked as a separate execution SOW, as in v0.10.0. +- SOW-status.md: updated on activation and close. + +Open-source reference evidence: + +- None needed. + +Open decisions: + +1. Final scope and semver, decided at preparation. + +## Implications And Decisions + +- 2026-10-31: user approved releasing 0.11.0 when the backlog is ready. + +## Plan + +1. Prepare, merge, publish, tag and verify per `project-release`. + +## Implementation And Review Plan + +Implementation: + +- Project manager implements; publication and tag pushes need the user's release request. + +Reviewers: + +- Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used. + +Repository boundary block for every external-agent prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Follow `RELEASING.md` recovery; stop for the user on conflicts. + +## Execution Log + +### 2026-10-31 + +- Created from the SOW backlog review. + +## Validation + +Acceptance criteria evidence: + +- Pending. + +Tests or equivalent validation: + +- Pending. + +Real-use evidence: + +- Pending. + +Reviewer findings: + +- Pending. + +Same-failure scan: + +- Pending. + +Sensitive data gate: + +- Pending. + +Artifact maintenance gate: + +- Pending. + +Specs update: + +- Pending. + +Project skills update: + +- Pending. + +End-user/operator docs update: + +- Pending. + +End-user/operator skills update: + +- Pending. + +Lessons: + +- Pending. + +Follow-up mapping: + +- Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. + +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative. diff --git a/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md new file mode 100644 index 00000000..eb03e482 --- /dev/null +++ b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md @@ -0,0 +1,248 @@ +# SOW-0164 - Netdata SDK-Exclusive Adoption And Vendored Journal Removal + +## Status + +Status: open + +`completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`. + +Sub-state: created on 2026-10-31 by user decision; supersedes SOW-0049 and +SOW-0050 (SOW-0048 closed as obsolete). Blocked on SOW-0162 and the v0.11.0 +release (SOW-0163). Changes the Netdata repository, which needs explicit user +authorization of that target at activation. + +## Requirements + +### Purpose + +Netdata uses this SDK exclusively for journal files; every vendored or local +journal implementation in Netdata is removed. + +### User Request + +"The idea is the old crates in netdata to vanish and use the SDK exclusively." +"One SOW for the removal of the old and the use of the new." The user also +asked to port the legacy logs viewer to the SDK. + +### Assistant Understanding + +Facts (Netdata master, `netdata/netdata @ eaeb95f5c4`, 2026-10-31): + +- `systemd-journal.plugin` uses vendored `src/crates/jf` (`journal_reader_ffi`) + in Docker and static builds (`Dockerfile:42`, + `packaging/makeself/jobs/70-netdata-git.install.sh:40`); other builds link + libsystemd. Netdata documents a broken no-libsystemd fallback + (`NetdataPluginSystemdJournal.cmake:16-22`). +- `otel-legacy-logs` (read-only viewer of the former OTel plugin's journal + files, compiled into `otel-plugin`, `otel-legacy-logs/src/lib.rs:1-10`) uses + vendored `journal-function` and `journal-index`; `journal-function` uses + vendored `journal-core`, `journal-engine`, `journal-index`, `journal-registry`. +- `ng-ingest` (experiment) uses vendored `journal-common` only for + `load_machine_id` / `load_boot_id`. +- Vendored `journal-log-writer` has no consumer. +- `netflow-plugin` already uses the published SDK at `0.8.1`; Go consumers use + `v0.10.0`. +- The SDK publishes `journal-core`, `journal-index`, `journal-engine`, + `journal-registry` with the types `journal-function` uses (`FileIndex`, + `Direction`, `Histogram`, `Table`). `journal-function` itself is + Netdata-side plugin glue and stays in Netdata. + +Unknowns: + +- API drift between vendored and SDK crates for `journal-function`'s use. +- Whether `netflow-plugin` needs code changes moving from `0.8.1`. + +### Acceptance Criteria + +- `systemd-journal.plugin` uses the SDK's published C reader crate (SOW-0162). +- `otel-legacy-logs` / `journal-function` and `ng-ingest` use SDK crates. +- `netflow-plugin` and Go consumers move to the new release. +- Netdata's `src/crates/{jf,journal-common,journal-core,journal-index,journal-engine,journal-registry,journal-log-writer}` + are deleted, and a fresh search proves no references remain. +- Netdata builds, tests and Docker/static packaging pass, including the + internal-reader path. + +## Analysis + +Sources checked: + +- Netdata master as cited above; superseded SOW-0049 and SOW-0050. + +Current state: + +- See Facts. + +Risks: + +- Behavior change in the system logs viewer for Docker/static users. +- Cross-repository change; Netdata review and CI apply. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- Netdata ships three journal implementations; the vendored ones drift from + the SDK and miss its fixes. + +Evidence reviewed: + +- Listed under Facts. + +Affected contracts and surfaces: + +- Netdata Rust workspace, CMake and packaging, `systemd-journal.plugin`, + `otel-plugin`, `netflow-plugin`, Go plugins. + +Existing patterns to reuse: + +- `netflow-plugin`'s published-crate dependency (`src/crates/Cargo.toml:176-182`). + +Risk and blast radius: + +- See Risks. + +Sensitive data handling plan: + +- No sensitive data expected. + +Implementation plan: + +1. To be written at activation. + +Validation plan: + +- To be written at activation. + +Artifact impact plan: + +- AGENTS.md: likely unaffected. +- Runtime project skills: likely unaffected. +- Specs: Netdata integration status in product scope. +- End-user/operator docs: integration docs if they mention vendored code. +- End-user/operator skills: none exist. +- SOW lifecycle: supersedes SOW-0049/0050; unblocks SOW-0066 Netdata gate. +- SOW-status.md: updated on activation and close. + +Open-source reference evidence: + +- `netdata/netdata @ eaeb95f5c4` paths as listed above. + +Open decisions: + +1. Netdata repository write authorization (required before any Netdata change). + +## Implications And Decisions + +- 2026-10-31: user chose one SOW for adoption plus removal, a published C + reader crate (SOW-0162), and porting the legacy logs viewer to the SDK. + +## Plan + +1. Activate after SOW-0163; refresh the Netdata inventory first. + +## Implementation And Review Plan + +Implementation: + +- Project manager implements once the Netdata target is authorized. + +Reviewers: + +- Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used. + +Repository boundary block for every external-agent prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Record failures here; stop for the user on design changes. + +## Execution Log + +### 2026-10-31 + +- Created from the SOW backlog review; SOW-0049 and SOW-0050 closed as superseded. + +## Validation + +Acceptance criteria evidence: + +- Pending. + +Tests or equivalent validation: + +- Pending. + +Real-use evidence: + +- Pending. + +Reviewer findings: + +- Pending. + +Same-failure scan: + +- Pending. + +Sensitive data gate: + +- Pending. + +Artifact maintenance gate: + +- Pending. + +Specs update: + +- Pending. + +Project skills update: + +- Pending. + +End-user/operator docs update: + +- Pending. + +End-user/operator skills update: + +- Pending. + +Lessons: + +- Pending. + +Follow-up mapping: + +- Pending. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. + +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative. From 7c7011d1e067270fa8e497478bc6d4056abcc16f Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 19:59:04 +0000 Subject: [PATCH 3/8] Fix stale lifecycle text, citations and gate wording in process docs Qualify the no-CGO goal to the SDKs, restore the compatible-tooling qualifier, separate completed from closed SOWs, point the validation gate at the template, complete skill script paths and coverage, and refresh stale SOW status text and Netdata citations. --- .../project-journal-compatibility/SKILL.md | 11 ++++++----- .../references/validation-recipes.md | 2 +- .agents/skills/project-release/SKILL.md | 2 +- ...0260528-go-reader-alignment-optimization.md | 5 ++--- ...me-purity-and-optional-platform-services.md | 4 ++-- ...20260528-netdata-netflow-sdk-integration.md | 2 ++ ...0528-netdata-otel-writer-sdk-integration.md | 2 +- ...28-netdata-reader-plugin-sdk-integration.md | 2 +- ...60621-file-backed-journalctl-full-parity.md | 4 ++-- ...1009-writer-tail-offset-representability.md | 2 +- ...0530-v1-release-and-registry-publication.md | 14 +++++++------- ...0162-20261031-publishable-c-reader-crate.md | 4 ++-- .../SOW-0163-20261031-v0-11-0-release.md | 6 ++++-- ...-20261031-netdata-sdk-exclusive-adoption.md | 10 +++++----- AGENTS.md | 18 +++++++++++------- 15 files changed, 48 insertions(+), 40 deletions(-) diff --git a/.agents/skills/project-journal-compatibility/SKILL.md b/.agents/skills/project-journal-compatibility/SKILL.md index 5c1cb422..adefdc43 100644 --- a/.agents/skills/project-journal-compatibility/SKILL.md +++ b/.agents/skills/project-journal-compatibility/SKILL.md @@ -113,10 +113,10 @@ SDKs; one language's tests never stand in for the other's. | Compact writing | `tests/interoperability/run_compact_matrix.py` | | Deterministic layout | `tests/interoperability/run_byte_identity.py --final-state all` (requires `all_equal: true`, including hash-chain depth) | | Optional writer-lock helper | `tests/interoperability/run_lock_matrix.py` | -| Directory reading / `--directory` | `tests/interoperability/run_directory_matrix.py`, `run_mixed_directory_matrix.py` | +| Directory reading / `--directory` | `tests/interoperability/run_directory_matrix.py`, `tests/interoperability/run_mixed_directory_matrix.py` | | journalctl query / follow | `tests/interoperability/run_journalctl_query_matrix.py` | -| journalctl options / parser | `tests/parser-parity/check_v260_manifest.py`, `run_parser_parity.py` | -| Verifier | `tests/interoperability/run_verify_matrix.py` | +| journalctl options / parser | `tests/parser-parity/check_v260_manifest.py`, `tests/parser-parity/run_parser_parity.py` | +| Verifier | `tests/interoperability/run_verify_matrix.py` (stock `journalctl --verify --file`, Rust and Go agree on regular, zstd/xz/lz4, compact, compact+compression and sealed files, and on object type/size, payload hash, hash-chain, entry-array, header-counter, seqnum, monotonic and TAG/FSS corruption) | | Traversal, compressed readers, recovery | `tests/interoperability/run_reader_hardening.py` with both adapters built and an explicit stock oracle | | Reader performance | `tests/benchmarks/run_reader_regression_comparison.py` | @@ -125,8 +125,9 @@ CGO-disabled SDK cross-build says nothing about CLI portability. ## Done -A compatibility change is done when the matrices covering it pass for every -affected language and file variant, cross-language files round-trip, stock +A compatibility change is done when the shared conformance tests and the +matrices covering it pass for every affected language and file variant, +cross-language files round-trip for every affected writer/reader pair, stock readers read live files from each affected writer, and the SOW records the stock systemd version, commands, durations, entry and reader counts, and failure criteria used. diff --git a/.agents/skills/project-journal-compatibility/references/validation-recipes.md b/.agents/skills/project-journal-compatibility/references/validation-recipes.md index 2ef8907b..9ae21e60 100644 --- a/.agents/skills/project-journal-compatibility/references/validation-recipes.md +++ b/.agents/skills/project-journal-compatibility/references/validation-recipes.md @@ -151,7 +151,7 @@ Use when changing directory writer retention, rotation, root inventory or header array locations declared by complete header fields. Equal pointers count once; additional arrays are permitted. Test historical counters before cached-tail fields exist and preserve bytes on inventory/startup rejection. - Test small in-bounds contradictions, not only overflowing offsets. Do not impose stable counter consistency on live mapping validation. + Test small in-bounds contradictions, not only overflowing offsets. Apply these counter checks to stable files only, not to live mapping validation. ## Byte identity and historical files diff --git a/.agents/skills/project-release/SKILL.md b/.agents/skills/project-release/SKILL.md index a57d8746..d217d59c 100644 --- a/.agents/skills/project-release/SKILL.md +++ b/.agents/skills/project-release/SKILL.md @@ -37,7 +37,7 @@ for the user on anything it does not cover. - Semver: adding a field to a public Rust struct, or a variant to a public enum, breaks downstream exhaustive literals and matches unless the type is `#[non_exhaustive]`. Record the semver decision in the preparation SOW. -- Before tagging, scan `README.md`, `rust/README.md`, `docs/` and +- Before tagging, scan `README.md`, `go/README.md`, `rust/README.md`, `docs/` and `.agents/sow/specs/` for Rust `version = "…"` and Go `@v…` install examples; every active one must name the new version. - Never record crates.io tokens or credential details in durable artifacts. diff --git a/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md b/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md index fa5b1a56..cf30c110 100644 --- a/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md +++ b/.agents/sow/current/SOW-0045-20260528-go-reader-alignment-optimization.md @@ -14,9 +14,8 @@ appended entry when the reader was empty or at forward EOF. This SOW owns the original refresh implementation and live-reader claim. It is queued/paused under the one-active-SOW rule; prerequisite SOW-0108 and SDK SOW-0159 are completed, with journalctl SOW-0121 next after the signed SDK commit. -Do not insert this separate positioning repair into the approved -SOW-0108 -> SOW-0159 -> SOW-0121 execution chain without a dependency finding -or a user scheduling decision. +At that time this repair stayed out of the approved +SOW-0108 -> SOW-0159 -> SOW-0121 execution chain. 2026-10-31: SOW-0108, SOW-0159 and SOW-0121 are all completed, so that ordering no longer blocks this SOW. It waits only for activation in the diff --git a/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md index 1a164570..9e5eca9d 100644 --- a/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md +++ b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md @@ -9,8 +9,8 @@ review rounds, final cleanup, and final local validation are complete. Regression reopened 2026-10-16: the static core-runtime guard misses standard Go process-launch API variants. Tracking is paused; no SDK runtime impurity is -proved and no scanner implementation is authorized by this finding. Preserve -the active SOW-0108, then SOW-0159, then SOW-0121 execution order. +proved and no scanner implementation is authorized by this finding. At that time +the SOW-0108, then SOW-0159, then SOW-0121 execution order took precedence. 2026-10-31: SOW-0108, SOW-0159 and SOW-0121 are all completed, so that ordering no longer blocks this SOW. It waits only for activation in the diff --git a/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md b/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md index 4302aca7..20ba607a 100644 --- a/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md +++ b/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md @@ -320,6 +320,8 @@ Follow-up mapping: integration. - SOW-0050 remains open for final vendored journal code removal after all component integrations are complete. +- 2026-10-31: SOW-0048 closed as obsolete; SOW-0049 and SOW-0050 superseded by + SOW-0164. ## Outcome diff --git a/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md b/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md index 9ea574fa..7f6d5fa6 100644 --- a/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md +++ b/.agents/sow/done/SOW-0048-20260528-netdata-otel-writer-sdk-integration.md @@ -231,7 +231,7 @@ Follow-up mapping: ## Outcome -Closed as obsolete without implementation. Netdata master checked on 2026-10-31 (`netdata/netdata @ eaeb95f5c4`): `src/crates/otel-plugin/Cargo.toml` has no journal crate dependency; OTel logs go through ingestor, WAL and `otel-ledger`. Vendored `journal-log-writer` has no consumer. Removal of the remaining vendored crates is tracked by SOW-0164. +Closed as obsolete without implementation. Netdata master checked on 2026-10-31 (`netdata/netdata @ eaeb95f5c4`): `src/crates/otel-plugin/Cargo.toml:23-25` has no journal crate dependency (only `otel-ledger` and `otel-legacy-logs`); OTel logs go through ingestor, WAL and `otel-ledger`. Vendored `journal-log-writer` has no consumer. Removal of the remaining vendored crates is tracked by SOW-0164. ## Lessons Extracted diff --git a/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md index 308727a7..7dcadee4 100644 --- a/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md +++ b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md @@ -242,7 +242,7 @@ Follow-up mapping: ## Outcome -Closed without implementation; scope moved to SOW-0164. Current Netdata reader consumers of vendored code: `systemd-journal.plugin` via `src/crates/jf` (`journal_reader_ffi`, Docker/static builds with `--internal-systemd-journal`) and `otel-legacy-logs` via `journal-function`/`journal-index`. The SDK-side publishable C reader crate is SOW-0162. +Closed without implementation; scope moved to SOW-0164. Current Netdata reader consumers of vendored code: `systemd-journal.plugin` via `src/crates/jf` (`journal_reader_ffi`, `packaging/cmake/Modules/NetdataRust.cmake:61-63`; Docker/static builds with `--internal-systemd-journal`, `packaging/docker/Dockerfile:42`) and `otel-legacy-logs` via `journal-function`/`journal-index` (`src/crates/otel-legacy-logs/Cargo.toml:19-20`). The SDK-side publishable C reader crate is SOW-0162. ## Lessons Extracted diff --git a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md index a9403434..d8f8129a 100644 --- a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md +++ b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md @@ -8,8 +8,8 @@ Status: completed Sub-state: PR12 review regressions completed locally 2026-10-26. Both CLIs use the approved directional SDK APIs. Three internal final scopes pass after -3,514 native comparisons and146 portable controls. Final-commit native CI is -pending publication; PR12 remains draft and no merge/release is authorized. +3,514 native comparisons and146 portable controls. PR #12 merged on +2026-10-27 (00d33d4); release is tracked by SOW-0163. ## Requirements diff --git a/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md b/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md index 2b53dc53..00258f42 100644 --- a/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md +++ b/.agents/sow/done/SOW-0155-20261009-writer-tail-offset-representability.md @@ -133,7 +133,7 @@ skills remain unchanged for this unapproved writer contract. ## Outcome -Pending activation; no implementation authorized. +Closed without implementation on 2026-10-31; merged into SOW-0156, which tracks the remaining work. ## Lessons Extracted diff --git a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md index 29db137c..a0c00c23 100644 --- a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md +++ b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md @@ -97,8 +97,8 @@ Current state: - The project has previous pre-1.0 tags and Go module tags, but no final `v1.0.0` publication workflow recorded as a SOW. -- Cross-platform portability and real-world corpus evaluation are pending. -- Future language parity closure is now tracked by SOW-0065. +- Cross-platform portability (SOW-0063) and real-world corpus evaluation (SOW-0064) are completed. +- SOW-0065 (language parity closure) was closed as superseded. - Netdata integration is tracked by SOW-0164 (supersedes SOW-0048/0049/0050). Risks: @@ -125,9 +125,9 @@ Problem / root-cause model: Evidence reviewed: - User request for final `v1.0.0` and language registry publication. -- Pending SOWs that still affect release readiness: SOW-0055, SOW-0063, - SOW-0064, SOW-0065, and SOW-0164. -- Project release-tagging skill exists and should be used when this SOW starts. +- Pending SOW that still affects release readiness: SOW-0164 (SOW-0055, + SOW-0063 and SOW-0064 are completed; SOW-0065 is closed). +- The `project-release` skill and `RELEASING.md` define the release process for this SOW. Affected contracts and surfaces: @@ -140,7 +140,7 @@ Affected contracts and surfaces: Existing patterns to reuse: -- `project-release-tagging` skill for tags and Go module release tags. +- `project-release` skill for tags and Go module release tags. - Existing language README files. - Existing tests and conformance matrices. - Existing SOW release/tagging history for `v0.2.0` and `v0.3.0` style @@ -192,7 +192,7 @@ Artifact impact plan: - AGENTS.md: no update expected unless release workflow becomes a project-wide rule. -- Runtime project skills: use and possibly update `project-release-tagging` if +- Runtime project skills: use and possibly update `project-release` if gaps are found. - Specs: update product scope with `v1.0.0` support guarantees. - End-user/operator docs: update install, usage, platform support, compatibility diff --git a/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md b/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md index bec4ae28..8805bb11 100644 --- a/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md +++ b/.agents/sow/pending/SOW-0162-20261031-publishable-c-reader-crate.md @@ -31,7 +31,7 @@ Facts: - Netdata's `systemd-journal.plugin` calls the `rsd_journal_*` C functions exported by `journal_reader_ffi` from its vendored `src/crates/jf` workspace (`packaging/cmake/Modules/NetdataPluginSystemdJournal.cmake:55-57`, - `NetdataRust.cmake:61-63`). Docker and static builds enable it with + `packaging/cmake/Modules/NetdataRust.cmake:61-63`). Docker and static builds enable it with `--internal-systemd-journal`. - This repository has its own evolved `rust/src/crates/jf` with the same `rsd_journal_*` entry points (spot-checked: `open_files`, `query_unique`, @@ -125,7 +125,7 @@ Artifact impact plan: Open-source reference evidence: -- Netdata repository (`netdata/netdata`) is the consumer; cite `owner/repo @ commit` at activation. +- `netdata/netdata @ eaeb95f5c4`: `packaging/cmake/Modules/NetdataPluginSystemdJournal.cmake:55-57`, `packaging/cmake/Modules/NetdataRust.cmake:61-63`; refresh at activation. Open decisions: diff --git a/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md index d502fe03..bcfe2c5c 100644 --- a/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md +++ b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md @@ -99,7 +99,9 @@ Validation plan: Artifact impact plan: - AGENTS.md: likely unaffected. -- Runtime project skills: `project-release` if the crate list changes. +- Runtime project skills: `project-release` crate list and publish order gain + SOW-0162's crate (updated by SOW-0162, verified here with `RELEASING.md` and + the dry-run). - Specs: product scope version references. - End-user/operator docs: install examples and migration notes. - End-user/operator skills: none exist. @@ -108,7 +110,7 @@ Artifact impact plan: Open-source reference evidence: -- None needed. +- `netdata/netdata @ eaeb95f5c4`: `src/go/go.mod:70` (Go `v0.10.0`), `src/crates/Cargo.toml:176-182` (Rust `0.8.1`). Open decisions: diff --git a/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md index eb03e482..d98a5d56 100644 --- a/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md +++ b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md @@ -29,16 +29,16 @@ asked to port the legacy logs viewer to the SDK. Facts (Netdata master, `netdata/netdata @ eaeb95f5c4`, 2026-10-31): - `systemd-journal.plugin` uses vendored `src/crates/jf` (`journal_reader_ffi`) - in Docker and static builds (`Dockerfile:42`, + in Docker and static builds (`packaging/docker/Dockerfile:42`, `packaging/makeself/jobs/70-netdata-git.install.sh:40`); other builds link libsystemd. Netdata documents a broken no-libsystemd fallback - (`NetdataPluginSystemdJournal.cmake:16-22`). + (`packaging/cmake/Modules/NetdataPluginSystemdJournal.cmake:16-22`). - `otel-legacy-logs` (read-only viewer of the former OTel plugin's journal - files, compiled into `otel-plugin`, `otel-legacy-logs/src/lib.rs:1-10`) uses + files, compiled into `otel-plugin`, `src/crates/otel-legacy-logs/src/lib.rs:1-10`) uses vendored `journal-function` and `journal-index`; `journal-function` uses vendored `journal-core`, `journal-engine`, `journal-index`, `journal-registry`. - `ng-ingest` (experiment) uses vendored `journal-common` only for - `load_machine_id` / `load_boot_id`. + `load_machine_id` / `load_boot_id` (`src/crates/ng-ingest/src/main.rs:148,155`). - Vendored `journal-log-writer` has no consumer. - `netflow-plugin` already uses the published SDK at `0.8.1`; Go consumers use `v0.10.0`. @@ -127,7 +127,7 @@ Artifact impact plan: Open-source reference evidence: -- `netdata/netdata @ eaeb95f5c4` paths as listed above. +- `netdata/netdata @ eaeb95f5c4`: the repository-relative `path:line` citations under Facts. Open decisions: diff --git a/AGENTS.md b/AGENTS.md index 53ac849e..6c5a7430 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,9 +14,10 @@ Pure Rust and Go SDKs for reading and writing systemd journal files, plus file-backed journalctl rewrites in both languages. Success means: -- Read and write valid journal files without CGO or system journal libraries. -- Files written by either language are readable by the other and by systemd - tooling. +- The SDKs read and write valid journal files without CGO or system journal + libraries (CLI-only native code: see Project-specific overrides). +- Files written by either language are readable by the other and by compatible + systemd tooling where applicable. - One shared conformance suite, fixture set, interoperability tests and benchmarks apply to both implementations. - APIs use the format's native performance features (see Performance Contract). @@ -204,11 +205,13 @@ SOWs live in `.agents/sow/{pending,current,done}/` (empty directories keep a The successful terminal status is `completed`; `done` is only a directory name, so never write `Status: done` or `Status: complete`. -- Close a SOW only after its implementation, docs, specs, skills, validation - and follow-ups are finished and `.agents/sow/audit.sh` reports +- Mark a SOW `completed` only after its implementation, docs, specs, skills, + validation and follow-ups are finished and `.agents/sow/audit.sh` reports `SOW initialization complete and clean`. The audit can exit zero while reporting partial state, so read the verdict. If it fails, record and repair it before closing or advancing. +- Mark an obsolete or superseded SOW `closed` with its reason and its + successor or disposition recorded; the audit must still be clean. - Commit the work, artifact updates, status change and move to `done/` together in one commit unless the user asks otherwise, so a completed SOW never lives apart from its implementation. Prefer committing each verified @@ -248,8 +251,9 @@ validation. Never prepend; the original narrative stays readable first. A SOW is complete when its Validation section lets a reader trust the outcome: acceptance criteria met with evidence, tests run and their results, real-use evidence where a runnable path exists, self-review findings and dispositions, -a search for the same failure elsewhere, the sensitive-data check, and what -happened to follow-ups. +a search for the same failure elsewhere, the sensitive-data check, what +happened to follow-ups, and every other record the template's `## Validation` +section lists. ### Artifact Maintenance Gate From 334a54a764ed29124f0a3e3a251ec6d5359f6736 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:08:34 +0000 Subject: [PATCH 4/8] Point stale SOW follow-ups at SOW-0164, narrow verifier coverage claim, track oracle pinning - SOW-0047 outcome and follow-ups name SOW-0164 instead of closed SOWs. - SOW-0049 citations pinned to a Netdata revision, with the static build path. - SOW-0163 marks SOW-0162's crate as unverified until it lands. - SOW-0066 cites the renamed project-release skill; SOW-0121 typo fixed. - Compat skill verifier row: say DATA hash-table bucket instead of hash-chain, and note that the stock reader comes from PATH. - New pending SOW-0169 tracks pinning the stock oracle across the matrices. --- .../project-journal-compatibility/SKILL.md | 2 +- .agents/sow/SOW-status.md | 2 + ...0260528-netdata-netflow-sdk-integration.md | 27 +-- ...8-netdata-reader-plugin-sdk-integration.md | 2 +- ...0621-file-backed-journalctl-full-parity.md | 2 +- ...530-v1-release-and-registry-publication.md | 2 +- .../SOW-0163-20261031-v0-11-0-release.md | 5 +- ...9-20261101-pin-stock-oracle-in-matrices.md | 171 ++++++++++++++++++ 8 files changed, 194 insertions(+), 19 deletions(-) create mode 100644 .agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md diff --git a/.agents/skills/project-journal-compatibility/SKILL.md b/.agents/skills/project-journal-compatibility/SKILL.md index adefdc43..129b299e 100644 --- a/.agents/skills/project-journal-compatibility/SKILL.md +++ b/.agents/skills/project-journal-compatibility/SKILL.md @@ -116,7 +116,7 @@ SDKs; one language's tests never stand in for the other's. | Directory reading / `--directory` | `tests/interoperability/run_directory_matrix.py`, `tests/interoperability/run_mixed_directory_matrix.py` | | journalctl query / follow | `tests/interoperability/run_journalctl_query_matrix.py` | | journalctl options / parser | `tests/parser-parity/check_v260_manifest.py`, `tests/parser-parity/run_parser_parity.py` | -| Verifier | `tests/interoperability/run_verify_matrix.py` (stock `journalctl --verify --file`, Rust and Go agree on regular, zstd/xz/lz4, compact, compact+compression and sealed files, and on object type/size, payload hash, hash-chain, entry-array, header-counter, seqnum, monotonic and TAG/FSS corruption) | +| Verifier | `tests/interoperability/run_verify_matrix.py` (stock `journalctl --verify --file` from `PATH`, version recorded in the report, oracle pinning tracked by SOW-0169; Rust and Go agree on regular, zstd/xz/lz4, compact, compact+compression and sealed files, and on object type/size, payload hash, DATA hash-table bucket, entry-array, header-counter, seqnum, monotonic and TAG/FSS corruption) | | Traversal, compressed readers, recovery | `tests/interoperability/run_reader_hardening.py` with both adapters built and an explicit stock oracle | | Reader performance | `tests/benchmarks/run_reader_regression_comparison.py` | diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index a621b6ff..a2d57d8a 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -64,6 +64,8 @@ order; the user may reorder it. - SOW-0163 - v0.11.0 Release: new, 2026-10-31. - SOW-0164 - Netdata SDK-Exclusive Adoption And Vendored Journal Removal: new, 2026-10-31; supersedes SOW-0049 and SOW-0050. +- SOW-0169 - Pin The Stock journalctl Oracle In Interoperability Matrices: new, + 2026-11-01; needs a decision at activation. ## Recently Closed Or Completed diff --git a/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md b/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md index 20ba607a..a83c5470 100644 --- a/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md +++ b/.agents/sow/done/SOW-0047-20260528-netdata-netflow-sdk-integration.md @@ -63,7 +63,8 @@ Current state: - NetFlow has been migrated to the published SDK crates at `0.7.0`. - The migration is scoped to `netflow-plugin`; other Netdata journal consumers - remain tracked by SOW-0048, SOW-0049, and SOW-0050. + were tracked at close by SOW-0048, SOW-0049, and SOW-0050 (now SOW-0164; see + Follow-up mapping). Risks: @@ -266,7 +267,8 @@ Same-failure scan: - `rg` over `src/crates/netflow-plugin` and `src/crates/Cargo.lock` found the expected SDK aliases and no contradictory NetFlow journal dependency path for the scoped binary. The scan also showed other Netdata journal consumers are - still separate work, as already tracked by SOW-0048, SOW-0049, and SOW-0050. + still separate work, tracked at close by SOW-0048, SOW-0049, and SOW-0050 + (now SOW-0164; see Follow-up mapping). Sensitive data gate: @@ -315,13 +317,12 @@ Lessons: Follow-up mapping: -- SOW-0048 remains open for Netdata OTEL writer integration. -- SOW-0049 remains open for Netdata reader plugin and static packaging - integration. -- SOW-0050 remains open for final vendored journal code removal after all - component integrations are complete. -- 2026-10-31: SOW-0048 closed as obsolete; SOW-0049 and SOW-0050 superseded by - SOW-0164. +- At close, SOW-0048 (Netdata OTEL writer integration), SOW-0049 (Netdata + reader plugin and static packaging integration) and SOW-0050 (final vendored + journal code removal) held the remaining Netdata work. +- 2026-10-31: SOW-0048 closed as obsolete; SOW-0049 and SOW-0050 closed as + superseded by SOW-0164, which now tracks all remaining Netdata integration + work. ## Outcome @@ -330,7 +331,8 @@ Completed. NetFlow is integrated with the published SDK crates at `0.7.0` for both writer and reader/query paths in the inspected Netdata checkout. This closes the SDK ledger item for NetFlow integration. The remaining Netdata integration work is -still tracked separately by SOW-0048, SOW-0049, and SOW-0050. +tracked separately by SOW-0164 (successor of SOW-0048, SOW-0049, and +SOW-0050). ## Lessons Extracted @@ -339,9 +341,8 @@ still tracked separately by SOW-0048, SOW-0049, and SOW-0050. ## Followup -- SOW-0048 - Netdata OTEL Writer SDK Integration. -- SOW-0049 - Netdata Reader Plugin SDK Integration. -- SOW-0050 - Netdata Vendored Journal Removal. +- SOW-0164 - remaining Netdata SDK integration (successor of SOW-0048, + SOW-0049 and SOW-0050, all closed). ## Regression Log diff --git a/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md index 7dcadee4..2b020e64 100644 --- a/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md +++ b/.agents/sow/done/SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md @@ -242,7 +242,7 @@ Follow-up mapping: ## Outcome -Closed without implementation; scope moved to SOW-0164. Current Netdata reader consumers of vendored code: `systemd-journal.plugin` via `src/crates/jf` (`journal_reader_ffi`, `packaging/cmake/Modules/NetdataRust.cmake:61-63`; Docker/static builds with `--internal-systemd-journal`, `packaging/docker/Dockerfile:42`) and `otel-legacy-logs` via `journal-function`/`journal-index` (`src/crates/otel-legacy-logs/Cargo.toml:19-20`). The SDK-side publishable C reader crate is SOW-0162. +Closed without implementation; scope moved to SOW-0164. Current Netdata reader consumers of vendored code (`netdata/netdata @ eaeb95f5c4`): `systemd-journal.plugin` via `src/crates/jf` (`journal_reader_ffi`, `packaging/cmake/Modules/NetdataRust.cmake:61-63`; Docker/static builds with `--internal-systemd-journal`, `packaging/docker/Dockerfile:42` and `packaging/makeself/jobs/70-netdata-git.install.sh:40`) and `otel-legacy-logs` via `journal-function`/`journal-index` (`src/crates/otel-legacy-logs/Cargo.toml:19-20`). The SDK-side publishable C reader crate is SOW-0162. ## Lessons Extracted diff --git a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md index d8f8129a..d9fa3147 100644 --- a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md +++ b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md @@ -8,7 +8,7 @@ Status: completed Sub-state: PR12 review regressions completed locally 2026-10-26. Both CLIs use the approved directional SDK APIs. Three internal final scopes pass after -3,514 native comparisons and146 portable controls. PR #12 merged on +3,514 native comparisons and 146 portable controls. PR #12 merged on 2026-10-27 (00d33d4); release is tracked by SOW-0163. ## Requirements diff --git a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md index a0c00c23..01d4eaad 100644 --- a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md +++ b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md @@ -89,7 +89,7 @@ Sources checked: - User request in this thread. - Existing release tagging project skill: - `.agents/skills/project-release-tagging/SKILL.md`. + `.agents/skills/project-release/SKILL.md`. - Existing product package roots: `rust/` and `go/`. - Existing project SOW/status dependency graph. diff --git a/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md index bcfe2c5c..23eebbbe 100644 --- a/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md +++ b/.agents/sow/pending/SOW-0163-20261031-v0-11-0-release.md @@ -100,8 +100,9 @@ Artifact impact plan: - AGENTS.md: likely unaffected. - Runtime project skills: `project-release` crate list and publish order gain - SOW-0162's crate (updated by SOW-0162, verified here with `RELEASING.md` and - the dry-run). + SOW-0162's crate once SOW-0162 lands (SOW-0162 is not yet activated). This + SOW must verify that crate with `RELEASING.md` and the release dry-run; until + then the crate is unverified. - Specs: product scope version references. - End-user/operator docs: install examples and migration notes. - End-user/operator skills: none exist. diff --git a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md new file mode 100644 index 00000000..67a44bbe --- /dev/null +++ b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md @@ -0,0 +1,171 @@ +# SOW-0169 - Pin The Stock journalctl Oracle In Interoperability Matrices + +## Status + +Status: open + +Sub-state: new, 2026-11-01; raised by a PR #13 review thread; not activated. + +## Requirements + +### Purpose + +Compatibility gates must prove agreement with the systemd release the project +targets, not with whatever `journalctl` the host happens to have. + +### User Request + +Tracked from a PR #13 review finding: the verifier gate runs the bare host +`journalctl`, so it can certify against a different systemd version than the +pinned oracle. No user request yet; activation needs user scheduling. + +### Assistant Understanding + +Facts: + +- `tests/interoperability/run_verify_matrix.py` runs `["journalctl", ...]` from + `PATH` for the stock reader (`systemd_version()` and `verify_command()`), and + records only that binary's `--version` line in its report. +- The same bare invocation appears in `run_matrix.py`, `run_binary_matrix.py`, + `run_live_matrix.py`, `run_compression_matrix.py`, `run_compact_matrix.py`, + `run_lock_matrix.py`, `run_directory_matrix.py`, + `run_mixed_directory_matrix.py`, `run_journalctl_portable_smoke.py` and + `journalctl_v262_cases.py`. +- `run_reader_hardening.py` already requires `--journalctl PATH`, and + `run_journalctl_query_matrix.py` accepts one; these are the patterns to reuse. +- The parser-parity helpers (`tests/parser-parity/`) compare against pinned + systemd source, not a stock binary, so they are out of scope. + +Inferences: + +- A report from a host with a different systemd version is still useful, but it + is not evidence for the pinned target. + +Unknowns: + +- Whether every matrix should require the flag, or default to `PATH` and mark + the report as unpinned. This is decision 1 at activation. + +### Acceptance Criteria + +- Every stock-invoking matrix runs the selected oracle for all stock calls, + including version reporting. +- Reports record the oracle path and version. +- The compat skill's test-tool table states how to pass the oracle. + +## Analysis + +Sources checked: + +- The scripts listed above (`rg '\["journalctl"' tests/interoperability`). + +Current state: + +- Bare host `journalctl` in eleven scripts; explicit oracle in two. + +Risks: + +- Low: test tooling only. Making the flag required changes how the matrices + are invoked locally and in any CI job that calls them. + +## Pre-Implementation Gate + +Status: needs-user-decision + +Problem / root-cause model: + +- The matrices predate the pinned-oracle rule and resolve `journalctl` from + `PATH`. + +Evidence reviewed: + +- See Assistant Understanding. + +Affected contracts and surfaces: + +- Interoperability test scripts and their reports; the + `project-journal-compatibility` skill test-tool table. No SDK or CLI code. + +Existing patterns to reuse: + +- `run_reader_hardening.py` `--journalctl` argument. + +Risk and blast radius: + +- Test tooling only. + +Sensitive data handling plan: + +- Reports record binary paths under the repository; no sensitive data. + +Implementation plan: + +1. Add a shared oracle argument and route every stock call through it. +2. Update the skill table. + +Validation plan: + +- Run each changed matrix against the v262 oracle build. + +Artifact impact plan: + +- AGENTS.md: unaffected. +- Runtime project skills: `project-journal-compatibility` test-tool table. +- Specs: unaffected. +- End-user/operator docs: unaffected. +- End-user/operator skills: none exist. +- SOW lifecycle: single SOW. +- SOW-status.md: updated on activation and close. + +Open-source reference evidence: + +- None needed; this concerns repository test tooling. + +Open decisions: + +1. Required flag versus `PATH` default with an "unpinned" report marker. + +## Implications And Decisions + +Pending activation. + +## Plan + +Pending activation. + +## Implementation And Review Plan + +Implementation: + +- Project manager, at activation. + +Reviewers: + +- Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used. + +## Execution Log + +### 2026-11-01 + +- Created from a PR #13 review thread. + +## Validation + +Pending implementation. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. From add616ba1ac6891f659aa8b30ba4a76c0172042c Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:14:55 +0000 Subject: [PATCH 5/8] Correct the SOW-0169 script inventory and list it in the backlog order --- .agents/sow/SOW-status.md | 4 +++- ...169-20261101-pin-stock-oracle-in-matrices.md | 17 +++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index a2d57d8a..26f455c4 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -1,6 +1,6 @@ # SOW Status -Last updated: 2026-10-31 +Last updated: 2026-11-01 ## Backlog Order @@ -25,6 +25,8 @@ order; the user may reorder it. 14. SOW-0163 - v0.11.0 release. 15. SOW-0164 - Netdata SDK-exclusive adoption and vendored journal removal. 16. SOW-0066 - v1.0.0 API stability decision. +17. SOW-0169 - Pin the stock journalctl oracle in interoperability matrices + (needs a decision at activation). ## Current diff --git a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md index 67a44bbe..f3ef7911 100644 --- a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md +++ b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md @@ -28,11 +28,14 @@ Facts: records only that binary's `--version` line in its report. - The same bare invocation appears in `run_matrix.py`, `run_binary_matrix.py`, `run_live_matrix.py`, `run_compression_matrix.py`, `run_compact_matrix.py`, - `run_lock_matrix.py`, `run_directory_matrix.py`, - `run_mixed_directory_matrix.py`, `run_journalctl_portable_smoke.py` and - `journalctl_v262_cases.py`. + `run_lock_matrix.py`, `run_directory_matrix.py` and + `run_mixed_directory_matrix.py` (nine scripts in total, counting + `run_verify_matrix.py`). - `run_reader_hardening.py` already requires `--journalctl PATH`, and - `run_journalctl_query_matrix.py` accepts one; these are the patterns to reuse. + `run_journalctl_query_matrix.py` accepts one and passes it to + `journalctl_v262_cases.py` as `tools["stock_journalctl"]`; these are the + patterns to reuse. `run_journalctl_portable_smoke.py` runs only the Go and + Rust binaries (its `"journalctl"` strings are expected command names). - The parser-parity helpers (`tests/parser-parity/`) compare against pinned systemd source, not a stock binary, so they are out of scope. @@ -57,11 +60,13 @@ Unknowns: Sources checked: -- The scripts listed above (`rg '\["journalctl"' tests/interoperability`). +- The scripts listed above (`rg '\["journalctl"' tests/interoperability`, + with each hit checked for whether it launches a stock binary). Current state: -- Bare host `journalctl` in eleven scripts; explicit oracle in two. +- Bare host `journalctl` in nine scripts; explicit oracle in two plus the + shared v262 case module. Risks: From 860bb2b908f1c42d4607276055db2fcd019b9d09 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Sun, 1 Nov 2026 21:19:14 +0000 Subject: [PATCH 6/8] Cite revision and line-level evidence in SOW-0169 --- ...9-20261101-pin-stock-oracle-in-matrices.md | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md index f3ef7911..db6d5815 100644 --- a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md +++ b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md @@ -31,8 +31,8 @@ Facts: `run_lock_matrix.py`, `run_directory_matrix.py` and `run_mixed_directory_matrix.py` (nine scripts in total, counting `run_verify_matrix.py`). -- `run_reader_hardening.py` already requires `--journalctl PATH`, and - `run_journalctl_query_matrix.py` accepts one and passes it to +- `run_reader_hardening.py` and `run_journalctl_query_matrix.py` already + require `--journalctl PATH`; the query matrix passes it to `journalctl_v262_cases.py` as `tools["stock_journalctl"]`; these are the patterns to reuse. `run_journalctl_portable_smoke.py` runs only the Go and Rust binaries (its `"journalctl"` strings are expected command names). @@ -82,9 +82,20 @@ Problem / root-cause model: - The matrices predate the pinned-oracle rule and resolve `journalctl` from `PATH`. -Evidence reviewed: - -- See Assistant Understanding. +Evidence reviewed (`netdata/systemd-journal-sdk @ 334a54a764ed29124f0a3e3a251ec6d5359f6736`): + +- Bare `["journalctl", ...]` launches in `tests/interoperability/`: + `run_verify_matrix.py:186,396`, `run_matrix.py:78,182,258`, + `run_binary_matrix.py:91,267,278,295,303,467,477`, + `run_live_matrix.py:183,353,749`, + `run_compression_matrix.py:104,307,318,323,331,464,475,486`, + `run_compact_matrix.py:339,350,355,363,483,494,512`, + `run_lock_matrix.py:87`, `run_directory_matrix.py:135,296,320`, + `run_mixed_directory_matrix.py:202,369,393`. +- Not launches (expected command names): `run_journalctl_portable_smoke.py:131`, + `journalctl_v262_cases.py:385`. +- Explicit oracle: `run_reader_hardening.py:177`, + `run_journalctl_query_matrix.py:2508`, `journalctl_v262_cases.py:40`. Affected contracts and surfaces: From ffbc84918ab39424bcd7ac021f4c18a1b6e3387b Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Tue, 3 Nov 2026 06:59:43 +0000 Subject: [PATCH 7/8] Record user decisions for SOW-0164, SOW-0066 and SOW-0169; track the plugin rewrite as SOW-0170 - SOW-0164: libsystemd stays where available; the SDK replaces the vendored reader with verified feature parity; mixed-format gate carried forward. - SOW-0066: v1.0.0 not scheduled. - SOW-0169: required --journalctl PATH; runs before the v0.11.0 release. - SOW-0170: Rust rewrite of systemd-journal.plugin, tracked, not now. --- .agents/sow/SOW-status.md | 34 ++-- ...530-v1-release-and-registry-publication.md | 15 +- ...20261031-netdata-sdk-exclusive-adoption.md | 26 ++- ...9-20261101-pin-stock-oracle-in-matrices.md | 10 +- ...ata-systemd-journal-plugin-rust-rewrite.md | 162 ++++++++++++++++++ 5 files changed, 221 insertions(+), 26 deletions(-) create mode 100644 .agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 26f455c4..f187091c 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -1,6 +1,6 @@ # SOW Status -Last updated: 2026-11-01 +Last updated: 2026-11-03 ## Backlog Order @@ -8,8 +8,8 @@ The user decided on 2026-10-31 to fix all open SOWs rather than close valid-but-low-impact leads. Execution is one SOW at a time in this proposed order; the user may reorder it. -1. SOW-0045 - Go reader regressions (refresh skips first append; directory - seek getters; swallowed time-search errors). +1. SOW-0045 - Go reader regressions (refresh skips first append; directory seek + getters; swallowed time-search errors). 2. SOW-0071 - Runtime purity guard misses Go process-launch variants. 3. SOW-0123 - Rust fresh-journal realtime clock seed (event-time lanes). 4. SOW-0137 - Rust writer residual array-open performance. @@ -22,11 +22,14 @@ order; the user may reorder it. 11. SOW-0153 - Consumer docs release clarity. 12. SOW-0125 - Netdata filter operators (needs design decisions first). 13. SOW-0162 - Publishable C reader crate. -14. SOW-0163 - v0.11.0 release. -15. SOW-0164 - Netdata SDK-exclusive adoption and vendored journal removal. -16. SOW-0066 - v1.0.0 API stability decision. -17. SOW-0169 - Pin the stock journalctl oracle in interoperability matrices - (needs a decision at activation). +14. SOW-0169 - Pin the stock journalctl oracle in interoperability matrices + (decided 2026-11-03: required `--journalctl PATH`). +15. SOW-0163 - v0.11.0 release. +16. SOW-0164 - Netdata SDK-exclusive adoption and vendored journal removal. +17. SOW-0066 - v1.0.0 API freeze and release (not scheduled; user decision + 2026-11-03). +18. SOW-0170 - Rust rewrite of Netdata systemd-journal.plugin (tracked; not + now, by user decision 2026-11-03). ## Current @@ -42,8 +45,8 @@ order; the user may reorder it. ## Pending -- SOW-0066 - V1 Release: now the v1.0.0 API stability decision; Netdata gate - is SOW-0164. +- SOW-0066 - V1 Release: v1.0.0 API freeze and release; not scheduled (user + decision 2026-11-03). Netdata gate is SOW-0164. - SOW-0097 - Go Codacy Metric Debt Refactor: real duplicated helpers; cited line numbers need refreshing at activation. - SOW-0098 - Rust Legacy Core Duplication Debt: real `jf` duplication; feeds @@ -64,10 +67,13 @@ order; the user may reorder it. - SOW-0161 - File Indexer Source Posting Membership: unproven indexer lead. - SOW-0162 - Publishable C Reader Crate: new, 2026-10-31. - SOW-0163 - v0.11.0 Release: new, 2026-10-31. -- SOW-0164 - Netdata SDK-Exclusive Adoption And Vendored Journal Removal: new, - 2026-10-31; supersedes SOW-0049 and SOW-0050. -- SOW-0169 - Pin The Stock journalctl Oracle In Interoperability Matrices: new, - 2026-11-01; needs a decision at activation. +- SOW-0164 - Netdata SDK-Exclusive Adoption And Vendored Journal Removal: + supersedes SOW-0049 and SOW-0050. libsystemd stays where available; the SDK + replaces the vendored reader, with verified feature parity (2026-11-03). +- SOW-0169 - Pin The Stock journalctl Oracle In Interoperability Matrices: + decided 2026-11-03 (required `--journalctl PATH`); runs before SOW-0163. +- SOW-0170 - Rust systemd-journal.plugin Rewrite: tracked 2026-11-03; not now, + by user decision. ## Recently Closed Or Completed diff --git a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md index 01d4eaad..285b2f98 100644 --- a/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md +++ b/.agents/sow/pending/SOW-0066-20260530-v1-release-and-registry-publication.md @@ -4,13 +4,13 @@ Status: open -Sub-state: pending final production release and Rust/Go publication -after SDK compatibility, portability, corpus validation, and integration gates -are complete. +Sub-state: not scheduled. The user decided on 2026-11-03 not to release +v1.0.0 now; activation needs a new user decision. 2026-10-31: registry publication and Go module tags already exist through -v0.10.0 and Netdata consumes them, so this SOW is now the v1.0.0 API stability -decision. Its Netdata integration gate is SOW-0164 (SOW-0048 closed as obsolete; +v0.10.0 and Netdata consumes them. The remaining deliverable is the v1.0.0 API +freeze decision followed by publication of v1.0.0 through the existing +release process (`RELEASING.md`, `project-release` skill). Its Netdata integration gate is SOW-0164 (SOW-0048 closed as obsolete; SOW-0049 and SOW-0050 superseded by SOW-0164). The compatibility, portability and corpus gates (SOW-0055, SOW-0063, SOW-0064, SOW-0065) are already done. @@ -220,6 +220,11 @@ Open decisions: ## Implications And Decisions +0. 2026-11-03 scope and timing (user) + - Decision: no v1.0.0 release now. When activated, this SOW decides the + v1.0.0 API freeze and publishes v1.0.0 through the existing release + process; first-time registry setup is already done. + 1. 2026-05-30 final release goal - Decision: record a future `v1.0.0` release and registry publication SOW. - Implication: consumers should eventually depend on published packages or diff --git a/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md index d98a5d56..8d5e14bf 100644 --- a/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md +++ b/.agents/sow/pending/SOW-0164-20261031-netdata-sdk-exclusive-adoption.md @@ -15,8 +15,10 @@ authorization of that target at activation. ### Purpose -Netdata uses this SDK exclusively for journal files; every vendored or local -journal implementation in Netdata is removed. +Netdata uses this SDK for journal files wherever it does not use libsystemd, +and every vendored or local journal implementation in Netdata is removed. +Builds that have libsystemd keep using it during a transition period (user +decision, 2026-11-03). ### User Request @@ -54,7 +56,18 @@ Unknowns: ### Acceptance Criteria -- `systemd-journal.plugin` uses the SDK's published C reader crate (SOW-0162). +- `systemd-journal.plugin` keeps libsystemd in builds where libsystemd is + available, and uses the SDK's published C reader crate (SOW-0162) instead of + vendored `jf` in builds without it (today Docker and static builds, + `packaging/docker/Dockerfile:42`, + `packaging/makeself/jobs/70-netdata-git.install.sh:40`). +- Feature parity between the libsystemd and SDK reader paths is verified on + the same journal inputs (queries, facets, histograms, boots, cursors and + directory reading), with every difference fixed or recorded with a user + decision. +- A mixed-format journal directory (regular, compact and compressed files, + carried over from superseded SOW-0049) is read correctly through the SDK + reader path, including ordered directory reading. - `otel-legacy-logs` / `journal-function` and `ng-ingest` use SDK crates. - `netflow-plugin` and Go consumers move to the new release. - Netdata's `src/crates/{jf,journal-common,journal-core,journal-index,journal-engine,journal-registry,journal-log-writer}` @@ -137,6 +150,13 @@ Open decisions: - 2026-10-31: user chose one SOW for adoption plus removal, a published C reader crate (SOW-0162), and porting the legacy logs viewer to the SDK. +- 2026-11-03 (user): keep libsystemd where it is available; this may be a + transition period. The SDK replaces only the vendored reader. Feature parity + must be verified. Rewriting `systemd-journal.plugin` in Rust, so it uses the + SDK without FFI, is the right long-term solution but not now; it is tracked + by pending SOW-0170. +- 2026-11-03 (user): carry SOW-0049's mixed-format directory gate forward as + an acceptance criterion. ## Plan diff --git a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md index db6d5815..8507621c 100644 --- a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md +++ b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md @@ -4,7 +4,7 @@ Status: open -Sub-state: new, 2026-11-01; raised by a PR #13 review thread; not activated. +Sub-state: decided 2026-11-03 (required `--journalctl PATH`); not activated. ## Requirements @@ -75,7 +75,7 @@ Risks: ## Pre-Implementation Gate -Status: needs-user-decision +Status: ready Problem / root-cause model: @@ -139,11 +139,13 @@ Open-source reference evidence: Open decisions: -1. Required flag versus `PATH` default with an "unpinned" report marker. +- None. Decision 1 resolved 2026-11-03 (see Implications And Decisions). ## Implications And Decisions -Pending activation. +1. 2026-11-03 (user): every stock-invoking matrix requires `--journalctl PATH`; + there is no `PATH` fallback. Run before SOW-0163 (v0.11.0 release), so the + release gates use the pinned oracle. ## Plan diff --git a/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md b/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md new file mode 100644 index 00000000..5f33a8b9 --- /dev/null +++ b/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md @@ -0,0 +1,162 @@ +# SOW-0170 - Rust Rewrite Of Netdata systemd-journal.plugin + +## Status + +Status: open + +Sub-state: tracked 2026-11-03 by user decision; not now. Activation needs a new +user decision and explicit authorization to change the Netdata repository. + +## Requirements + +### Purpose + +Netdata's `systemd-journal.plugin` uses the SDK natively, without FFI, so the +SDK's reader, index and facet paths serve the system logs viewer directly. + +### User Request + +"The right solution is to rewrite the systemd-journal.plugin in rust for that +it is natural without FFI. but not yet please." (2026-11-03, while deciding +SOW-0164.) + +### Assistant Understanding + +Facts: + +- `systemd-journal.plugin` is C. It reads journals through libsystemd, or + through vendored `jf` (`journal_reader_ffi`) in Docker and static builds + (`netdata/netdata @ eaeb95f5c4`, `packaging/docker/Dockerfile:42`, + `packaging/makeself/jobs/70-netdata-git.install.sh:40`). +- SOW-0164 keeps libsystemd where available and replaces vendored `jf` with + the SDK's published C reader crate (SOW-0162); that is the transition step. + +Inferences: + +- A Rust plugin removes the C/Rust FFI boundary and can use the SDK's + indexed query paths instead of a libsystemd-shaped facade. + +Unknowns: + +- Plugin feature inventory, the Netdata function protocol surface it must + keep, and whether libsystemd-only features (for example namespaces or the + live system journal) must stay. + +### Acceptance Criteria + +- To be written at activation. + +## Analysis + +Sources checked: + +- SOW-0164 facts and the Netdata citations above. + +Current state: + +- Not started. + +Risks: + +- Large cross-repository change to a user-facing Netdata feature. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- Not analyzed yet; not scheduled. + +Evidence reviewed: + +- See Assistant Understanding. + +Affected contracts and surfaces: + +- Netdata `systemd-journal.plugin`, its function protocol and packaging; SDK + Rust reader and explorer APIs. + +Existing patterns to reuse: + +- Netdata `netflow-plugin`, a Rust plugin that already uses published SDK + crates. + +Risk and blast radius: + +- See Risks. + +Sensitive data handling plan: + +- Synthetic journals only in SDK artifacts. + +Implementation plan: + +1. To be written at activation, after SOW-0164. + +Validation plan: + +- To be written at activation. + +Artifact impact plan: + +- AGENTS.md: likely unaffected. +- Runtime project skills: likely unaffected. +- Specs: Netdata integration status in product scope. +- End-user/operator docs: likely unaffected in this repository. +- End-user/operator skills: none exist. +- SOW lifecycle: follows SOW-0164. +- SOW-status.md: updated on activation and close. + +Open-source reference evidence: + +- `netdata/netdata @ eaeb95f5c4`: paths cited under Facts. + +Open decisions: + +1. When to schedule it, and Netdata repository write authorization. + +## Implications And Decisions + +1. 2026-11-03 (user): track it; do not do it yet. + +## Plan + +Pending activation. + +## Implementation And Review Plan + +Implementation: + +- Project manager, at activation. + +Reviewers: + +- Whole-SOW self-review after local validation; tests are the gate. External + reviewers are not used. + +## Execution Log + +### 2026-11-03 + +- Created from the SOW-0164 decision. + +## Validation + +Pending implementation. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. From 7b1e1acb6bfc571bb2c219cefde86f65e234e602 Mon Sep 17 00:00:00 2001 From: Costa Tsaousis Date: Tue, 3 Nov 2026 07:18:34 +0000 Subject: [PATCH 8/8] Complete the SOW-0169 oracle inventory and the SOW-0170 tracking record --- ...9-20261101-pin-stock-oracle-in-matrices.md | 14 ++++---- ...ata-systemd-journal-plugin-rust-rewrite.md | 33 ++++++++++++++++--- 2 files changed, 37 insertions(+), 10 deletions(-) diff --git a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md index 8507621c..d6ab9247 100644 --- a/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md +++ b/.agents/sow/pending/SOW-0169-20261101-pin-stock-oracle-in-matrices.md @@ -33,8 +33,8 @@ Facts: `run_verify_matrix.py`). - `run_reader_hardening.py` and `run_journalctl_query_matrix.py` already require `--journalctl PATH`; the query matrix passes it to - `journalctl_v262_cases.py` as `tools["stock_journalctl"]`; these are the - patterns to reuse. `run_journalctl_portable_smoke.py` runs only the Go and + `journalctl_v262_cases.py` and `journalctl_timestamp_cases.py` as + `tools["stock_journalctl"]`; these are the patterns to reuse. `run_journalctl_portable_smoke.py` runs only the Go and Rust binaries (its `"journalctl"` strings are expected command names). - The parser-parity helpers (`tests/parser-parity/`) compare against pinned systemd source, not a stock binary, so they are out of scope. @@ -46,8 +46,8 @@ Inferences: Unknowns: -- Whether every matrix should require the flag, or default to `PATH` and mark - the report as unpinned. This is decision 1 at activation. +- None. The oracle policy is decision 1 (Implications And Decisions): the flag + is required, with no `PATH` fallback. ### Acceptance Criteria @@ -66,7 +66,7 @@ Sources checked: Current state: - Bare host `journalctl` in nine scripts; explicit oracle in two plus the - shared v262 case module. + shared v262 and timestamp case modules. Risks: @@ -95,7 +95,9 @@ Evidence reviewed (`netdata/systemd-journal-sdk @ 334a54a764ed29124f0a3e3a251ec6 - Not launches (expected command names): `run_journalctl_portable_smoke.py:131`, `journalctl_v262_cases.py:385`. - Explicit oracle: `run_reader_hardening.py:177`, - `run_journalctl_query_matrix.py:2508`, `journalctl_v262_cases.py:40`. + `run_journalctl_query_matrix.py:2508`, `journalctl_v262_cases.py:40`, and + `journalctl_timestamp_cases.py:65` (assignment; launches at 68, 141, 203), + which the query matrix also feeds `tools["stock_journalctl"]`. Affected contracts and surfaces: diff --git a/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md b/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md index 5f33a8b9..a7d145de 100644 --- a/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md +++ b/.agents/sow/pending/SOW-0170-20261103-netdata-systemd-journal-plugin-rust-rewrite.md @@ -11,8 +11,9 @@ user decision and explicit authorization to change the Netdata repository. ### Purpose -Netdata's `systemd-journal.plugin` uses the SDK natively, without FFI, so the -SDK's reader, index and facet paths serve the system logs viewer directly. +Planned: rewrite Netdata's `systemd-journal.plugin` in Rust so it uses the +SDK natively, without FFI, and the SDK's reader, index and facet paths serve +the system logs viewer directly. ### User Request @@ -28,8 +29,9 @@ Facts: through vendored `jf` (`journal_reader_ffi`) in Docker and static builds (`netdata/netdata @ eaeb95f5c4`, `packaging/docker/Dockerfile:42`, `packaging/makeself/jobs/70-netdata-git.install.sh:40`). -- SOW-0164 keeps libsystemd where available and replaces vendored `jf` with - the SDK's published C reader crate (SOW-0162); that is the transition step. +- SOW-0164 keeps libsystemd where available and, once the planned C reader + crate (SOW-0162, not yet activated) is published, replaces vendored `jf` + with it; that is the transition step. Inferences: @@ -135,6 +137,29 @@ Reviewers: - Whole-SOW self-review after local validation; tests are the gate. External reviewers are not used. +Repository boundary block for every external-agent prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Netdata repository changes need explicit user authorization recorded here at +activation; until then the boundary above applies unchanged. + +Failure handling: + +- Activation without both a scheduling decision and Netdata write + authorization stops and is reported to the user. +- Validation or audit failures are recorded in the Execution Log with + evidence; the SOW does not advance until they are fixed or a user decision + is recorded. + ## Execution Log ### 2026-11-03