Skip to content

Integrate native Git publication and collaboration with resident owner routing - #36

Merged
forhappy merged 56 commits into
mainfrom
codex/native-metadata-replay
Oct 6, 2026
Merged

forhappy merged 56 commits into
mainfrom
codex/native-metadata-replay

Conversation

@forhappy

@forhappy forhappy commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Repository reads and generated writes still used SQL object/ref metadata removed by the packed-catalog cutover. This follow-up integrates native catalog reads and owner-fenced publication into HTTP/SSH pushes, checks, pull reviews, candidate preparation, and reviewed merges. Symbolic HEAD and the existing constant-size ref summary publish atomically with certified native roots.

Generated candidates now run under resident staging ownership. Concurrent observations join the frozen candidate intent; Git emits a non-thin pack of only request-private generated objects. Independent physical verification and private commit-semantic verification precede atomic candidate result/reserved-ref/catalog publication (operation 55, codec 1). Generated reviewed merges select the immutable Ready candidate, validate its native audit and reserved ref, and recheck the current pull revision, reviews, checks, branch policy, authorization and joint roots (operation 9, codec 8). The existing candidate and merge rows retain the permanent typed audit; exact registered command recovery preserves the original SDK receipt. No legacy object mirror is rebuilt.

Line-thread creation now binds the verified hunk anchor and complete editorial intent to native ref facts before the owner transaction rechecks the pull revision and current access (operation 56, codec 1). Remote Git/LFS HTTP streams go to the verified current resident owner, which independently authenticates the original credential; bodies stream through existing admission, with bounded forwarding and no automatic retry of consumed receive-pack bodies. Final catalog CAS denial now selects the same pre-frozen refusal from an authenticated two-command recovery bundle, retaining exact SDK identities, phase order, and receipts. Reused push UUIDs from a different actor return the existing HTTP 409 conflict.

Backup now traverses the authenticated native catalog/ref/input/outcome/audit/recovery graph from the pinned Cell snapshot. It pages SQL roots, uses bounded traversal and disk-admitted physical deduplication, copies original creating namespaces, and verifies source and destination parts. Closed command records retain selected permanent roots without demanding retired command bodies or impossible future refusals. LFS remains paged and verified. The cold residency test now compares authenticated table snapshots: all repository/product rows, staging intents and existing receipts remain exact, while serving pins/custody and runtime sequence/time may advance. This replaces a physical-root equality assertion incompatible with durable serving custody. All six repos restore and pass clone/fsck/LFS checks locally.

The production push pipeline also waits for the controller to restore Active/Bound after an input checkpoint, while preserving the original receipt independently; a gated regression reproduced the old receipt-before-readiness race for both object formats.

Cold fetches now use authenticated provider-part reads, a private sparse pack for native Git decoding, and streaming canonical verification into bounded service caches. Request workspaces materialize the selected structural history and Git filter candidates. Guessed wants remain gated by certified live-ref membership; the native reachable-want repeat check is relaxed only on those validated gateway paths. Both protocol versions, lazy blob wants, tree/type/nested-combine filters, and negative cache assertions pass locally. Size inspection uses a disposable workspace and preserves the candidate cache behavior within combined tree/type selection.

Late SSH access downgrade now returns the original per-ref failure report only for a known inactive terminal attempt with current access below write. It cannot publish or acknowledge success; uncertain mutations retain their recovery/error path. Backup counts are checked against the actual retained manifest inventory rather than a platform-dependent deduplicated count, with explicit LFS and orphan exclusion checks. Standalone fixtures now use leased resident servers instead of detached handles or retired ingestion commands. The atomic fixture covers 600 native objects in both formats; restart destroys the first owner's disk before cloning on a new node. No workflow or integration test target is disabled. Coverage mapping and source evidence are in docs/evidence/native-ci-fixture-migration-20261006.md and native-selective-ci-20261006.json.

Validation: the complete locked local workspace suite at 85abb77 passes: 758 server unit tests, 107 multi-server tests (9 existing ignored), both supporting crates, every standalone target, and doc tests. This includes contended HTTP clone, expiry and exact renewal, paused-pack discovery with unchanged two-second requests, the HTTP/SSH filter matrix and negative cache assertions, backup after original storage deletion, and SSH/SHA-256 recovery. All-target workspace Clippy with warnings denied, formatting, diff checks, and the server build pass. The Python harness passes in GitHub CI. Both complete GitHub workflows pass at 85abb77: formatting, all-target Clippy, the Python harness, the full locked Linux Rust suite (758 server unit tests and 111 multi-server tests), RustFS qualification, and the server build. Verified PR run: https://github.com/crabbuild/canopy/actions/runs/37426775956; push run: https://github.com/crabbuild/canopy/actions/runs/37426771454.

Long fetch/discovery preparation now uses the existing renewable serving-session path, with a fresh final authority check. A gated regression holds a provider read beyond the initial lease and requires two renewals of the same pin. Expiry metadata warms before its unchanged one-second lease; the renewal fixture uses one reachable native blob to isolate lease behavior. No lease duration, resource limit, or negative assertion is relaxed. Discovery streams fully peeled packed refs from certified metadata in bounded pages, checks tag-edge kinds, and uses an admitted atomic replacement; native Git still owns the wire protocol and requires no native pack body for ref listing.

Large-team capacity, final DDL/retention/collection, shared membership acceleration and full workload qualification remain incomplete. This PR is not release qualified.

The SSH publication fixture selected a port by binding and immediately
closing it, then asynchronously initialized a server before binding again.
A competing listener at that real call site reproduces AddrInUse. Retain
the original bound listener and hand it to the existing supervised startup
API for initial startup and every restoration in this fixture family.

Assert that another binder cannot claim the fixture port before handoff.
All original refusal, disconnect and cold-preparation scenarios pass in
the 105-case multi-server rerun. No bind retries or deadline changes.
Reuse the existing root purpose and ETag reservation in a bounded
canopy-pack-v1 envelope. Reject unversioned or unknown remote formats
before workspace reclamation, probes, identity writes or Cell activation.
Reuse the owner/worker fences with the new managed runtime directory;
reject and retain legacy local state. Include the boundary in release
source hashing and update affected fixtures and harness paths.

Local hard-cutover work only: the old production Git schema and command
registry still require conversion with every producer and reader before
this branch can be released. No legacy decoder or migration is added.

Validation: 661 unique workspace Rust cases, 8 isolated RustFS cases,
96 Python harness cases, warnings-denied all-target Clippy, formatting,
doctest completion, and server build. Original red format regressions
and the earlier SSH suite failure are retained separately.
Reuse the bounded typed publication contract for actual Repository Cell migrations, activation and maintenance. Initialize private empty catalog/ref roots before Ready and authenticate retained initialization on cold load without new allocation.

Production Git producer/reader conversion remains an unreleasable local cutover. Qualification reads published Cell roots through the sparse VFS, uses scoped provider keys for corruption injection and releases connection guards before awaits.
Retain the actual accepted Begin receipt in the existing logical request row,
sharing the bounded authenticated record and lookup with staging. Startup
recovers it before another Begin and checks current fence/custody separately.
Explicit Claim can recover the authenticated original after operation reaping
without restoring old custody or displacing a successor.

Update initialization/compaction admission checks, command codecs and source
hashing. Qualify both object formats, SDK expiry, fresh-owner restore, rollback,
immutability and purpose separation. Preserve initial receipts in fixture
state hashes and target fault injection at the actual publication update.

Validation: 271 publication + 3 actual startup tests; all-target workspace
Clippy with warnings denied; canopy binary build; fmt/diff and frozen sources.
Local checkpoint only: remaining producer/reader cutover and full durable
custody-command recovery are required before release.
Persist original SDK snapshots before Begin and retain positive and denied
custody phases atomically with domain execution. Reuse the typed domain
receivers, recorded receipts, namespace allocator and independent pins.
Discover pending initialization and successor grants after cold restore.

Unbind raw custody commands in production; explicit qualification fixtures
retain domain receivers. Keep this partial cutover local pending service
conversion, compact history archival and full capacity qualification.
Retain original custody and registrar commands in the staging service and
fair preparation dispatcher. Preserve both identities through cancellation,
registrar uncertainty and closed recovery; gate absent execution on the local
fence and deadlines while returning recorded knowledge first.

Remove caller-owned raw renewal APIs. Restore renewals with the existing
shared session fence so a failed fresh observation also fences old resolvers.
Charge bounded intent/body copies without raising admission or byte caps.

Qualify 286 publication and nine startup/workspace tests, warnings-denied
all-target Clippy, server build, frozen sources, dependency pins and protected
checkout files. Document asynchronous file attribution and its native studies.

This is an unpublished, unreleasable cutover checkpoint. Current-owner fencing
for cold sessions, orphan lifecycle/history archival, complete production
producer/reader/schema conversion and the remaining runtime/capacity gates
are still required.
A fresh CheckPreparation can still return the previous owner's historical lease after actual owner restoration. Reopening that result incorrectly made the old token usable. Require a target-bound server-owned authority source and compare the admitted incarnation/epoch before and after fresh lease probes. Production reuses validated Cell Control and live node advertisements.

Carry the source through preparation and staging handoff, base/frontier loading and standalone positive recovery. Failed observations permanently fence shared sessions; original known outcomes remain recoverable. Explicit registered Claim under the current owner can still restore a usable session.

Validation: 289 publication and nine real startup/workspace tests on frozen source, workspace/all-target Clippy with warnings denied, canopy binary build, formatting, dependency/protection and documentation-link checks. New SHA-1/SHA-256 regressions cover cold old-owner renewal replay, current-owner takeover and missing/corrupt authority repair without un-fencing old sessions.

This is a local unreleasable cutover checkpoint. Cold staging reconstruction, orphan intent lifecycle/history archival, production ingress/read conversion, retention and full-history/team capacity remain open.
Restore all seven original custody actions without changing identities or
receipts. Retain positive and negative history before fresh owner/lease
checks, including after closed-service recovery and SDK expiry.

Wake bound callback supervisors from the shared permanent session fence;
join cancellation and drop owned resources before returning worker credit.

Validate 296 publication and nine real startup/workspace lifecycle tests,
all-target workspace Clippy with warnings denied, server build and format.
The production cutover and full-history/team capacity gates remain open.
Preserve exact original execution uncertainty in a separate authenticated stop fact. Reuse dispatcher admission for bounded keyset discovery and exact retirement recovery, and allow a stop beside its own pending preparation so its shared session can fence and drain.

All 307 publication and nine real lifecycle tests pass, with workspace all-target Clippy, build, format and frozen-source/static checks. Full library qualification passes 585 of 590 cases; five legacy consumers still query the already-removed objects table. Production lifecycle wiring, reader/producer cutover and capacity qualification remain open. This is an unpublished implementation checkpoint, not a release.
Retire only the tracked transition's expired unresolved initialization head using the existing authenticated stop factory and exact completion. Preserve original outcomes and SDK identities; choose an explicit successor from a receipt-watermarked indexed observation and reject inconsistent state instead of treating it as absence.

Seven new production-registry/schema regression families cover both object formats, real owner restoration after SQLite deletion, original history, automatic retirement, and authority/purpose/context refusal. Full library:592pass/5fail; the same five legacy readers still query the removed objects table. Nine real lifecycle cases, Clippy, build, fmt and frozen/static checks pass. The cutover remains unpublished and incomplete.
@forhappy forhappy changed the title Bound native preparation and add resident workflow ownership Publish native HTTP and SSH pushes through resident custody Oct 5, 2026
A busy publication mutex was reported as exhausted capacity and aborted
resident receive-pack workflows. Wait under the original custody ceiling,
then capture the held command synchronously with fresh custody checks.
Keep genuine quota refusals immediate and retain the original prepared
command, registered recovery and completion receipt.

Surface bounded controller diagnostics before and after Bind without
retaining errors that own physical worker pins. Record stop before those
payloads are dropped so credits cannot be reused before cleanup.

Add real bound-handoff contention, quota and expiry regressions plus
resident error-ownership coverage for both Git object formats. Report CI
tool versions and preserve failed reproductions and qualification limits.
The full workspace still has 34 integration failures; no tests are hidden.
Start the shared release deadline at first retirement so initial Cell selection latency cannot consume it. Reproduce worker contention in both object formats while preserving physical drain and capacity limits. Observe bound expiry through lifecycle completion rather than intermediate staging handoff; record passing library tests and remaining integration failures.
@forhappy forhappy changed the title Publish native HTTP and SSH pushes through resident custody Integrate native pushes, checks, and pull metadata Oct 5, 2026
Bind exact merge intent, native ref versions, mandatory ancestry and the
conditional ref snapshot in the existing catalog certificate. Atomically
publish joint roots, pull state, applied UUID and original-command recovery.
Reuse private ownership, fair dispatch, current reviews/checks and typed
recovery; include both new implementations in the Cell source fingerprint.

Keep public merge integration and terminal graph certification explicitly
open. Preserve complete failed integration results and both-format rollback,
UUID replay and owner-restoration evidence.
Bind a permanent StoredInputRoot audit in native merge codec 6 and save
its descriptor atomically with the immutable UUID result. Verify the
selected original audit before archiving recovery and releasing a pin.

Keep negative and replay attempt receipts independent, require actual
operation closure, and preserve merge/release receipts across body loss
and owner restoration. Reuse existing root and archive structures.

Add five retirement regression families and preserve frozen validation
and unchanged integration failures. Public merge and full qualification
remain open.
@forhappy forhappy changed the title Integrate native pushes, checks, and pull metadata Integrate native pushes, checks, pulls, and reviewed merges Oct 5, 2026
@forhappy forhappy changed the title Integrate native pushes, checks, pulls, and reviewed merges Integrate native pushes, checks, pulls, candidates, and reviewed merges Oct 5, 2026
@forhappy forhappy changed the title Integrate native pushes, checks, pulls, candidates, and reviewed merges Integrate native Git publication and collaboration with resident owner routing Oct 6, 2026
@forhappy
forhappy merged commit aa18585 into main Oct 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant