Skip to content

Cell attachment verifies registration and fences activation - #735

Open
zlareb1 wants to merge 122 commits into
mainfrom
feat/cell-attach-cli
Open

zlareb1 wants to merge 122 commits into
mainfrom
feat/cell-attach-cli

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Related: #693

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

loom cell attach places a prepared first cell’s orgs, installs its tenancy triggers and makes it serve routed traffic. Attach verifies store pairing and fences stale control requests; native token minting requires reachable, migrated stores, and manual participant retirement verifies physical death instead of accepting --yes.

Why

A prepared first cell cannot serve routed traffic until its placements, tenancy mode and triggers agree. Attach advances those states in order, verifies the durable result and resumes an interrupted attempt from recorded state. A cell already marked serving must pass the same deployment, configuration, cell and bucket-pairing checks before the command reports success.

Scope and risk

  • Attach scans the existing org sources in bounded batches, checks writer privileges and current process evidence, places the orgs, installs tenancy triggers and raises local mode to enforced. Failure after that mode commit leaves the cell enforced; recovery proceeds forward.
  • Router mutations compare registered configuration and a fresh UUID control claim. A stale request cannot regain authority when configuration values return to earlier values or its database-lock backend ends. Control cleanup preserves the first failure through connection loss and repeated cancellation.
  • Online Postgres index creation verifies physical readiness separately from migration-ledger visibility. Cancellation settles the owned backend before retry, standby promotion or cleanup can establish success.
  • Participant retirement requires a completed drain or verified exact physical death with settled queries. Expired heartbeats, deleted pod objects and completed rollouts do not certify retirement. Server, worker and CLI ownership covers request tails, background producers and native database work. The worker parent retains its child’s birth identity and can certify that stopped child only after the tagged-query census is zero, within the existing stop or restart deadline.
  • LOOM_NATIVE_COMMAND_DRAIN_TIMEOUT_S gives a native command its complete cleanup allowance, defaulting to 15 seconds. LOOM_WORKERS_CHILD_RETIREMENT_TIMEOUT_S caps a reaped child’s proof work, defaulting to one second within the original remaining stop or restart allowance. Both must be positive and finite; expiry leaves retirement unproved.
  • The chart uses the documented operator procedure for schema-changing control work and removes pod-death retirement permissions. The existing private probe listener and network restrictions remain intact.
  • Failure diagnostics retain only bounded status, elapsed time, allowlisted codes, cause classes, numeric error identifiers and source sites. They retain the original failure and exclude response content, SQL, addresses, credentials and settings. Failed environment resets can report bounded read-only participant and lease counts before owned teardown.

The main risks are incomplete control-fleet turnover, a writer role that can bypass tenancy checks, and store configuration changing during attach. The command refuses detectable violations; the runbooks name the process and configuration evidence an operator must establish. Later-cell administration and a complete capacity model are separate concerns.

Compatibility, rollout and rollback

  • CLI: native loom mint-token requires reachable, migrated stores; offline minting refuses. loom participant retire --yes is removed. Manual retirement runs on the trusted host that can verify exact physical death and settled queries.
  • Wire: control mutations require registered configuration and mutation generation; missing generations refuse with CONTROL_UPGRADE_REQUIRED. No data-plane operation is removed.
  • Operator steps: apply the additive router migrations, complete the control-fleet turnover in docs/operations/cell-control.md, and follow docs/operations/cell-attach.md. Prepare and attach require direct Postgres connections and a synchronous standby. Rotate the ClickHouse writer credential after the cell rollout and before attach.
  • N-1 data-plane builds can read the UUID column. Old control handlers and one-off clients must be positively retired before generation-aware control traffic resumes; schema compatibility does not prove retirement.
  • Rollback leaves committed tenancy rows, placements, triggers and enforced mode in place. An attached cell remains attached. Recovery does not clear those fences.

Tests

  • The original 21 attach and diagnostic regressions demonstrate nine assertion failures with twelve passing controls, then the same 21 cases pass. They cover serving-registration mismatches, refused fleet reads, the actual pending-publication error envelope and strict error-code bounds.
  • Small related controls cover native command budgets, first-error preservation, repeated cancellation, physical-death evidence, process census churn, fixture ownership and strict metadata validation. Scoped lint, formatting, types and generated references are recorded with their corresponding changes; broad suites run in GitHub CI.
  • The retirement census distinguishes method calls from unrelated local cleanup functions. A focused check over fourteen related sources demonstrates the false positive and then passes with a strict additional-owner rejection control; every original owner assertion and the full-server CI scan remain intact.
  • The environment-reset diagnostic has three demonstrated assertion failures and a healthy control before correction; the same cases and existing strict controls pass together, fourteen cases. Diagnostic failures cannot replace the original reset failure.
  • Actual parent shutdown and natural child-exit regressions each demonstrate a missing-retirement assertion failure. The same cases and identity, physical-death, query-settlement, bounded-lookup and deadline controls pass together: twenty cases. A child already certified by its own shutdown is not rewritten; proof work consumes the existing restart delay.
  • The package-boundary and settings regressions found by CI are corrected through a required callback wired at the server composition root and a positive finite proof-work setting. The same twenty lifecycle cases plus the runner, changed-leaf contracts and defaults control pass together: 24 cases. Eight additional setting cases reject zero, negative and nonfinite values and verify that the configured cap and original remaining deadline each dominate. They pass with the two existing exhausted-budget and natural-exit controls: ten cases. The feature inventory remains unchanged.
  • The native fixture now captures each controlled web worker’s exact direct background child before crash or recreation and requires retirement of the captured subtree. Two regressions demonstrate the omitted child and an incorrectly accepted recreation; the same cases plus strict identity, certificate, query-settlement and crash controls pass together: 30 cases. Closed role counts aid failure diagnosis without weakening the original retirement authority or deadline. This does not establish the cause of an earlier native reset failure.
  • Exact single-thread zombie exit has three demonstrated missing-proof failures, with eighteen passing refusal controls. The same 21 cases pass after correction. Two owned process cases establish the kernel state before any wait: a zombie leader with a live thread refuses, and its exact birth becomes provably stopped only after the group has one zombie thread. Host, boot, PID namespace, birth, query-settlement and retirement ownership checks remain mandatory. This addresses a reproduced proof gap; it does not establish the kernel state of the earlier native failure. Bounded role diagnostics add closed kernel-state counts without changing authority or deadlines. Scoped lint, formatting and the production-file type check pass.
  • The trigger driver’s owning-path cause regression demonstrates three missing-metadata assertion failures and one healthy pass. The same cases, sanitizer and diagnostic-failure controls, strict reader validation and all existing native metadata controls pass together: 33 cases. The original INVALID result survives a diagnostic failure.
  • Prior-head GitHub CI completed both Postgres 16 and 18 jobs: each main batch reports 830 passed, one skipped and one xpassed, followed by the physical-standby index drill, 348 authorization-isolation cases and six reconciliation drills. A later prior head completed the same Postgres 18 batches and the entire amd64 Quickstart, including provider recovery and installer rehearsal. These results are retained evidence, not final-head CI. The two-replica fixture verifies real HTTP identities and shared writes against its isolated stores.
  • Prior native Quickstart validation completed the worker crash, certified retirement, replacement and recreated-stack data checks. Its final reset failed its existing deadline. The bounded snapshot found four retained participants: three acknowledged the current drain and one acknowledged another drain, with no held command lease. These aggregate counts do not identify the participant. The child-lifecycle correction addresses independently reproduced ownership gaps; native CI must establish its effect.
  • The native trigger experiment is not passing. An earlier attempt completed three arms and 8,200 successful API writes before a canceled authority lock query stopped staging. A later attempt failed the initial HTTP store handshake before running the workload. The cancellation and connection-reset origins remain unproved. Neither attempt establishes complete acceptance or a performance result. Original workload, retry and timeout bounds remain intact.
  • A later trigger acceptance attempt records 8,200 successful API writes and three settled arms, then a canceled Postgres head-lock query during delivery staging. It remains INVALID. The cancellation mechanism and failed query plan are unproved. The staging failure path records only bounded elapsed time, key count and closed cancellation reasons. For a query-cancel cause it can retain a bounded fresh planning observation from a separate idle connection; that observation cannot certify the failed backend's cached plan, prior wait state or runtime. Diagnostics preserve the original failure, trigger restoration and owned cleanup, and do not change the workload, timeouts or performance thresholds. Eighteen missing-evidence and healthy controls show seventeen assertion failures and one pass before correction; the same eighteen plus six existing owner controls pass together. The first corrected-code attempt exposed seven invalid exception-constructor fixtures; using the driver's real server-error constructor preserves every assertion and wait.
  • A dedicated CI job retains the fixed 45,500 identities, trigger arm order, payloads and performance thresholds, with isolated stores, source and process verification, trigger restoration and owned cleanup. Full current-head unit, coverage, integration, native, mixed-version, browser and trigger acceptance checks must pass before merge.
  • After composing the heartbeat with session-protection refresh, the same 21 process-proof cases and three existing heartbeat/startup controls pass together: 24 cases. The heartbeat remains tracked by retirement and refreshes protection before reading the lease.
  • The owning subtree diagnostic test reproduces its stale fixture failure in CI. Its corrected kernel records and full closed-count expectations pass with all existing identity and secrecy controls and six additional kernel-observation cases: twenty cases. Product behavior, retirement authority and deadlines remain unchanged.
  • Four store-identity refusals retain distinct source sites in the existing bounded failure envelope. The exact checks, observation order, refusal message and resource limits remain unchanged. The fatal-transport integration test also kills and reaps its own child when communication is interrupted, preserving the original timeout or cancellation. Three assertion failures and three healthy controls become six passing checks; the caller controls use a fake transport and open no database or subprocess.
  • Store readiness queries the published HTTP listener through the existing bounded creator command. A regression demonstrates that native queries can accept the temporary initialization server before that listener is ready, while the Postgres readiness control passes. The same two cases and eight HTTP response, disconnect, closure and command-budget controls pass together: ten cases. The readiness deadline, resource checks and command cancellation remain unchanged. This reproduced readiness gap is consistent with the observed initial HTTP disconnect; it does not uniquely establish that failure’s cause.
  • The native exit witness waits for the full single-thread zombie condition before checking its unchanged proof assertion. Its existing two-second budget and all identity, live-thread refusal, exact-exit and owned-child reaping assertions remain intact. Both existing native witnesses pass; production behavior is unchanged.
  • The native retirement fixture reaps the exact child it starts when its wait is interrupted. Its ten-second wait and all store and physical-death assertions remain unchanged; CI runs the native scenario.
  • The attach lock-contention fixture collects its exact delayed rollback task before either dedicated connection closes, including when the control operation fails or is cancelled. The three-second release, wait allowance and contention assertions remain unchanged; the native scenario runs in CI.
  • In-process integration scenarios use a fresh real retirement gate per test. Shutdown keeps that gate permanently closed within the scenario; a separate patch context preserves isolation through explicit test monkeypatch cleanup. The existing source-push scenario exposes reused process state at startup and must pass in final-head CI. Production retirement behavior is unchanged.
  • Fresh independent review of the entire corrected pushed change is required alongside CI. No inherited result, skip, incomplete experiment or earlier review is presented as final-head success.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Claude Code and Codex. Independent reviews check the complete control protocol, failure paths, store pairing, operator procedures and regression evidence.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Small related local checks and scoped static checks are recorded; broad suites run in CI.
  • Required full GitHub CI and native acceptance pass on the final head.
  • CLI references, settings references and snapshots are regenerated through their generators.
  • Router migrations are additive and no merged migration is edited.
  • Documentation names operator steps, control-protocol requirements and recovery limits.
  • Commits are signed off.

zlareb1 and others added 30 commits November 16, 2026 21:13
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup.

Give gate batch scopes a unique prefix per run so cleanup preserves existing rows.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Count privileges a login can reach through SET ROLE or an ADMIN regrant, including noninherited group grants. Keep the owner exemptions and explicit role allow list.

Repeat the writer, trigger, Postgres catalog and standby checks after the rescan and placement before marking the cell serving. Retain the heartbeat watch and its final wait floor.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check USAGE of pg_read_all_stats and pg_signal_backend because attach uses their privileges without SET ROLE. Keep MEMBER for the separate untrusted-writer reachability checks.

Refuse noninherited capabilities before attach writes, and explain the inherited grant requirement in the refusal labels and runbook. Test both inherited neighbors and signaling a scratch backend.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Claim a fresh directory UUID before acceptance writes, and compare the generation, registration and exact fingerprint pair on control mutations. Keep every local acceptance transaction on its attach-lock backend and stop after a lost session.

The additive router column permits older data-plane reads and inserts. Online control requires turnover of older clients and router handlers; the recovery guide covers claimed state cleanup and offline generation rotation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Refuse explicit open requests under the guarded attaching row lock, preserving quota atomicity and the durable zero-placement boundary. Inventory control writers and their required boundaries, including dynamic cleanup SQL.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise bounded Postgres phases independently, reserving cleanup inside each
existing deadline. Send public close while query and transaction work are alive,
then cancel and reap within that same bound. Borrowed checks and trigger DDL
leave failed-session cleanup to their owner.

Bound the lock owner's close and preserve its original failure across cleanup
errors or repeated cancellation. Keep wait floors and normal busy retries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise bounded Postgres phases independently, reserving cleanup inside each
existing deadline. Send public close while query and transaction work are alive,
then cancel and reap within that same bound. Borrowed checks and trigger DDL
leave failed-session cleanup to their owner.

Bound the lock owner's close and preserve its original failure across cleanup
errors or repeated cancellation. Keep wait floors and normal busy retries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise attach, prepare, removal and backout on their exact lock backend. Keep cleanup inside phase deadlines, forward fatal guard exceptions after close, and retry mode contention only after acknowledged rollback and ownership checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A same-name string projection masks the UUID filter in the native query. Read the native UUID and normalize its returned identity while retaining every retirement predicate.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Restart an incomplete census only after a nonroot process is proven absent. Bound retries by the original freshness interval and retain identity, ownership and resource refusals.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every workload field and acceptance oracle unchanged while updating the fixed descriptor checksum.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The option can be visible before its keyboard selection effect runs. Require that selection before the existing Enter action and retain every geometry assertion.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep numeric error codes and bounded cause classes without exception text or connection details. Validate the sidecar before attaching it to the original failure; execution and cleanup guards remain unchanged.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Collect read-only participant and configuration-lease counts before the failed fixture stack is removed. Keep the original activation failure and deadline when diagnostics fail or time out.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve the wrapped cause, error identifiers and commit uncertainty when staging refuses, without changing workload, retry or acceptance rules.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A child can exit before completing its own retirement. The parent records
retirement only after its captured child identity is physically gone and
every tagged query has ended, within the existing stop or restart deadline.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Wire exact-proof retirement from the serving composition root and expose its bounded proof timeout through settings. Keep the original child identity, query-settlement and cleanup deadlines.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Capture exact background-child births before controlled crash and recreation. Keep certified retirement and settled queries mandatory for every captured participant, and report bounded role counts on failure.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Cover finite positive retirement budgets and the original remaining allowance. Distinguish an already issued scoped token from native minting after enforcement.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Model the captured process stat records and assert all closed kernel counts. Keep identity and secrecy controls and cover missing, unreadable and reused process observations.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the existing store checks and failure envelope while giving each refused observation its own source site. Protect the native test child with the same kill and wait ownership as neighboring tests.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The initialization server can answer native queries before its final HTTP listener starts. Probe the published endpoint through the existing bounded creator command.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A zombie leader can still report multiple threads. Keep polling within the existing allowance until the complete asserted exit state is observed.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The fixture owns the child through timeout and cancellation. Keep its existing wait budget and always collect that exact process before store cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Cancel and join the owned task before dedicated sessions close on an interrupted control operation. Preserve the contention checks and wait allowances.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each in-process test its own real gate while keeping shutdown permanent within the scenario. A private patch context preserves the fixture through explicit test monkeypatch cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A canceled head-lock query leaves its cancellation mechanism and query plan unknown. Record closed reasons and a bounded fresh planning observation while retaining the original failure and cleanup contract.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant