[Zxf-xufeng]

Which issue or RFC does this PR close?

Follow-up to the OceanBase/seekDB CI reliability work in #1945 and #2243. No issue is automatically closed.

Rationale for this change

OceanBase CI can spend its test budget pulling an image or waiting on an unresponsive database command, then lose useful database evidence during teardown. Migration shards also need budgets that account for every selected case and process cleanup. Keep this PR focused on OceanBase and seekDB CI reliability.

What changes are included in this PR?

• Separate bounded image downloads from database startup. Reuse one OceanBase helper for readiness, configuration verification, process deadlines, and best-effort diagnostics.
• Collect database state, sessions, transactions, locks, and resource evidence before terminating timed-out commands or removing failed replay containers. Preserve the original command result when diagnostics fail.
• Balance OceanBase clean-migration work across two shards using recorded durations and a per-shard case-count cap. Require every collected node exactly once; unknown cases remain covered.
• Run the 13 full-Server grant migration cases in a separate group with a 480-second process deadline, including fixture teardown. Keep the other 34 clean migration cases at 360 seconds. Require both groups in the coverage gate; reserve 110/65 minutes for the two steps and 295 minutes for the job, including 40 minutes outside test ceilings.
• Reserve time for setup, test subprocesses, diagnostic uploads, and cleanup in the affected OceanBase/seekDB jobs. Preserve mandatory acceptance and failure gates.
• Reuse only schema in native snapshot tests, with independent Scopes and idempotency keys. Remove an unnecessary HTTP startup from grant-migration fixture setup while retaining post-migration HTTP acceptance and fresh migration databases.
• Keep seekDB timeout enforcement and lock diagnostics in the parent process, disable its timed in-process stack dump, and report native termination signals explicitly.

Are there any user-facing changes?

No production runtime, SQL query, public API, persisted-format, or dependency changes. The PR changes database CI orchestration, diagnostics, and test fixtures. General Python/Gateway CI, ACP/Bub installation, license tooling, desktop builds, and download recovery use the master implementations.

How was this change tested?

• make check: passed.
• Database CI, watchdog, Compose teardown, scope selection, workflow contracts, and upgrade-gate regression selection: 321 passed, 15 platform-dependent skips.
• actionlint on all five changed workflows: passed, allowing the repository's existing custom runner labels.
• Scope audit against master 63f021d6: 28 changed files; no changes under production src/, integrations, or release compatibility metadata.
• Cleanup-budget follow-up: 91 focused checks passed (34 platform-dependent skips); all 47 migration runner and checker tests passed in a real Linux container, including teardown deadlines and rejection of missing grant reports. make check and workflow lint passed.
• In the earlier native run, both failed grant nodes printed PASSED before the 360-second process deadline. The shard-0 rootservice log shows database cleanup beginning at 08:39:21.584 UTC, about 349 seconds into the process, and continuing to drop tables after the deadline at 08:39:32. Another grant case completed in 335 seconds, including 21 seconds of teardown.
• Real OceanBase 4.3.5.6 validation: both previously timed-out grant nodes completed successfully against a fresh ARM64 container from a macOS Python 3.14.6 client. Process times were 68.72s and 65.38s; pytest teardown took 1.35s and 2.38s. Both exited with code 0 and produced passing JUnit reports. The local run did not reproduce the 360-second CI timeout; Linux/x86 CI timing and complete acceptance remain to be verified. Current CI tests commit 77b22dc5. An earlier narrowed run failed during mise signature verification (error refreshing TUF metadata); the next run passed installation and ran the Python 3.11 unit suite (8,674 passed), exposing three checker tests whose fixture omitted the new grant group. The fixture is corrected and the same three failures have been reproduced and verified locally. Full CI database acceptance remains pending behind the Python matrix.

AI usage statement

OpenAI Codex (GPT-6) assisted with analysis, implementation, regression tests and PR preparation.

[hidb4ai]

Merge readiness blocker: this head conflicts with current master (9697d5f264adfafd9f7b15f19e1ca6b0176ab8c6) in .github/workflows/e2e-harness.yml, at Maintain the registered Server chain on real OceanBase without skips. git merge-tree --write-tree d8492f56 9697d5f2 reproduces the content conflict: this PR adds the diagnostic wrapper with a 600-second deadline, while master extends that maintenance step to 900 seconds for fixture cleanup and JUnit output. Please reconcile the wrapper with the updated maintenance timeout and rerun the affected acceptance checks on the resolved commit. The current successful CI run does not validate that integration.

[Zxf-xufeng]

Addressed the merge-readiness review in 08a34940.

Merged the latest fetched master (ed8902b2) and resolved the maintenance-step conflict by keeping the diagnostic wrapper with master's 900-second timeout. The five maintenance cases and the no-skips/JUnit checks remain intact.

Also fixed the diagnostic lifecycle: the native snapshot is collected before replay, and failed OceanBase replay collects a separate bounded snapshot before compose down. Diagnostic failures preserve the original replay exit status and do not prevent teardown.

Local validation passed: make check, actionlint for the affected workflows, and the focused CI/launcher regression suites (289 passed, 15 skipped for platform/external-runtime conditions). The launcher regression uses a Docker substitute to verify evidence preservation and cleanup; it does not establish real OceanBase acceptance. The pushed merge commit triggers a fresh CI run, which must validate the integrated native acceptance before merge.

[AlexStocks]

[P2] 分片 3→2 之后，这一步的 90 分钟上限不再覆盖逐用例超时之和

同一个 diff 里 --test-timeout 300 保持不变，但每片用例数从 16 增到 24，最坏路径从 16 × (300 + 10s kill-after) = 82.7 分钟（小于 90 分钟上限）变成 24 × 310s = 124 分钟（大于 90 分钟上限）。新注释把推导从「16 个用例 × 300s」改成「实测约 21m per worker」，等于用均值替换了原来的最坏情况保证。

数字来自本 PR 新增的 .github/oceanbase-migration-durations.json：SUITES 三个文件共 47 个用例、实测合计 2543.7s（42.4 分钟），二分片后每片 24 个、平均 21.2 分钟——注释里的「21m」是准的，但它是最坏情况之外的另一种口径。

影响：本 PR 要治的失败模式正是 OceanBase 启动/迁移阶段的挂起。同片内多个用例挂起时，scripts/oceanbase_ci.py run 的 wait(timeout - min(30, timeout/2)) → collect(budget=20) → execution.json / before-timeout/ 这条取证路径还没轮到，步骤就先被 90 分钟上限杀掉，失败归因退化成「步骤超时」而不是带诊断的逐用例超时。同一个作业里其它三步仍然自洽（background-metrics：1 个用例 / 10 分钟；startup-snapshot：最多 2 个 × 310s + 180s = 13.3 分钟 / 15 分钟；long-query：4 个 × 310s + 180s = 23.7 分钟 / 30 分钟），只有这一处被打破。

建议二选一：

• 提高该步骤上限到 ≥ 130 分钟，与 24 × (300 + 10s) 对齐；或
• 把 --test-timeout 降到 200s（24 × 210s = 84 分钟 ≤ 90 分钟），并同步改写注释里的推导。

如果确实有意只按实测均值设上限，请在注释里写明这一点，否则后续维护者会继续按「上限 ≥ 逐用例超时之和」读这组数字。

[Zxf-xufeng]

Fixed in ad6b9b5b, with cold-start headroom completed in da11dcd4. The duration-based assignment produces 22 and 25 clean-migration cases. The larger shard needs up to 25 × (300 + 10) + (180 + 10) = 7,940s (132m 20s), including collection and termination grace. The step now allows 140 minutes. The job allows 250 minutes: 210 minutes of test-step budgets plus 40 minutes for a separately bounded cold image pull, startup, diagnostics and upload. The per-case deadline remains 300 seconds.

The regression check fails with the original 90-minute step and passes with the corrected budget. The runner still stops at the first failed case; the budget also covers slow passing cases before that failure. This changes ceilings, not the number of executions or retries. A separate cold-pull regression reproduces the previous 180-second startup failure and verifies warm-cache reuse and failure propagation.

The same changes are included in #2526. Both preceding runs passed all Python 3.11–3.14 unit/E2E jobs, but full backend CI is not yet green. Native logs from the separate background-metrics case show successive full-text index DDL still progressing at 277 seconds, before listener checks. That full-Server lifecycle now has a bounded 600-second deadline and a 15-minute step (including collection/grace); ordinary clean-migration cases retain 300 seconds. Lifecycle checkpoints are preserved, assertions remain intact, and no retry is added. This addresses premature termination of observed progress; the underlying engine latency and final pass still require CI verification. Final native acceptance is pending the new CI runs.

[Zxf-xufeng]

Merged current master (2291237a) and pushed b7fc8bdb. The Desktop conflict is resolved using master's independent synthetic compatibility fixture, which covers the earlier regression without changing release qualification. Both PR branches and their combined merge are conflict-free.

The migration case-count cap and separate native startup/test/cleanup budgets remain in place. I also fixed the new harness validation failure: preinstalling uv changed the task images, so all ten shipped task manifests now pin their actual Harbor checksums. A regression check reproduces the mismatch before the fix and passes afterward; checksum enforcement remains enabled.

Current CI: https://github.com/oceanbase/powercontext/actions/runs/42633344135. Local checks pass, but complete native CI validation is still pending.

The tool-setup fallback also now works on internal runners without gh. It reuses mise only after checking its pinned digest, or obtains the pinned archive through curl with the token on stdin. Archive/binary digests, HTTPS-only redirects, the download deadline and pnpm signature verification remain enforced. The failure before unit tests was a pnpm verification-service TLS error followed by a missing gh command; the new CI still needs to validate the live installation path.

[AlexStocks]

[P2] long-query 步骤的 40 分钟上限同样只对当前基线成立，而该基线缺 5 个已收集的用例

clean-migration 这一条已核实（exact head c63f9b42，e2e-harness.yml / durations.json / test_oceanbase_ci.py 与 1742c5fb 逐 blob 相同）：--test-timeout 360、步骤 160 分钟、作业 280 分钟；-k oceanbase --collect-only 收集 47 个节点，assigned_nodes 打包为 22/25，25 × (360+10) + 190 = 9,440s ≤ 9,600s；tests/test_oceanbase_ci.py 的预算用例在本机实跑 15 passed，把该步骤改回 90 分钟即失败（assert 5400 >= 9440）。这条视为闭合。

同一作业里还有一个步骤靠的是同一种等式，但没有对应的断言：

• 位置：.github/workflows/e2e-harness.yml 的 Test long-query retrieval and admission without skips，注释写「Seven cases in the larger shard: 7 * (300s + 10s grace) + 190s collection < 40m」，timeout-minutes: 40，--test-timeout 300。
• 该步骤收集到 13 个 oceanbase 节点，但 .github/oceanbase-migration-durations.json 的 source_commit 是 4b6cf42a，早于引入这 5 个用例的提交，13 个里只有 8 个有时长：在册的是 27–163s（test_supported_long_query_preserves_all_terms_in_both_http_searches 四条为 132/136/136/163s）；缺失的 5 个全在 tests/e2e/test_atomic_memory_fts_coverage.py——test_fts_coverage_counts_analyzer_terms_omitted_by_space_parser[oceanbase-{com,the,www}-legacy]、test_fts_coverage_keeps_distinct_terms_and_filters_before_limits[oceanbase]、test_hybrid_coverage_keeps_vectors_and_validates_rows_outside_lexical_matches[oceanbase]。
• assigned_nodes 对缺失节点用全体中位数（29s）估算，于是它们被排到最后、按「负载」而不是「条数」分摊。回复里的 13 个用例、6/7 正是这个估算下的结果。把缺失节点的耗时按同组在册用例的量级代入（≥120s）重跑同一算法，分配变成 8/5，最坏路径 8 × (300+10) + 190 = 2,670s = 44.5 分钟 > 40 分钟；翻转阈值落在 100s 与 120s 之间。
• 后果与原问题同形：同片多个用例挂起时，步骤上限先杀掉进程，scripts/oceanbase_ci.py run 的 wait(timeout - min(30, timeout/2)) → collect(budget=20) → execution.json / before-timeout/ 这条取证路径还没轮到，失败退化成「步骤超时」。

差别只在覆盖面：tests/test_oceanbase_ci.py 里 largest_shard × (case_timeout + 10) + 190 的断言只对 clean-migration 步骤生效（该用例名即 test_clean_migration_shard_budget_covers_slow_cases_and_keeps_cleanup_time），long-query、startup-snapshot、background-metrics 都没有同类断言，所以 master 再加用例时只有注释在维持等式。

建议：

1. 用包含这 5 个节点的运行刷新 .github/oceanbase-migration-durations.json，并把 source_commit 前移到包含它们的提交；
2. 把该断言按步骤表驱动到所有运行 run_atomic_memory_clean_migration.py 的步骤（从 --test-timeout 取逐用例上限、从该组 SUITES 的收集数取条数），而不是只钉住 clean-migration 一个步骤。

验证范围说明：13 个节点是用 -k oceanbase --collect-only 在本机 WSL 收集的（head 1742c5fb，上述 6 个文件在 c63f9b42 上 blob 相同）；5 个缺失节点的真实耗时无法在无 OceanBase 的环境测得，因此这一条的依据是「基线缺测 + 打包会翻转」的算术，不是实测耗时。

[Zxf-xufeng]

Fixed in b7fc8bd. The scheduler now enforces a hard ceil(collected_cases / shards) case-count cap while balancing estimated durations. Missing or refreshed timing entries can no longer turn the 13-case long-query group into an 8/5 split: each worker gets at most 7 cases, so 7 × (300 + 10) + 190 = 2,360s fits the 40-minute step. Clean migration is capped at 24 cases per worker, fitting its 160-minute step.

The budget regression now runs actual pytest collection for all six OceanBase migration-runner steps and derives each step's case timeout and shard count from the workflow. It covers long-query, startup-snapshot, background metrics and native FTS as well as clean migration. Adversarial duration distributions reproduced the old overflow before the scheduler change.

I have not fabricated measurements for the five missing legacy-FTS cases: the latest runs stopped in clean migration before reaching that group. The hard cap makes deadline safety independent of those estimates; the timing baseline can be refreshed once a complete native run supplies those measurements. make check and the updated budget tests pass locally. Native CI validation remains pending on this pushed head.

[AlexStocks]

[P2 复核] 本线程两项在 head b9b9bf9f 上已闭合

两项判据均为实测/源码级复算，不是按提交标题推定。

1. long-query 步骤（本轮上一回复指出的 2,670s > 2,400s）：5d053a8e 在 scripts/run_atomic_memory_clean_migration.py::assigned_nodes 增加了 capacity = math.ceil(len(nodes) / count) 与 sizes[shard] < capacity 过滤，条数上界不再依赖 durations 估计。该步骤收集 13 个节点、2 分片 ⇒ 每片 7 个 ⇒ 7 × (300 + 10) + 190 = 2,360s ≤ 2,400s。
2. clean-migration 步骤：47 个节点、2 分片 ⇒ 每片 24 个 ⇒ 24 × (360 + 10) + 190 = 9,070s ≤ 9,600s（timeout-minutes: 160）。

新增断言确实能检出这两类回退：test_duration_estimates_cannot_exceed_the_per_shard_case_budget 在去掉 capacity 后会把 13 个节点分成 6/6/1，超过 ceil(13/3)=5 而失败；test_all_oceanbase_migration_step_budgets_cover_collected_cases_and_cleanup 的分母改用 pytest --collect-only -q ... -k oceanbase 的实际收集数，而不是 durations 键集。

实测（exact head b9b9bf9f，WSL，Python 3.13.11，镜像树 D:\test\github\review\oceanbase-powercontext-pr-2515\source\b9b9bf9f）：tests/test_oceanbase_ci.py + tests/test_ci_pipeline.py + tests/test_e2e_workflow_contract.py = 79 passed / 1 failed；唯一失败 test_ci_e2e_runs_each_case_once_and_keeps_hook_tests_serial 的栈是 ERROR: Error importing plugin "xdist.plugin"，本机 venv 未装 pytest-xdist（pyproject.toml 已声明），属环境缺件，非本 PR 引入。

仍未闭合的证据缺口：head b9b9bf9f 的 check-runs 只有 12 条且全部 queued/in_progress（0 条完成），因此上述预算结论目前只有静态断言与本地用例支持，没有该 head 上的 OceanBase 原生验收跑。

[Zxf-xufeng]

The remaining native validation gap is still open. The branches now include master 2291237a, and the count cap and collected-case budget checks are preserved in b7fc8bdb. I also corrected the Harbor manifest checksums that blocked harness validation after the task-image change. I am tracking https://github.com/oceanbase/powercontext/actions/runs/42633344135 and will report the actual OceanBase results; local budget checks are not being treated as native acceptance.

[AlexStocks]

head 已移到 8c3a7f74（原 b9b9bf9f），复核此前两条 P2 的对账口径：

• atomic-memory-clean-oceanbase-shards：作业上限 280，计时步骤 11+14+10+15+160+15+40+3 = 268 ≤ 280；clean-migration 步骤 160 分钟 = 9600s、2 分片、逐用例 360s ⇒ 每片需 ≤ 26 例，当前 24 例/片，仍满足「步骤上限 ≥ 每片用例数 × 逐用例超时」。
• release/upgrade-support.json 的 persistence_sha256 复算为 eecd03c76b8aa30ba6a80c49380366c971b639d849e3d846abd8345451603c07，与声明一致（241 个受管文件）。

缺口：该 head 只有 tests (3.11/3.12, unit)、docs-tests、check-website、CI preflight 完成，6 个 e2e/acceptance 腿仍是 queued，本轮没有可引用的验收证据。

[hireview001]

• CI failed: CI result (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase) (failure).
• CI failed: harness-validation / Acceptance (oceanbase) (failure).
• CI failed: harness-validation / Recall effort (OceanBase) (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 1/2) (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 0/2) (failure).
• CI failed: desktop-validation / linux (failure).
• CI failed: desktop-validation / macos (macos-15) (failure).
• CI failed: desktop-validation / windows (failure).
• CI failed: desktop-validation / macos (macos-15-intel) (failure).
• Additional CI issues: PR checks.

[hidb4ai]

Re-reviewed 8e9bf23f: the previous merge conflict is resolved. Native migration acceptance remains a merge-readiness blocker.

In run 42758142646, both shard 0 and shard 1 timed out after 360 seconds: test_legacy_grant_replay_and_conflicts_survive_migration[oceanbase-False] and test_migration_rejects_unverifiable_grant_receipts[oceanbase-False-missing] logged PASSED, but their processes exited with 124 before completing cleanup/reporting. The grant test file, migration runner, diagnostic wrapper, and per-case deadline are unchanged in this head.

Please complete both native shards and the exactly-once coverage gate on the current head, including process teardown and JUnit output. If these timeouts recur, investigate the post-test shutdown before treating the acceptance as complete. The current run is still in progress; the earlier test-body passes do not clear this validation gap.

[AlexStocks]

本轮只提一条，且不是代码缺陷，是合并顺序问题：#2518 正在把同一个改动再做一遍，而它今天 09:40Z 已被 LGTM。

先说明本轮我没有重复自己：11-09 06:19 那条已经要求过「用包含这 5 个节点的运行刷新 .github/oceanbase-migration-durations.json 并把 source_commit 前移」，assigned_nodes 的 capacity 复算也在 11-09/11-10 两条里对过账，所以 durations 基线陈旧、--test-timeout 与基线解耦这两点我不再重复提。

阻断点：与 #2518 撞同一 hunk

按各自 PR 的 base 重算（2515 63f021d6、2518 7fef43d6），两侧改动的文件集重叠 4 个：

文件	2515	2518
.github/scripts/run_seekdb_migration_test.py	改	改（同一 hunk）
.github/workflows/master.yml	16 行	4 行
scripts/replay_candidate_migration.py	4 行	2 行
tests/test_upgrade_release_gate.py	2 行	89 行

那个 hunk 两版都是 @@ -162,7 +162,9 @@：faulthandler_timeout 120 → 0，位置、缩进、上下文行完全一致，只有注释不同。这意味着后合的一方必然产生文本冲突，而冲突解决时会丢掉其中一份注释。

更值得停一下的是：两份注释给出的根因互不相同——

• 本 PR：Native CI crashes have ended inside timed Python stack dumps.
• #2518：The C watchdog can itself segfault while native/greenlet frames change.

「栈转储期间崩溃」与「watchdog 自己在原生帧上 segfault」不是同一件事。如果两者都对，那这是两个独立的缺陷，关掉转储只压住了其中一个的症状；如果只有一个对，另一个 PR 的理由就是错的。请给出触发该判断的那次 CI 运行链接（哪个 job、哪一步、崩溃栈长什么样），否则这个注释无法被后来者验证。

关于改动本身我不反对：我在 #2518 上也审过这条（discussion_r4504866607、discussion_r4524783365），确认父进程的两级硬期限仍在、关掉 C 侧 watchdog 不会让挂起作业无限等待。所以本条的诉求只有一句：别让两个 PR 各合一半。

请二选一：

1. 由 #2518 作为载体（它已被 LGTM，且 12 个文件更聚焦），本 PR 去掉 run_seekdb_migration_test.py 这个 hunk，其余 4 个文件按 master 现状 rebase；
2. 由本 PR 作为载体，请先把 #2518 的审批撤回或降级，避免它在冲突状态下被合入。

本轮顺手核对、确认不是问题的两处

为避免作者按旧结论白改：

• 整段删除的 Run recall, statistics and prepared-context regressions（原 10 个文件）不是覆盖损失。该步骤里只有 tests/builtin/persistence/test_recall_effort_statistics.py 带 [sqlite, oceanbase] 参数化，5 个测试里仅 1 个，而 head 的 contracts 步骤仍然跑 test_recall_effort_schema_upgrade_and_atomic_daily_scope_accounting[oceanbase]；其余 9 个文件 grep 到的「oceanbase」是许可证头的 Copyright (c) 2026 OceanBase。且 master.yml 从不设置 POWERCONTEXT_TEST_OCEANBASE_URL，OceanBase 只在 e2e-harness.yml 跑，与删除无关。
• 新增的 test_exit_code() 没有屏蔽失败：它只把负返回码转成 128 + signum（SIGKILL → 137），仍非零。

评审基线：head 77b22dc5，base 63f021d6，git diff 复算 28 文件 +1354/-414，与 PR 页面的 numstat 一致。以上重叠与 hunk 判定来自 git diff <各自 baseRefOid> 的文本比对，非按提交标题推定；本轮没有该 head 上的 OceanBase 原生验收证据可引用。

[AlexStocks]

[P2] 这个 hunk 与 #2518 逐行重复，且后合者必冲突

#2518（head 716ff071，已在 09:40Z 被 LGTM）在同一个 hunk（@@ -162,7 +162,9 @@）也把 faulthandler_timeout 从 120 改成 0，上下文、缩进一模一样，只有注释不同：

• 本 PR：Native CI crashes have ended inside timed Python stack dumps.
• #2518：The C watchdog can itself segfault while native/greenlet frames change.

两个 PR 是两位不同作者（本 PR Zxf-xufeng，#2518 PsiACE），彼此看不到对方的评审线程。无论哪个先合，另一个都要在这个 hunk 上解冲突，而解冲突时通常只会保留一份注释——另一份关于「为什么必须关掉转储」的判断就此消失，后续没人能还原。

改动本身我在 #2518 上审过并不反对（discussion_r4524783365）：父进程的 wait(diagnose_after) → 采集 → wait(timeout - elapsed) → 124 两级期限都还在，关掉 C 侧 watchdog 不会让挂起无限等待。所以这里只需要一处落地。

建议本 PR 去掉这个 hunk，把载体让给已被审批的 #2518；若反过来以本 PR 为载体，请同时把 #2518 的审批降级。另外请补上支撑该注释的 CI 运行链接——「crashes have ended inside timed Python stack dumps」目前在仓库内没有出处，而两份注释的根因说法并不一致。

[hidb4ai]

Follow-up for final head 77b22dc5, now that this PR is closed and the reliability changes were consolidated into #2526: native acceptance is still incomplete.

In run 42800079766, shard 0, the previously failing test_legacy_grant_replay_and_conflicts_survive_migration[oceanbase-False] now exits successfully in about 367 seconds. However, test_migration_rejects_unverifiable_grant_receipts[oceanbase-False-actor] reaches the new 480-second deadline with exit 124 and never prints PASSED. Please investigate this stalled case separately from the earlier post-test cleanup overruns and obtain complete native coverage in the integrated workflow.

Shard 1 fails during the bounded image pull before executing its tests; this is a separate infrastructure/validation failure. These logs do not establish a production-runtime regression, but they do not clear the remaining native acceptance gap.