Repository navigation
feat(processing): classify Dream, Handoff, and Evidence processing failures with safe diagnostics (#2126) - #2539
Conversation
…ilures with safe diagnostics (oceanbase#2126) Add _dream_error_diagnostics() and _handoff_error_diagnostics() to extend the bounded failure classification vocabulary to the remaining processing families (Experience, Profile, Skill via Dream; Handoff standalone). - _dream_error_diagnostics() covers 25 DreamError and EvidenceResolutionError codes mapped to the shared ProcessingStage / ProcessingFailureReason enums. - _handoff_error_diagnostics() covers all 6 InvalidHandoffGenerationError codes. - classify_block_code() now supports dream-stage durable blocks in addition to topic_memory. - classify_processing_error() routes DreamError, InvalidHandoffGenerationError, HandoffGenerationUnavailableError, and EvidenceResolutionError through the new classifiers. All classifications are bounded vocabulary lookups; unknown codes safely return None, consistent with the existing topic_memory pattern. No persistence or integration guidance files are affected. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
| if value == "evidence_limit_exceeded": | ||
| return ProcessingDiagnostics( | ||
| stage=ProcessingStage.BUSINESS_VALIDATION, | ||
| reason=ProcessingFailureReason.CONTEXT_OVERFLOW, |
There was a problem hiding this comment.
[P2] Avoid inferring context overflow from evidence caps
EvidenceResolver raises evidence_limit_exceeded for node, edge, depth, item, and byte limits, so this mapping reports context_overflow for structural limits that establish no model-context overflow. Preserve the business_validation stage and failure location, but leave the reason unset unless context-specific evidence supports it.
There was a problem hiding this comment.
Fixed in f7bdb2e. evidence_limit_exceeded now retains business_validation and its failure location, with the reason unset. The real-runtime Evidence depth-limit regression in tests/e2e/test_artifact_dreaming.py checks this classification before generation; the classification matrix was updated in both locales. Structural evidence limits no longer imply model context overflow.
| if value == "memory_entry_inactive": | ||
| return ProcessingDiagnostics( | ||
| stage=ProcessingStage.PERSISTENCE, | ||
| reason=ProcessingFailureReason.STORAGE_FAILURE, |
There was a problem hiding this comment.
[P2] Classify inactive memory as a domain-state rejection
EvidenceResolver._read_atomic raises memory_entry_inactive after successfully reading a deliberately forgotten, retired, or merged Memory, but this mapping diagnoses a storage failure. Classify the rejection as business validation and avoid assigning storage_failure without evidence of a storage defect.
There was a problem hiding this comment.
Fixed in f7bdb2e. memory_entry_inactive now maps to business_validation, with no storage-failure reason. The real-runtime regression forgets a Memory and checks the resulting Evidence rejection and diagnostic fields, while retaining the pending review result. The mapping no longer treats a normal inactive domain state as a storage defect.
| ) | ||
| if value == "budget": | ||
| return ProcessingDiagnostics( | ||
| stage=ProcessingStage.GENERATION, |
There was a problem hiding this comment.
[P2] Classify Handoff byte limits as output validation
HandoffService._validate_generated_draft raises budget when the serialized draft exceeds PrepareHandoff.max_bytes, but this branch identifies that output-contract violation as generation resource exhaustion. Map it to business_validation / invalid_output so the diagnostics identify the actual failing boundary.
There was a problem hiding this comment.
Fixed in f7bdb2e. Handoff budget now maps to business_validation / invalid_output, retaining failure_location=budget. test_prepare_rejects_pipeline_contract_violations calls the real HandoffService.prepare() with an oversized draft, asserts the diagnostic classification, and verifies that no Handoff was committed. This identifies the serialized output contract rather than provider resource exhaustion.
hidb4ai
left a comment
There was a problem hiding this comment.
Reviewed ebfae3932b70bc0842a8e56c86a0a8b53447cc79. The three existing P2 findings remain reproducible:
- Evidence caps: two tiny Sources exceeding
max_nodes=1are classified as context overflow before any model call. - Inactive Memory: citing a forgotten Memory that remains readable in SQLite is classified as a storage failure.
- Handoff output budgets: a 1596-byte generated draft exceeding
max_bytes=512is classified as generation usage exhaustion.
Please address these existing threads before merging: leave unsupported context-overflow reasons unset, treat inactive Memory as business validation, and classify the Handoff output-size rejection as business_validation / invalid_output.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
@hireview001 @hidb4ai The three diagnostic-classification findings have been addressed in f7bdb2e, with an individual reply in each original thread. Please re-review the current head when convenient. Validation: Linux Remaining CI evidenceDSH job 130367919597 passed both 469-test JS runs, the 8 JS E2E tests and npm archive verification. It then failed A local WSL attempt reached an earlier, different failure: native DSH could not resolve the temporary This PR does not change DSH integration, its setup fixture, dependency lockfiles or CI configuration. The available evidence does not identify this PR as the cause, but it does not establish that the failure is unrelated either. CI remains red; the |
| return _topic_memory_diagnostics(code) if stage == "topic_memory" else None | ||
| if stage == "topic_memory": | ||
| return _topic_memory_diagnostics(code) | ||
| elif stage == "dream": |
There was a problem hiding this comment.
[P2] classify_block_code("dream", ...) 在生产路径不可达
新增的 elif stage == "dream" 分支只有在 durable block 的 stage 等于 "dream" 时才会命中,但仓内不存在这样的 block。
证据(源码级,head f7bdb2e5)
classify_block_code的唯一生产调用点是src/powercontext/builtin/runtime/artifact_processing.py:1338,入参block.stage。ArtifactProcessingBlock全仓只有一处构造:src/powercontext/builtin/runtime/topic_memory_scope.py:88,那里硬编码stage="topic_memory";ArtifactProcessingBlockedError也只在同文件:173抛出。git grep -n 'stage="dream"' src/无命中,grep -rn 'classify_block_code' src/只有上述一个消费点。
影响
_dream_error_diagnostics 的 durable-block 映射只能被 tests/builtin/test_processing_diagnostics.py::test_dream_blocks_classify_for_dream_stage 直接调用命中,不覆盖任何生产行为;而这条用例的通过容易被读成「Dream 的 durable block 已分类覆盖」。文档矩阵新增的行只对 dream.attempt.failed 成立,durable block 那一半仍是空的。
建议(二选一)
- 让 Dream 侧真正产出带
stage="dream"的 block(ArtifactProcessingBlock+ArtifactProcessingBlockedError),并把测试改为从 blocking 入口驱动,而不是直接调classify_block_code; - 删除该分支,不在测试与文档中声称 durable block 覆盖。
There was a problem hiding this comment.
Addressed in e7efb1a. I removed the unreachable stage == "dream" branch from classify_block_code and replaced its synthetic positive mapping test with an explicit unsupported-stage assertion. Dream coverage now describes production dream.attempt.failed logs and persisted Run outcomes; I did not add or imply a Dream durable-block lifecycle. The behavior and documentation change is in e7efb1a; its test-format-only follow-up is 7029bcd.
There was a problem hiding this comment.
head 7029bcde 已按「删除不可达分支」的方式处理:
classify_block_code现在只有stage == "topic_memory"一条分类路径,elif stage == "dream"分支已移除;tests/builtin/test_processing_diagnostics.py新增test_dream_block_stage_is_not_supported,断言classify_block_code("dream", "budget_exceeded")与未知 code 均为None;- 文档同步说明 durable-block 分类当前只覆盖 Topic Memory,Dream 走既有 Run 结果与 attempt 日志。
实测(Windows,exact head 7029bcde,PYTHONPATH=<mirror>/src):tests/builtin/test_processing_diagnostics.py + tests/builtin/artifacts/handoff/test_generation.py + test_service.py 共 181 passed;ruff 0.15.20 format --check 对改动文件通过(仓库钉的是 0.15.7,属工具版本差异)。该条已关闭。
| except TimeoutError: | ||
| except TimeoutError as error: | ||
| self._log_attempt_failure(record, error) | ||
| await self._fail_or_retry(record, "budget_exceeded", retry=False) |
There was a problem hiding this comment.
[P2] 同一次超时失败在 run 记录与 dream.attempt.failed 之间用了两套分类
现象
execute() 的 except TimeoutError 把 run 记为 budget_exceeded(本行),随后 _log_attempt_failure 用 classify_processing_error 给同一次失败分类;但 classify_processing_error 先命中 isinstance(error, TimeoutError) 分支,而 asyncio.timeout 抛出的裸 TimeoutError 没有 .operation。
实测(head f7bdb2e5,Windows,Python 3.14.3,PYTHONPATH=<head>/src)
classify_processing_error(TimeoutError()) -> stage=None, reason=timeout
classify_processing_error(DreamError("budget_exceeded")) -> stage=generation, reason=usage_limit
即超时那一刻:持久化里是 budget_exceeded,日志里是 failure_reason=timeout 且 processing_stage 缺失。
影响
本 PR 新增的覆盖矩阵把 budget_exceeded 归为 generation / usage_limit。于是按 failure_reason=usage_limit 检索日志看不到 supervisor 超时,按 run 的 budget_exceeded 反查日志也匹配不到同一事件——两侧词表对不上,矩阵描述的分类在实际发生的这条路径上不成立。tests/e2e/test_artifact_dreaming.py 覆盖的是 InferenceUnavailableError 的日志字段,没有覆盖这条超时路径。
建议
在超时分支显式给出 stage(例如 _log_attempt_failure 接受分类 override,或在此处构造 ProcessingDiagnostics(stage=PROCESSING_STAGE_FOR_TIMEOUT, reason=TIMEOUT)),并补一条用例断言「超时 run 的 error code 与日志 failure_reason 的对应关系」。
There was a problem hiding this comment.
Addressed in e7efb1a, with the stable-format follow-up in 7029bcd. Only the outer Dream execution-deadline TimeoutError now uses a DreamError("budget_exceeded") classification override for diagnostics. The original exception still supplies exception_type=TimeoutError; the Run remains terminal budget_exceeded, with no retry. The earlier InferenceTimeoutError handler stays first and retains its retry/timeout classification. Regression coverage triggers a real asyncio.timeout cancellation after the generator starts, then checks the Run outcome, one attempt, no candidate, idempotent replay, matching log fields, and diagnostic/logging failure isolation. Linux make check passed; the expanded Dream/logging/scheduler suite passed 356 tests with 87 skipped (including unconfigured external-backend cases).
There was a problem hiding this comment.
head 7029bcde 保留了 Run 码与 attempt 日志的两套分类,但把差异写成了显式契约,并补了回归:
docs/{en,zh}/development/processing-failures.md明确:执行边界捕获的裸TimeoutError保留终止 Run 码budget_exceeded,attempt 日志记generation/usage_limit且failure_location=budget_exceeded,同时保留exception_type=TimeoutError;stage 表示「Dream 执行预算的归属」,不声称期限耗尽时正在调用模型。InferenceTimeoutError仍保留操作对应 stage 与timeout原因。_log_attempt_failure对裸超时显式传classification_error=DreamError("budget_exceeded"),使日志分类与落库 Run 码对齐,不再依赖超时异常自身可被分类。tests/e2e/test_artifact_dreaming.py::test_execution_deadline_matches_terminal_budget_diagnostics断言record.failure_location == run.error、failure_reason == "usage_limit",并用diagnostic_failure参数覆盖分类失败与日志失败两种降级路径。
实测(exact head 7029bcde):tests/builtin/test_processing_diagnostics.py 等 181 passed。未本地执行 tests/e2e/test_artifact_dreaming.py(依赖数据库夹具,超出本轮预算),该文件在 head 的 CI 里随 tests (3.x, e2e) 运行——本 head 69 个 check run 为 47 success / 22 skipped / 0 failure。按「文档化 + 回归」方式视为已处理。
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
@AlexStocks The two new Dream diagnostics comments are addressed in e7efb1a. I removed the unreachable Dream durable-block classifier rather than adding a new block lifecycle, and aligned only the outer execution-deadline log classification with its existing terminal Validation after the Dream fixes: Linux CI is not green. DSH job 130367919597 failed the DSH runtime include/setup test; CI exposes only that the fixture setup subprocess exited 1, not its stderr. The main-branch comparison failed earlier in a different JS assertion ( The DSH failure is in an integration not changed by this PR; the Ubuntu mirror timeout from the other PR does not apply to this PR. I am not claiming that the remaining DSH failure is definitively unrelated. The failed |
hidb4ai
left a comment
There was a problem hiding this comment.
@guamiao Re-reviewed e7efb1a3. The three original classification findings are fixed; the removed Dream block branch and the deadline/provider-timeout distinction are also addressed. The remaining blocker is the current-head preflight failure described inline. Please commit the formatter output and rerun CI.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
@AlexStocks The two Dream findings are addressed in e7efb1a; the follow-up 7029bcd only applies Ruff formatting required by CI. Individual replies are posted in both original threads. Linux The CI run for the formatted head 7029bcd is now green: Python 3.11/3.14 unit tests, Python 3.12 E2E, |
hidb4ai
left a comment
There was a problem hiding this comment.
Re-reviewed 1eb34be0. The previous classification and formatting findings are addressed. Dream execution deadlines are now distinguished from dependency timeouts, and request/evidence rejections retain business-validation diagnostics without inferring unsupported failure reasons. The focused regression tests and current-head CI pass. No remaining blocking findings; no new database tables or public APIs.
Extends the unified processing failure classification (issue #2126) to Dream, Handoff, and Evidence error types, covering the Experience, Profile, Skill, and Handoff artifact families.
Changes
src/powercontext/builtin/processing_diagnostics.pytests/builtin/test_processing_diagnostics.pyVerification
uv run ruff check → All checks passed
uv run python -m pytest tests/builtin/test_processing_diagnostics.py → 149 passed
uv run python -m pytest tests/builtin/{test_processing_diagnostics,test_topic_memory_*,test_processing_scheduler} → 280 passed
uv run python scripts/upgrade_release_gate.py → passed
uv run python scripts/release_version.py --check → passed
uv run python scripts/integration_guidance_skills.py --check → passed
uv run python scripts/check_workflow_actions.py → 264 references checked
uv lock --locked → 265 packages resolved
Related
Review fixes and CI evidence
Dream review fixes:
e7efb1a3; formatting-only follow-up:7029bcde. Linuxmake check, focused tests and expanded regressions passed. CI for the current head is fully green (all non-skipped checks passed; 22 conditionally skipped). Detailed evidence and review replies: review summary. Reviewers have been asked to re-review the current head.