Skip to content

feat(processing): classify Dream, Handoff, and Evidence processing failures with safe diagnostics (#2126) - #2539

Merged
hidb4ai merged 6 commits into
oceanbase:masterfrom
guamiao:fix/2126-experience-profile-skill-diagnostics
Nov 12, 2026
Merged

hidb4ai merged 6 commits into
oceanbase:masterfrom
guamiao:fix/2126-experience-profile-skill-diagnostics

Conversation

@guamiao

@guamiao guamiao commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Extends the unified processing failure classification (issue #2126) to Dream, Handoff, and Evidence error types, covering the Experience, Profile, Skill, and Handoff artifact families.

  • DreamError: 25 error codes classified via _dream_error_diagnostics() — covers generation configuration, usage limits, business validation, storage failures, and control-flow concurrency.
  • InvalidHandoffGenerationError: All 6 codes classified via _handoff_error_diagnostics().
  • HandoffGenerationUnavailableError and EvidenceResolutionError: Routed to the new classifiers.
  • classify_block_code(): Now supports stage="dream" durable blocks alongside existing stage="topic_memory".

Changes

File Lines
src/powercontext/builtin/processing_diagnostics.py +125
tests/builtin/test_processing_diagnostics.py +130

Verification

uv run ruff check → All checks passed
uv run python -m pytest tests/builtin/test_processing_diagnostics.py → 149 passed
uv run python -m pytest tests/builtin/{test_processing_diagnostics,test_topic_memory_*,test_processing_scheduler} → 280 passed
uv run python scripts/upgrade_release_gate.py → passed
uv run python scripts/release_version.py --check → passed
uv run python scripts/integration_guidance_skills.py --check → passed
uv run python scripts/check_workflow_actions.py → 264 references checked
uv lock --locked → 265 packages resolved

Related

Review fixes and CI evidence

Dream review fixes: e7efb1a3; formatting-only follow-up: 7029bcde. Linux make check, focused tests and expanded regressions passed. CI for the current head is fully green (all non-skipped checks passed; 22 conditionally skipped). Detailed evidence and review replies: review summary. Reviewers have been asked to re-review the current head.

…ilures with safe diagnostics (oceanbase#2126)

Add _dream_error_diagnostics() and _handoff_error_diagnostics() to extend
the bounded failure classification vocabulary to the remaining processing
families (Experience, Profile, Skill via Dream; Handoff standalone).

- _dream_error_diagnostics() covers 25 DreamError and EvidenceResolutionError
  codes mapped to the shared ProcessingStage / ProcessingFailureReason enums.
- _handoff_error_diagnostics() covers all 6 InvalidHandoffGenerationError codes.
- classify_block_code() now supports dream-stage durable blocks in addition to
  topic_memory.
- classify_processing_error() routes DreamError, InvalidHandoffGenerationError,
  HandoffGenerationUnavailableError, and EvidenceResolutionError through the
  new classifiers.

All classifications are bounded vocabulary lookups; unknown codes safely
return None, consistent with the existing topic_memory pattern. No persistence
or integration guidance files are affected.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@guamiao guamiao changed the title ffeat(processing): classify Dream, Handoff, and Evidence processing failures with safe diagnostics (#2126)eat(processing): classify Dream, Handoff, and Evidence processing fa… feat(processing): classify Dream, Handoff, and Evidence processing failures with safe diagnostics (#2126) Nov 7, 2026
if value == "evidence_limit_exceeded":
return ProcessingDiagnostics(
stage=ProcessingStage.BUSINESS_VALIDATION,
reason=ProcessingFailureReason.CONTEXT_OVERFLOW,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid inferring context overflow from evidence caps

EvidenceResolver raises evidence_limit_exceeded for node, edge, depth, item, and byte limits, so this mapping reports context_overflow for structural limits that establish no model-context overflow. Preserve the business_validation stage and failure location, but leave the reason unset unless context-specific evidence supports it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7bdb2e. evidence_limit_exceeded now retains business_validation and its failure location, with the reason unset. The real-runtime Evidence depth-limit regression in tests/e2e/test_artifact_dreaming.py checks this classification before generation; the classification matrix was updated in both locales. Structural evidence limits no longer imply model context overflow.

if value == "memory_entry_inactive":
return ProcessingDiagnostics(
stage=ProcessingStage.PERSISTENCE,
reason=ProcessingFailureReason.STORAGE_FAILURE,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Classify inactive memory as a domain-state rejection

EvidenceResolver._read_atomic raises memory_entry_inactive after successfully reading a deliberately forgotten, retired, or merged Memory, but this mapping diagnoses a storage failure. Classify the rejection as business validation and avoid assigning storage_failure without evidence of a storage defect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7bdb2e. memory_entry_inactive now maps to business_validation, with no storage-failure reason. The real-runtime regression forgets a Memory and checks the resulting Evidence rejection and diagnostic fields, while retaining the pending review result. The mapping no longer treats a normal inactive domain state as a storage defect.

)
if value == "budget":
return ProcessingDiagnostics(
stage=ProcessingStage.GENERATION,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Classify Handoff byte limits as output validation

HandoffService._validate_generated_draft raises budget when the serialized draft exceeds PrepareHandoff.max_bytes, but this branch identifies that output-contract violation as generation resource exhaustion. Map it to business_validation / invalid_output so the diagnostics identify the actual failing boundary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7bdb2e. Handoff budget now maps to business_validation / invalid_output, retaining failure_location=budget. test_prepare_rejects_pipeline_contract_violations calls the real HandoffService.prepare() with an oversized draft, asserts the diagnostic classification, and verifies that no Handoff was committed. This identifies the serialized output contract rather than provider resource exhaustion.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed ebfae3932b70bc0842a8e56c86a0a8b53447cc79. The three existing P2 findings remain reproducible:

  • Evidence caps: two tiny Sources exceeding max_nodes=1 are classified as context overflow before any model call.
  • Inactive Memory: citing a forgotten Memory that remains readable in SQLite is classified as a storage failure.
  • Handoff output budgets: a 1596-byte generated draft exceeding max_bytes=512 is classified as generation usage exhaustion.

Please address these existing threads before merging: leave unsupported context-overflow reasons unset, treat inactive Memory as business validation, and classify the Handoff output-size rejection as business_validation / invalid_output.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@guamiao

guamiao commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@hireview001 @hidb4ai The three diagnostic-classification findings have been addressed in f7bdb2e, with an individual reply in each original thread. Please re-review the current head when convenient.

Validation: Linux make check passed. The final Handoff generation/service and diagnostic selection passed 181 tests; the broader earlier diagnostics/business/logging selection passed 351 tests with 81 skipped. The latter skipped cases are not claimed as verified external backends. Current CI Python 3.11/3.14 unit tests, Python 3.12 E2E, worker observability and SQLite acceptance passed.

Remaining CI evidence

DSH job 130367919597 passed both 469-test JS runs, the 8 JS E2E tests and npm archive verification. It then failed setup validates native include trees observed by a real DSH startup: the fixture's powercontext setup dsh --source <temporary fixture> --server-url <local server> --json child exited 1; the runtime suite reports 18 passed, 1 failed. Its captured CLI error output is not exposed in the job log, so the underlying CI cause is still unconfirmed.

A local WSL attempt reached an earlier, different failure: native DSH could not resolve the temporary powercontext-dsh fixture package before the CLI setup invocation. That is not an equivalent reproduction of the CI failure (local Node 20 / pnpm 12 also differ from the CI toolchain), so I am not treating it as proof of the CI root cause. The master DSH job failed at a different JS assertion (appended versus skipped) and skipped runtime tests; it is not a matching baseline failure.

This PR does not change DSH integration, its setup fixture, dependency lockfiles or CI configuration. The available evidence does not identify this PR as the cause, but it does not establish that the failure is unrelated either. CI remains red; the CI result failure aggregates the failed DSH job.

return _topic_memory_diagnostics(code) if stage == "topic_memory" else None
if stage == "topic_memory":
return _topic_memory_diagnostics(code)
elif stage == "dream":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] classify_block_code("dream", ...) 在生产路径不可达

新增的 elif stage == "dream" 分支只有在 durable block 的 stage 等于 "dream" 时才会命中,但仓内不存在这样的 block。

证据(源码级,head f7bdb2e5)

  • classify_block_code 的唯一生产调用点是 src/powercontext/builtin/runtime/artifact_processing.py:1338,入参 block.stage。
  • ArtifactProcessingBlock 全仓只有一处构造:src/powercontext/builtin/runtime/topic_memory_scope.py:88,那里硬编码 stage="topic_memory";ArtifactProcessingBlockedError 也只在同文件 :173 抛出。
  • git grep -n 'stage="dream"' src/ 无命中,grep -rn 'classify_block_code' src/ 只有上述一个消费点。

影响
_dream_error_diagnostics 的 durable-block 映射只能被 tests/builtin/test_processing_diagnostics.py::test_dream_blocks_classify_for_dream_stage 直接调用命中,不覆盖任何生产行为;而这条用例的通过容易被读成「Dream 的 durable block 已分类覆盖」。文档矩阵新增的行只对 dream.attempt.failed 成立,durable block 那一半仍是空的。

建议(二选一)

  1. 让 Dream 侧真正产出带 stage="dream" 的 block(ArtifactProcessingBlock + ArtifactProcessingBlockedError),并把测试改为从 blocking 入口驱动,而不是直接调 classify_block_code;
  2. 删除该分支,不在测试与文档中声称 durable block 覆盖。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in e7efb1a. I removed the unreachable stage == "dream" branch from classify_block_code and replaced its synthetic positive mapping test with an explicit unsupported-stage assertion. Dream coverage now describes production dream.attempt.failed logs and persisted Run outcomes; I did not add or imply a Dream durable-block lifecycle. The behavior and documentation change is in e7efb1a; its test-format-only follow-up is 7029bcd.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head 7029bcde 已按「删除不可达分支」的方式处理:

  • classify_block_code 现在只有 stage == "topic_memory" 一条分类路径,elif stage == "dream" 分支已移除;
  • tests/builtin/test_processing_diagnostics.py 新增 test_dream_block_stage_is_not_supported,断言 classify_block_code("dream", "budget_exceeded") 与未知 code 均为 None;
  • 文档同步说明 durable-block 分类当前只覆盖 Topic Memory,Dream 走既有 Run 结果与 attempt 日志。

实测(Windows,exact head 7029bcde,PYTHONPATH=<mirror>/src):tests/builtin/test_processing_diagnostics.py + tests/builtin/artifacts/handoff/test_generation.py + test_service.py 共 181 passed;ruff 0.15.20 format --check 对改动文件通过(仓库钉的是 0.15.7,属工具版本差异)。该条已关闭。

except TimeoutError:
except TimeoutError as error:
self._log_attempt_failure(record, error)
await self._fail_or_retry(record, "budget_exceeded", retry=False)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 同一次超时失败在 run 记录与 dream.attempt.failed 之间用了两套分类

现象
execute() 的 except TimeoutError 把 run 记为 budget_exceeded(本行),随后 _log_attempt_failure 用 classify_processing_error 给同一次失败分类;但 classify_processing_error 先命中 isinstance(error, TimeoutError) 分支,而 asyncio.timeout 抛出的裸 TimeoutError 没有 .operation。

实测(head f7bdb2e5,Windows,Python 3.14.3,PYTHONPATH=<head>/src)

classify_processing_error(TimeoutError())          -> stage=None,             reason=timeout
classify_processing_error(DreamError("budget_exceeded")) -> stage=generation, reason=usage_limit

即超时那一刻:持久化里是 budget_exceeded,日志里是 failure_reason=timeout 且 processing_stage 缺失。

影响
本 PR 新增的覆盖矩阵把 budget_exceeded 归为 generation / usage_limit。于是按 failure_reason=usage_limit 检索日志看不到 supervisor 超时,按 run 的 budget_exceeded 反查日志也匹配不到同一事件——两侧词表对不上,矩阵描述的分类在实际发生的这条路径上不成立。tests/e2e/test_artifact_dreaming.py 覆盖的是 InferenceUnavailableError 的日志字段,没有覆盖这条超时路径。

建议
在超时分支显式给出 stage(例如 _log_attempt_failure 接受分类 override,或在此处构造 ProcessingDiagnostics(stage=PROCESSING_STAGE_FOR_TIMEOUT, reason=TIMEOUT)),并补一条用例断言「超时 run 的 error code 与日志 failure_reason 的对应关系」。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in e7efb1a, with the stable-format follow-up in 7029bcd. Only the outer Dream execution-deadline TimeoutError now uses a DreamError("budget_exceeded") classification override for diagnostics. The original exception still supplies exception_type=TimeoutError; the Run remains terminal budget_exceeded, with no retry. The earlier InferenceTimeoutError handler stays first and retains its retry/timeout classification. Regression coverage triggers a real asyncio.timeout cancellation after the generator starts, then checks the Run outcome, one attempt, no candidate, idempotent replay, matching log fields, and diagnostic/logging failure isolation. Linux make check passed; the expanded Dream/logging/scheduler suite passed 356 tests with 87 skipped (including unconfigured external-backend cases).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head 7029bcde 保留了 Run 码与 attempt 日志的两套分类,但把差异写成了显式契约,并补了回归:

  • docs/{en,zh}/development/processing-failures.md 明确:执行边界捕获的裸 TimeoutError 保留终止 Run 码 budget_exceeded,attempt 日志记 generation / usage_limit 且 failure_location=budget_exceeded,同时保留 exception_type=TimeoutError;stage 表示「Dream 执行预算的归属」,不声称期限耗尽时正在调用模型。InferenceTimeoutError 仍保留操作对应 stage 与 timeout 原因。
  • _log_attempt_failure 对裸超时显式传 classification_error=DreamError("budget_exceeded"),使日志分类与落库 Run 码对齐,不再依赖超时异常自身可被分类。
  • tests/e2e/test_artifact_dreaming.py::test_execution_deadline_matches_terminal_budget_diagnostics 断言 record.failure_location == run.error、failure_reason == "usage_limit",并用 diagnostic_failure 参数覆盖分类失败与日志失败两种降级路径。

实测(exact head 7029bcde):tests/builtin/test_processing_diagnostics.py 等 181 passed。未本地执行 tests/e2e/test_artifact_dreaming.py(依赖数据库夹具,超出本轮预算),该文件在 head 的 CI 里随 tests (3.x, e2e) 运行——本 head 69 个 check run 为 47 success / 22 skipped / 0 failure。按「文档化 + 回归」方式视为已处理。

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@guamiao

guamiao commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@AlexStocks The two new Dream diagnostics comments are addressed in e7efb1a. I removed the unreachable Dream durable-block classifier rather than adding a new block lifecycle, and aligned only the outer execution-deadline log classification with its existing terminal budget_exceeded Run outcome. The two original evidence/inactive-Memory/Handoff classification findings remain fixed in f7bdb2e. Individual replies are in the corresponding review threads. Please re-review the current head.

Validation after the Dream fixes: Linux make check passed; focused timeout/provider-timeout and diagnostic selection passed (11 passed, 9 skipped); expanded Dream, logging, scheduler and Topic Memory regressions passed (356 passed, 87 skipped; skips include unconfigured external backend tests).

CI is not green. DSH job 130367919597 failed the DSH runtime include/setup test; CI exposes only that the fixture setup subprocess exited 1, not its stderr. The main-branch comparison failed earlier in a different JS assertion (appended versus skipped) and skipped runtime tests, so it is not the same baseline failure. A local WSL run did expose ERR_MODULE_NOT_FOUND for the temporary powercontext-dsh fixture package at DSH startup, but this failed before the fixture’s later CLI setup check and is not an equivalent reproduction of the CI failure. The exact CI cause remains unconfirmed.

The DSH failure is in an integration not changed by this PR; the Ubuntu mirror timeout from the other PR does not apply to this PR. I am not claiming that the remaining DSH failure is definitively unrelated. The failed CI result check is the workflow aggregate.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guamiao Re-reviewed e7efb1a3. The three original classification findings are fixed; the removed Dream block branch and the deadline/provider-timeout distinction are also addressed. The remaining blocker is the current-head preflight failure described inline. Please commit the formatter output and rerun CI.

Comment thread tests/e2e/test_artifact_dreaming.py Resolved
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@guamiao

guamiao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@AlexStocks The two Dream findings are addressed in e7efb1a; the follow-up 7029bcd only applies Ruff formatting required by CI. Individual replies are posted in both original threads. Linux make check passed and the expanded Dream/logging/scheduler regression suite passed 356 tests with 87 skipped (unconfigured external backends are not claimed as verified).

The CI run for the formatted head 7029bcd is now green: Python 3.11/3.14 unit tests, Python 3.12 E2E, dsh-package, SQLite acceptance and all other non-skipped checks passed. Twenty-two checks were skipped by workflow conditions. Please re-review the current head.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 1eb34be0. The previous classification and formatting findings are addressed. Dream execution deadlines are now distinguished from dependency timeouts, and request/evidence rejections retain business-validation diagnostics without inferring unsupported failure reasons. The focused regression tests and current-head CI pass. No remaining blocking findings; no new database tables or public APIs.

@hidb4ai
hidb4ai merged commit f6d85d3 into oceanbase:master Nov 12, 2026
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.6.0 v1.6.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants