Skip to content

feat(reward-memory): skip duplicate turn reflections - #8565

Open
cocolord wants to merge 1 commit into
mainfrom
codex/reward-memory-skip-duplicate-20261111
Open

cocolord wants to merge 1 commit into
mainfrom
codex/reward-memory-skip-duplicate-20261111

Conversation

@cocolord

Copy link
Copy Markdown
Member

Outcome\n\nAdds one minimal producer-side comparison at the existing Reward Memory Turn reflection seam.\n\nWhen the same Turn both delivers recall guidance and enables automatic ingest, the reflection instruction now:\n- compares independent task evidence only with comparable procedural experience guidance;\n- leaves the reflection empty when evidence merely confirms the same future behavior and limitations;\n- emits a candidate only for distinct reusable applicability or a material change to trigger, action, validation, stop condition, or limitations;\n- keeps distinct applicability separate instead of blending it; and\n- leaves merge, update, retirement, and supersession decisions with configured provider lifecycle owners.\n\nThere is no new state, schema field, store, scheduler, provider route, or background extraction pipeline.\n\n## Observable semantics\n\ndecision_text_assessment: intentional_change_validated\n\nAuthorization basis: the requested outcome was a minimal execution-evidence versus recalled-experience comparison without expanding the already complex chain.\n\nConsumer: the Turn agent deciding whether to populate reward_memory_reflection_json.\n\nClause assessment:\n- Trigger: comparison applies only when guidance was actually delivered and ingest is enabled.\n- Evidence provenance: only current independent task evidence may qualify a reflection.\n- Action: duplicate confirmation produces an empty reflection; distinct or materially changed reusable behavior may produce one candidate.\n- Scope: only comparable procedural experience guidance participates; other guidance preserves the ordinary eligibility rule.\n- Authority: the reflection cannot claim provider lifecycle mutations or new action authority.\n- Stop condition: no comparable change means no candidate.\n\nCounterfactual coverage:\n- recall-only or ingest-only Turn: comparison instruction absent;\n- oversized optional recall omitted by the context budget: comparison instruction absent while ingest remains enabled;\n- no comparable procedural experience: ordinary eligibility still applies;\n- same behavior and limitations: empty reflection;\n- distinct applicability: separate candidate rather than a blended rule.\n\n## Validation\n\n- Exact-head affected tests: 9 passed.\n- Reward Memory Turn, quota, and lifecycle suite during implementation: 175 passed.\n- Ruff checks for changed Python: passed.\n- Standard diff-aware premerge canary: passed, 19 checks selected, 0 failures, 0 manual holds.\n- Public-boundary scan: passed.\n\n## Merge boundary\n\nThis changes agent-consumed behavior in the runtime/control-plane path. Passing validation is not merge authority; maintainer review and explicit merge authorization remain required.

Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · max

评审提交 2ed89fd

[P2] 新增反思提示挤掉原可交付的召回。 同一组 8 条通过 scope、lifecycle 和经验质量校验的记录,主干以 65,190 字节交付全部经验。候选加上 747 字节的提示后需要 65,937 字节,超过 65,536 字节;App guard 省略 8 条经验和比较提示。召回状态仍为 applied,反思字段仍启用。 保留现有 64 KiB 上限,用原有有界指令渠道或经证实无损的预算调整保存原先可交付的召回。加近上限的真实 guard 回归,断言这 8 条经验和比较提示均交付,同时保留真正超限的降级对照。

动机 / Motivation

在同一 Turn 启用 Reward Memory 召回与反思的开发者或 Agent。 本轮拿到多条既有经验并产生新证据时,Agent 需要判断是否有值得保留的变化,避免重复写入同一经验。 小包能够收到比较提示;接近上限的包可能失去全部召回,也得不到比较提示。 本轮只改变反思提示,原 Todo 证据校验、结算及 Provider 生命周期继续负责实际写入。

改动思路 / Approach

已注册钩子先按本轮 Goal、Agent 和自动化开关组装实际 guidance、提示和反思 schema。比较提示加进同一有上限的包;总量超限时重新生成无召回包。 现有实现边界成立;在同一 owner 修复。没有新增状态、schema、权限或迁移。 生产增加 4 行,同步英中文档与 App/宿主用例。条件分支正确;原有召回预算被新增提示共享,出现实际交付回归。

具体改动 / Changes

关键代码讲解

生产增加 4 行,同步英中文档与 App/宿主用例。条件分支正确;原有召回预算被新增提示共享,出现实际交付回归。 小包交付 8 条经验和比较提示;候选 65,529 字节的对照仍正常。原本已超限的输入两边都降级。单开或关闭无比较提示。

依据主干约定(beff0f3e55d1f70eceebbd8e78c11f1cff34643b):Independent task evidence:反思仍要求当轮独立证据(通过);Preserve in-budget recall:原先可交付的召回仍可交付(未达到);Independent ingest authority:不增加 Provider 写入或生命周期权限(通过)。

对主干的风险 / Risks

PASS:原生 App 与宿主隔离测试主干、候选各 153 项通过;Ruff、Git diff --check、quick premerge 的 6 项直接和 4 项选定检查通过。 FAIL:同一组 8 条通过 scope、lifecycle 和经验质量校验的记录,主干以 65,190 字节交付全部经验。候选加上 747 字节的提示后需要 65,937 字节,超过 65,536 字节;App guard 省略 8 条经验和比较提示。召回状态仍为 applied,反思字段仍启用。 召回单开、反思单开、关闭三种对照保持,关闭路径没有 Provider 调用。schema 和已授权写入条件未改变。 只替换 Provider IO,未测试真实服务或模型是否遵从,也未测实际重复写入率。 远端 CI 按评审配置未询,合并资格仍须单独确认。

语义与 CI 对齐

原生 Turn 的召回预算、反思提示与既有校验沿用既有约定,没有新增语义词汇。修复后重跑同一真实入口反例及定向测试,并保留对应的正向和降级对照。

我的整体评价 / Verdict

该改变值得在现有能力中交付;独立反例仍阻断完整结果,修复后应重跑同一对照。 保留现有 64 KiB 上限,用原有有界指令渠道或经证实无损的预算调整保存原先可交付的召回。加近上限的真实 guard 回归,断言这 8 条经验和比较提示均交付,同时保留真正超限的降级对照。 本轮只改变反思提示,原 Todo 证据校验、结算及 Provider 生命周期继续负责实际写入。

English verdict: REQUEST_CHANGES. The additional instruction drops all otherwise deliverable recall at the native App guard boundary while leaving reflection enabled.

)
if has_guidance and ingest_enabled:
instructions.append(
"Before proposing a reflection, compare current independent task evidence with any comparable procedural_experience in the delivered reward_memory_recall guidance. If no comparable procedural experience was delivered, apply the ordinary eligibility rule above. Leave reward_memory_reflection_json empty when the evidence only confirms the same future behavior and limitations; emit a candidate only when it establishes distinct reusable applicability or materially changes the trigger, action, validation, stop condition, or limitations. Keep distinct applicability conditions separate rather than blending them. Do not claim to merge, update, retire, or supersede provider records; those lifecycle decisions remain with their configured owners."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 新增反思提示挤掉原可交付的召回。 同一组 8 条通过 scope、lifecycle 和经验质量校验的记录,主干以 65,190 字节交付全部经验。候选加上 747 字节的提示后需要 65,937 字节,超过 65,536 字节;App guard 省略 8 条经验和比较提示。召回状态仍为 applied,反思字段仍启用。 保留现有 64 KiB 上限,用原有有界指令渠道或经证实无损的预算调整保存原先可交付的召回。加近上限的真实 guard 回归,断言这 8 条经验和比较提示均交付,同时保留真正超限的降级对照。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants