Skip to content

fix(prompt): don't delete or rewrite memories nobody asked to change - #53

Merged
kuitos merged 1 commit into
fix/frame-extraction-transcriptfrom
fix/memory-edit-guardrails
Oct 7, 2026
Merged

kuitos merged 1 commit into
fix/frame-extraction-transcriptfrom
fix/memory-edit-guardrails

Conversation

@kuitos

@kuitos kuitos commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

#49 reports memories that were deleted or rewritten when nobody asked, each traced to a prompt line:

  • Main agent: on a read-only "How do I deploy this project?", it deleted a memory that named a missing scripts/deploy.sh in 6/6 runs (Claude Code: 0/2). The line it followed: "update or remove the stale memory rather than acting on it" (sections.ts).
  • Extraction:
    • turned "last week" into an invented "Oct 2, 2026" ("Convert relative dates to absolute");
    • appended "Unverified as of …" to a memory about an unrelated system and changed that memory's description;
    • added a second memory next to the one the main agent had just saved.
  • Auto-dream:
    • rewrote every description and the whole MEMORY.md ("Rewrite vague descriptions");
    • added "Why: not recorded in the original memories." ("ensure content is structured as").
  • Logging: there was no record of what a dream pass changed.

Fix

Main agent (prompt/sections.ts, prompt/systemPrompt.ts, memory_delete / memory_save descriptions):

  • Answer from what is observed now.
  • A memory that looks stale, or names something that cannot be found, is updated with what was observed and when (e.g. "scripts/deploy.sh not found as of 2026-03-05"), not deleted. It may live on another branch, and the note is still history.
  • Delete only when the user asks to forget, or when a corrected or merged memory replaces it.
  • Resolve a relative date only when it can be worked out exactly. Otherwise keep the user's wording, anchored to today ("last week, as of 2026-03-05"), and never estimate a date.

Extraction (extraction/prompts.ts, ExtractionCoordinator.ts):

  • Today's date is passed with the transcript.
  • The same date rule as the main agent applies.
  • Why/How lines only with what the conversation says.
  • Touch only memories this conversation adds to: never unrelated ones, and never "could not verify" notes.
  • Memories written while the conversation ran are listed under "Memories already saved during this conversation" so they are not duplicated.

Auto-dream (AUTODREAM_PROMPT):

  • Merge duplicates and prune clearly obsolete entries, as before.
  • Do not reword names or descriptions unless the content changed.
  • Never add placeholder Why/How lines.
  • Do not edit MEMORY.md by hand.
  • "Making no changes is the expected outcome of most passes."

Logging:

  • Memory extraction completed {sessionID, saved}.
  • Auto-dream consolidation completed {saved, deleted}, also on failure.
  • memory_delete now reports deletions to the coordinator, as memory_save already did.

Tests

  • test/prompt.test.ts: the deletion rule, the date rules, the unrelated-memory rule and the saved-during list.
  • test/extraction/ExtractionCoordinator.test.ts: the fork gets today's date and the memories written while the conversation ran.
  • test/extraction/autodream.test.ts: the summary log of saved and deleted files.
  • test/tools.test.ts: the memory_delete guidance, the memory_save content guidance, and that deletions are reported.
  • bun run lint, bun run typecheck, bun test and bun run build pass.

Real-environment verification (OpenCode 2.0.22, isolated DB/config, memory agents on opencode/gpt-6-luna; counts are honest, some are not wins)

1. Stale deploy.sh memory + "How do I deploy this project?"

  • main@ef15ddb deleted it in 0/6 runs (3 big-pickle, 3 luna), so the issue's 6/6 did not reproduce with these models. There is no before/after delta here; the change follows the issue's analysis of sections.ts.
  • An intermediate wording of this PR still saw big-pickle delete it once (1/3, judging the note "wrong").
  • Final wording: 0/10 deletions (big-pickle 0/7, luna 0/3). In 4 runs the memory was updated through memory_save with what was observed ("not found … as of 2026-10-07").
  • In 1 big-pickle run, the main agent edited the hand-written MEMORY.md line with its own edit tool. Nothing in the plugin can prevent that.

2. Extraction after a read-only question (decorators):

  • The unrelated memory was unchanged in 3/3 runs.
  • In 1 run the main agent wrote a memory itself through the shell. The fork saw it in the saved-during list and did not duplicate it.

3. "Last week we finished migrating billing to Postgres 17":

  • Extraction fork: 6/6 wrote "last week, as of 2026-10-07", with no invented date and the unrelated memory untouched.
  • Main agent on gpt-6-luna: 2/2 correct.
  • Main agent on big-pickle: estimated a date in 12/12 runs ("~2026-09-30", "week of 2026-09-28", "early Oct 2026"), across four rounds of wording, including the memory_save content guidance. This is a model-adherence limit for that model, not fixed here.
  • main@ef15ddb was not measured for this scenario.

4. Auto-dream over 5 seeded memories (hand-written index under ## headings, a duplicate pair, an expired sprint goal):

  • main@ef15ddb, 2/2:
    • rewrote the unrelated no-db-mocks.md (new description plus an invented "How to apply:" line);
    • rewrote the merged memory with new Why/How lines;
    • replaced every surviving hand-written index line.
  • This stack, 3/3:
  • Each new-code pass logged e.g. {"saved":["short-replies.md"],"deleted":["terse-answers.md","sprint-goal.md"]}.
  • Footnote: in 2/3 new-code runs the dream reworded the description of the merged file. Its content changed in the merge, so the prompt allows this.

Refs #49

🤖 Generated with Claude Code

@kuitos
kuitos added this pull request to stack #55 October 7, 2026 14:53
@kuitos
kuitos force-pushed the fix/memory-edit-guardrails branch 2 times, most recently from 9d3d552 to ceedb02 Compare October 7, 2026 15:23
Reported in #49 and traced to the prompts:
- the main agent deleted a memory naming a missing script on a read-only
  "How do I deploy?" in 6/6 runs, following "update or remove the stale
  memory rather than acting on it";
- extraction turned "last week" into an invented "Oct 2, 2026" ("Convert
  relative dates to absolute"), appended an unverified note to a memory
  about an unrelated system, and duplicated a memory the main agent had
  just saved;
- auto-dream rewrote every description ("Rewrite vague descriptions") and
  added "**Why:** not recorded" lines ("ensure content is structured as").

Prompts:
- main agent: answer from what is observed now; update a stale memory with
  what was observed instead of deleting it; delete only on request or when
  clearly wrong and superseded (memory_delete's description says the same);
- extraction: resolve a relative date only when it can be worked out from
  today's date (now passed with the transcript), else keep the wording;
  touch only memories this conversation adds to; list the memories written
  while the conversation ran so they are not duplicated;
- auto-dream: keep names and descriptions unless the content changed or the
  description is misleading, never add placeholder Why/How lines, keep
  merging duplicates and pruning clearly obsolete entries.

Logging: extraction and auto-dream log the memory files their fork saved
and deleted (memory_delete now reports deletions like memory_save does).

Refs #49
@kuitos
kuitos force-pushed the fix/memory-edit-guardrails branch from ceedb02 to be1bff1 Compare October 7, 2026 15:33
@kuitos
kuitos marked this pull request as ready for review October 7, 2026 15:35
@kuitos
kuitos merged commit 9947099 into main Oct 7, 2026
6 checks passed
@kuitos
kuitos deleted the fix/memory-edit-guardrails branch October 7, 2026 15:35
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.1.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant