Skip to content

fix(extraction): put the extraction instructions in the fork's prompt text - #52

Merged
kuitos merged 2 commits into
mainfrom
fix/frame-extraction-transcript
Oct 7, 2026
Merged

kuitos merged 2 commits into
mainfrom
fix/frame-extraction-transcript

Conversation

@kuitos

@kuitos kuitos commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

On V2, the extraction fork's only user message was the raw ### User / ### Assistant transcript (ExtractionCoordinator: system: buildExtractionSystemPrompt(...), text: conversation). The task itself lived only in the system prompt. #48 (point 1) reports that Claude Haiku 4.5, GPT-5.4 Mini and GPT-5.4 Nano read the transcript as a conversation waiting for a reply and answered its last question instead of extracting, in 9/9 runs. One fork explained Python decorators; after a coding turn, another re-ran pytest. Recall already restates its task in the prompt text (buildGeneratePrompt); extraction did not.

Fix

buildExtractionUserMessage() (src/extraction/prompts.ts) builds the fork's prompt text:

  • a header that states the task and says the transcript is data to read, not a conversation to answer or continue;
  • the transcript inside a <transcript> block, with any quoted </transcript> escaped so it cannot end the block early;
  • a closing reminder: the only job is memory extraction; do not reply to the user in the transcript or answer their questions.

The system prompt is unchanged. The auto-dream fork's message restates its task the same way. Both hosts send the same text, through MemoryHost.runFork.

(The V2 fork-sandbox half of #48 is handled in a separate PR.)

Tests

  • test/extraction/ExtractionCoordinator.test.ts:
    • the message delimits the transcript and restates the task after it;
    • a quoted closing tag does not end the transcript early;
    • the fork receives the framed message.
  • test/v2/plugin.test.ts: the V2 fork prompt starts with the task and contains the closing reminder.
  • bun run lint, bun run typecheck, bun test and bun run build pass.

Real-environment verification (OpenCode 2.0.22, isolated DB/config)

Setup:

  • The main agent (opencode/big-pickle) was asked a question about Python decorators, with memory_save, memory_delete and external_directory denied so that only the fork could save anything.
  • The extraction fork ran on opencode/gpt-6-luna.
  • Measured on the top of this stack.

Results:

  • 3/3 forks called only memory_list / memory_save and ended with a one-line summary.
  • None answered the decorator question or touched the repo.

Limitation, stated plainly: the models that failed in #48 (Haiku 4.5, GPT-5.4 Mini/Nano) could not be run here. Zen's free models return 403 for sandboxed forks, and gpt-6-luna may well have extracted correctly before this change too. So this run shows no regression, not the fix itself. The fix follows the issue's diagnosis: the same framing recall already uses.

Refs #48

Also: flaky V2 fork timeout (CI fix)

runV2Fork checked remaining() <= 0 after a TimeoutError, but a deadline timer can fire a millisecond before Date.now() reaches the budget. The timeout was then rethrown as a plain error and the fork was removed without being interrupted, which made runV2Fork > a hung wait times out… and V2 setup: extraction > ⑧ fail intermittently on ubuntu. A deadline that ran on the fork's whole remaining budget now always counts as the fork timeout. New regression test freezes Date.now() with setSystemTime and fails without the fix.

🤖 Generated with Claude Code

@kuitos
kuitos added this pull request to stack #55 October 7, 2026 14:53
@kuitos
kuitos force-pushed the fix/frame-extraction-transcript branch from 4ecdc41 to 8f64339 Compare October 7, 2026 15:19
Base automatically changed from fix/keep-hand-written-index-lines to main October 7, 2026 15:23
… text

The extraction fork's only user message was the raw `### User / ### Assistant`
transcript; the task lived in the system prompt alone. Fast models (Claude
Haiku 4.5, GPT-5.4 Mini/Nano) read it as a conversation waiting for a reply
and answered its last question instead of extracting, in 9 of 9 reported
V2 runs.

The fork's prompt text now states the task, wraps the transcript in a
<transcript> block (a quoted closing tag is escaped), and ends with a
reminder to only record memories and not answer or continue the
conversation. The system prompt is unchanged. The auto-dream fork's
message restates its task the same way. Both hosts send the same text.

Refs #48
@kuitos
kuitos force-pushed the fix/frame-extraction-transcript branch from 8f64339 to 227b6d6 Compare October 7, 2026 15:23
@kuitos
kuitos marked this pull request as ready for review October 7, 2026 15:24
A deadline timer can fire a millisecond before Date.now() reaches the
fork's budget. remaining() then still read > 0, so the TimeoutError was
rethrown as a plain error: the fork was removed without being
interrupted. This made 'a hung wait times out, interrupts, then removes'
and the V2 setup timeout test flaky on ubuntu CI.

A deadline that ran on the fork's whole remaining budget now always
counts as the fork's timeout.
@kuitos
kuitos merged commit a2c4e34 into main Oct 7, 2026
6 checks passed
@kuitos
kuitos deleted the fix/frame-extraction-transcript branch October 7, 2026 15:35
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.1.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant