Skip to content

fix(bundle): a bundle rewrites the ids a step's context paths name, as it does every other id - #653

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/context-fact-path-refs
Nov 13, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/context-fact-path-refs

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

#641 records the ids a bundle gives the entities a task names (Task.entity_ids). Context facts then looked a missed key up through that map at run time. Only code that calls the helper got that translation, so every other reader of a task author's ids would have needed the same call.

The framework already has a general mechanism for this. A step type declares the fields that name entities (entity_refs), and the bundle resolver rewrites them before the run; nothing at run time reads the declarations. Context-fact paths escaped it only because a path isn't an id field. This change brings them in.

  • Steps declare path fields. TaskStep.context_paths names the fields that hold paths into a run's context. Like entity_refs, it uses [] / {} for each element or value and isn't inherited. rubrics_verifier and always_on_agent_verifier declare context_facts_for_judge{}.
  • A path's env follows the task's env fields. sync_context_paths rebuilds each declared path from the one the task wrote, giving its env the id the task's env fields hold now. It runs after each place that rewrites those fields: resolving, pointing an installed alias, repointing a variant, and importing. Running it again changes nothing.
  • Only an env segment changes. ENV_KEYED_METADATA records where each metadata map names an env: the segment after env_trigger_state, env_changes, env_trajectory and live_websites, and the third segment of env_skills, after the agent's name. Only a path whose env is renamed is written. A key that only shares an entity's name is kept, such as solver in verifications.solver.score when an agent is called solver, or an agent named like an env in env_skills.
  • A path can hold any id. A rewritten id contains @, / and ~, so a segment may be an id in brackets, for example env_trigger_state[@local/~/runs/pn/world].statuses.wf-1. A plain dotted path reads as before. context_path_segments and renamed_context_path in agent_env.task_step.context parse and rewrite paths.
  • Lookups are plain again. fix(rubrics_verifier): a context fact naming an env as the task wrote it resolves in a bundle run #641's retry of a missed key through entity_ids is gone. The judge sees each fact's label and value, never its path, so its prompt doesn't change.
  • Registration carries over. bundle register renames such ids to registry ids too, and refuses an @local id left inside a path.
  • Importing carries over. Importing a stored task into a folder rewrites each id a field names to the folder's name for that entity, which gets a number when the name is taken. A declared path now takes the same name. Without that, re-importing a bundle-built task left the old bundle's @local ids in its paths, and a fact whose env was renamed crm-env-2 still named crm-env. The import leaves out scoped fields and masks credentials before it keeps the steps a path is rebuilt from, so a masked credential stays masked.
  • A missing declaration is reported. Resolving warns of a path that names a renamed env, by the same rule, in a field its step type doesn't declare. A file name like tickets.json isn't reported.
  • Scripts get the same. An outcome-verifier script is code, so nothing rewrites it before the run. Instead, Run.firings(env=…) and Run.firing(…, env=…) take an env as the run's task names it.
  • Bundles written before this change pick it up. The ledger hashes a task's resolved config, and this change alters it, so a stored task is written again ("config changed"). Nothing else is rewritten.

Run data keeps the ids the run used, so one id names one entity everywhere. Task.entity_ids stays, for tools that read a stored run.

How it was tested

Unit:

  • tst/unit/task_step/context_path_test.py:

    • a path parses into its segments, and a bracketed id is one segment;
    • any other string isn't a path;
    • renaming brackets an id only when a dotted path can't hold it, and leaves other paths and strings unchanged.
  • tst/unit/bundle/resolve_test.py:

    • a fact naming a bundle env names it by the bundle's id, in env_skills at its third segment;
    • a fact naming a store env is unchanged, and so is a verifier key that shares an agent's name, an output under an env-keyed key, and an env_skills agent named like the env;
    • syncing again changes nothing;
    • a path in an undeclared field is warned of, and a file name isn't.
  • tst/unit/bundle/installed_entities_test.py: a fact naming an env by an installed alias names the env the alias names. The test fails without the change to planning.

  • tst/unit/bundle/variants_test.py: a fact naming the env by id is repointed at the variant.

  • tst/unit/bundle/register_test.py: a bracketed @local id in a path takes its registry id, and an @local id left in a path is refused.

  • tst/unit/bundle/importer_test.py:

    • when two envs share a last segment and one folder is numbered crm-env-2, each fact names its own env's folder;
    • a credential in a path whose env is renamed stays masked.

    Each test fails without its importer change.

  • tst/unit/bundle/ledger_test.py: a task written by a build that didn't rename its paths is written again ("config changed"), then reused.

  • tst/unit/task_step/verifiers/test_rubrics_verifier_context_facts_bundle_ids.py:

    • a bracketed id is one key;
    • a path resolves only as written, and a string that isn't a path resolves to nothing;
    • the step keeps bracketed paths and refuses malformed ones;
    • the judge's prompt shows the fact.
  • tst/unit/task_step/verifiers/test_run_context.py: firings(env=…) finds a renamed env by the id the task names it by.

  • tst/unit/task_step/test_entity_refs.py:

    • the built-in context_paths name real fields;
    • a subclass declares none until it does;
    • field_sites reaches each value or element of a last {} or [].

tst/unit/bundle, tst/unit/task and tst/unit/task_step pass: 4,565 passed, 14 skipped.

End to end. A benchmark task with a scripted world (timed events and colleague agents) was run as a bundle with agent-env run on a Linux VM. The build combined this branch with #640, so a validity verifier can fail the task.

A dry run of a folder an earlier revision of #641 wrote planned only its task anew: v2 (its stored version doesn't record the ids the bundle gave what it names; config changed). The first reason is there because an earlier revision of #641 wrote that folder. Its envs and artifacts were unchanged.

Folder World Its task What the validity judge wrote Result
written by an earlier build broken: one scheduled event delayed past the run rewritten as v2; its paths name env_trigger_state[@local/…/<env>] "The context facts list 'wf-3 status: armed', so wf-3 never fired." v-world failed, and the run failed at validity with VerifierNotPassingError
fresh copy intact v1 "Context facts: wf-1, wf-2, wf-3, wf-4 and wf-5 are all 'fired'." validity 1.0; the scoring step ran

In both runs no fact rendered "(not available)", and the facts match the trigger statuses the run stored under the namespaced env id.

Import round trips and adversarial cases, on the same VM against its real local store. Each case checks that every @local id in a resolved fact path is an env id the resolved task deploys:

Case This branch Without the importer change
Import a stored bundle-built task whose paths hold bracketed ids, then resolve the folder pass: the paths name the new folder's env fail: the paths still name the old bundle's env
Two envs with the same last segment; the import numbers one -2 pass: each fact names its own env fail
In the same task, verifications.coopbench-star-pm-v2.score, a verifier key sharing the env's name pass: kept through import and resolve fail on this PR's previous commit, which renamed it
A bundle root with dots in it (chaos.dot.v2) pass: the bracketed id parses as one key fail
An undeclared field holding a path, beside a file name coopbench-star-pm-v2.json pass: one warning, for the path only pass

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

The PR rewrites declared context paths when bundles rename environments, including during import, registration, alias lookup, and variant creation.

  • Bundle tasks now match context-fact paths to their chosen environment ids.
  • Run firing queries accept environment names from the task.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Authored task paths] --> B[Rewrite declared env fields]
  B --> C[Find each env's new ID]
  C --> D[Rebuild declared context paths]
  D --> E[Store task]
  E --> F[Read facts exactly as written]
Loading

Reviews (8) · Last reviewed commit: "fix(bundle): env_skills paths name their..." · Reviewed by Greptile

…s it does every other id

A step type declares the fields that hold paths into a run's context (TaskStep.context_paths), and the bundle
resolver renames each id such a path names to the one the bundle gives that entity, before the run. A renamed id
that a dotted path can't hold is bracketed: env_trigger_state[@local/<root>/<env>].statuses.wf-1.

- rubrics_verifier and always_on_agent_verifier declare context_facts_for_judge, whose lookup is plain again.
- bundle register renames such ids to registry ids, and refuses an @Local id left inside a path.
- A path naming a renamed id in a field its step doesn't declare is warned of when the bundle is resolved.
- Run.firings(env=...) and Run.firing(..., env=...) take an env as the run's task names it.
- A task written before this is written again: its resolved config, which the ledger hashes, changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 12, 2026 22:42
Comment thread src/agent_env/bundle/resolve.py Resolved
earakely-scale and others added 2 commits November 12, 2026 14:51
…r names it

A task's import rewrites each id a field of its steps names to the name the folder gives that entity, numbered when
the name is taken. A declared path into the run's context now takes the same name, so a fact naming an env the folder
writes as crm-env-2 names it so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… env fields, and nothing else in it changes

A path is rebuilt from the one the task wrote, after each place that rewrites a task's env fields: resolving, pointing an
installed alias, repointing a variant, and importing. Only the segment that names an env changes: the one after a key
whose entries are keyed by env id (ENV_KEYED_METADATA). A key that only shares an entity's name, such as a verifier's id,
is kept, and an env named by an installed alias is named by what the alias names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/bundle/resolve.py Resolved
Comment thread src/agent_env/bundle/importer.py Resolved
…ds a path back from before masking

- ENV_KEYED_METADATA records where each key's entries name an env: the segment after env_trigger_state, env_changes,
  env_trajectory and live_websites, and env_skills' third segment, after the agent's name.
- A path is written only when its env is renamed.
- An import leaves out scoped fields and masks credentials before it keeps the steps a path is rebuilt from, so a
  credential masked in a path stays masked when the path's env is renamed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit cc0f248 into main Nov 13, 2026
61 of 64 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/context-fact-path-refs branch November 13, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant