Repository navigation
fix(rubrics_verifier): a context fact naming an env as the task wrote it resolves in a bundle run #641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
fix(rubrics_verifier): a context fact naming an env as the task wrote it resolves in a bundle run #641
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
bc74d18
fix(rubrics_verifier): a context fact naming an env as the task wrote…
earakely-scale 95488f1
fix(task): a task carries the ids its bundle gave what it names, and …
earakely-scale d4cbaf1
fix(bundle): rewrite a task whose stored version predates its entity_ids
earakely-scale 9a5b7de
fix(bundle): entity_ids are the ids a task's own fields were rewritte…
earakely-scale File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| """Task.entity_ids: the ids a bundle gave what a task names, stored with the task and put in each run's metadata.""" | ||
|
|
||
| import pytest | ||
|
|
||
| from agent_env.config.runtime import Config | ||
| from agent_env.task import Task | ||
| from agent_env.task_step.context import ENTITY_IDS_KEY, TaskStepContext, run_entity_id | ||
| from agent_env.task_step.task_step import TaskStep | ||
|
|
||
| IDS = {"world": "@local/~/runs/pn/world"} | ||
|
|
||
|
|
||
| class _Seen(TaskStep): | ||
| """Records the metadata its run reached it with.""" | ||
|
|
||
| type = "entity_ids_test" | ||
| entity_refs = () | ||
| seen: list[dict] = [] | ||
|
|
||
| @classmethod | ||
| def from_dict(cls, data): | ||
| return cls(**cls._base_from_dict(data)) | ||
|
|
||
| async def execute(self, context): | ||
| _Seen.seen.append(dict(context.metadata)) | ||
| return context | ||
|
|
||
|
|
||
| @pytest.fixture(autouse=True) | ||
| def registries(monkeypatch): | ||
| steps = {**Config().task_step_registry(), _Seen.type: _Seen} | ||
| monkeypatch.setattr(Config, "task_step_registry", lambda self: steps) | ||
| monkeypatch.setattr(_Seen, "seen", []) | ||
|
|
||
|
|
||
| def _task(**kw) -> Task: | ||
| return Task(id="t", version=None, steps=[_Seen(id="s", version=None)], **kw) | ||
|
|
||
|
|
||
| def test_they_are_stored_with_the_task_and_left_out_when_there_are_none(): | ||
| assert Task.from_dict(_task(entity_ids=IDS).to_dict()).entity_ids == IDS | ||
| assert "entity_ids" not in _task().to_dict() | ||
| assert "entity_ids" not in _task(entity_ids={}).to_dict() | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_a_run_puts_them_in_its_metadata_for_run_entity_id(local_stores): | ||
| await _task(entity_ids=IDS).run(context=TaskStepContext()) | ||
|
|
||
| metadata = _Seen.seen[0] | ||
| assert metadata[ENTITY_IDS_KEY] == IDS | ||
| assert run_entity_id(metadata, "world") == "@local/~/runs/pn/world" | ||
| assert run_entity_id(metadata, "elsewhere") == "elsewhere" | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_a_task_stored_as_written_adds_nothing(local_stores): | ||
| await _task().run(context=TaskStepContext()) | ||
|
|
||
| assert ENTITY_IDS_KEY not in _Seen.seen[0] | ||
| assert run_entity_id(_Seen.seen[0], "world") == "world" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.