Skip to content

fix(a2a_agent): bounded retry after a trajectory turn ends, and alertable read failures - #691

Draft
andrewpark-scaleai wants to merge 1 commit into
mainfrom
andrew-park/rel-trajectory-follower-backoff
Draft

andrewpark-scaleai wants to merge 1 commit into
mainfrom
andrew-park/rel-trajectory-follower-backoff

Conversation

@andrewpark-scaleai

Copy link
Copy Markdown
Contributor

Why

AgentEnv should have enough logs, metrics and alerts to debug every failure, and heal itself where code can. The live trajectory follower fails that bar in two ways.

  1. Once the turn has ended (stop set), a failed read gives up on the first failure. One transient 5xx or dropped connection loses the rest of the live trajectory. Before fix(a2a_agent): a trajectory follower ends with its step #618, the same branch retried every 0.2 s with no limit and no backoff, so a gone agent made the follower spin.
  2. The retry warning is an unstructured string (consec=%d inside the message). A spinning follower cannot be alerted on, because nothing is searchable and the level never rises.

What

  • After stop is set, a failed read is retried up to 3 consecutive failures, sleeping 1 s then 2 s between them. A gone agent frees the step after about 3 s. A read that succeeds resets the count.
  • Each failed read logs event="live_trajectory_read_failed" with consec and task_id in extra. At 10 consecutive failures and above the level is ERROR instead of WARNING, so an alert can key on it.
  • Giving up after the turn ends logs event="live_trajectory_follow_abandoned" with consec.

Design decision and tradeoff

The drain is bounded: 3 failures, about 3 s of waiting. A longer drain keeps more of the trajectory tail when the agent is briefly down, but holds the step's activity slot while it waits on an agent that is probably gone. A shorter drain, which is what main did (stop on the first failure), frees the slot at once but loses the tail on any blip.

Losing the tail is cheap here. The step marks the turn's end when its follower stops early, and the final trajectory still holds every event, so only the live view comes up short. Holding a slot is not cheap. The bound errs toward freeing the slot while giving one blip a chance to pass.

Test

PYTHONPATH=src:packages/agentenv-protocol/src python -m pytest tst/unit/a2a_agent/test_trajectory_follower.py -q

25 passed. Against the old follower source, 3 of the new or changed tests fail:

  • test_an_agent_that_fails_after_the_turn_ends_is_retried_a_bounded_number_of_times (always-failing agent, stop set: ends within 4 reads)
  • test_an_agent_that_recovers_after_the_turn_ends_still_gets_its_tail_read
  • test_a_failing_read_is_logged_with_its_count_and_escalates_to_error (checks event, consec, level via caplog)

I ran these in a venv without pytest-socket (no package index here), so conftest.py imported a no-op stub of it. The full unit suite was not run.

…mes after the turn ends, and make read failures alertable

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant