Skip to content

Evaluation, dashboard and onboarding: one match tool, a Strategies tab, docs cut to 10 - #153

Merged
isaac0804 merged 59 commits into
mainfrom
infra/evaluate-dashboard-onboarding
Nov 13, 2026
Merged

Evaluation, dashboard and onboarding: one match tool, a Strategies tab, docs cut to 10#153
isaac0804 merged 59 commits into
mainfrom
infra/evaluate-dashboard-onboarding

Conversation

@isaac0804

@isaac0804 isaac0804 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

So a new member can write a strategy, see how it plays and know if it is better, without learning a dozen tools first. Start at tutorials/first_strategy.md.

What changes

Evaluation

  • One match tool: pixi run evaluate NAME. Round 1 is a round robin of the core strategies (the health check); with a candidate, it screens on seeds 1-6 and accepts only on fresh seeds 101-106 (paired t-test). Three flags: candidates, --at REF, --all.
  • The reference is the champion at the last release, played from a .strategy file, so a change to shared code never moves it.
  • One match results store (replays/match_results/): a match it holds is never played again.
  • Fixed reference opponents, world and robot settings (tools/evaluation/reference/); the sim keeps a measured world and assumes league-level robots (2 m/s, one kick power).
  • Removed: the screen, the restart fuzzer, the situation bench, round_robin.py and other one-question tools; triage is find_bugs, the goal ledger is goal_sources.

Strategies

  • Twelve strategies named after their modules; ten retired ones deleted (old names still load today's strategy).
  • Fixes in shared skills, tactics, ball handling and the referee's restart rules.

Dashboard (pixi run dashboard)

  • The Strategies tab: each strategy's description, verdicts, position and shot maps, signals beside the average, and its matches, playable. Light theme.

Structure (renamed packages, no aliases)

  • custom_referee/ → referee/, rsoccer_simulator/ → simulator/, shared/ and global_utils/ → geometry/, no more src/ folders; tests mirror the package names; tools grouped by who uses them.

Docs

  • 27 files down to 10. tutorials/ holds step-by-step pages (first strategy, field day), docs/ the reference.
  • What was learned now lives in the docs: "Tried and rejected" and "Known open bugs" (docs/strategy.md), "Our robots" (docs/real_robots.md).
  • Agent playbooks moved to .agents/skills/ (any agent; .claude/skills is a symlink), listed in AGENTS.md.

Release notes

  • Imports from custom_referee, rsoccer_simulator, global_utils, *.src.* and strategy.examples must be updated.
  • Results are not comparable with earlier versions.
  • After merge: the round-1 replays are attached to v2.6.0 as replays-v2.6.0.zip (the tutorial links to it).

Tests

pixi run test: 13613 passed, 83 xfailed. pixi run lint clean.

isaac0804 and others added 30 commits November 10, 2026 17:08
@isaac0804 @claude
sim: keep the world measured and assume league-level robots
641dcaa 
The simulator's settings fall into two kinds. The world (ball friction, vision noise and
dropouts, latency) comes from measurements; robot capability (kick, dribbler, motors) is assumed
good at league level, with real robots' shortfalls handled in the control layer. Drop per-robot
real kicks from rsim, and have the sim's commands compensate for its latency as real robots do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
referee: fix restart rules and the free-kick actions
0659239 
Fixes to the in-process referee and the restart actions robots run under it, with tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
strategy: fix shared skills, tactics and ball handling
03c5697 
Sprint at contested balls, fix the free-kick first-touch stall and ball release, and
improve approach, goalkeeping and man marking. Sim results for these are in the frozen
evaluation against v2.5.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
evaluation: fixed reference opponents, world and robot settings
33defac 
Replace pool_v1 with tools/evaluation/reference/ (opponents, world, robots) and add the
field-physics tooling that the world settings come from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
strategy: name each strategy after its module
7af415d 
A strategy is now strategy/<name>.py with one build(outfield_robot_ids, motion_controller),
called <name> everywhere. strategy/catalog.py finds them, so a new strategy is a new file;
kernel_strategy.py and the build_*_kernel_strategy factories are gone. Names in old replays
and summaries still resolve through catalog.plain_name.

fingerprint.py still reads the removed registry until the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
evaluation: .strategy files, a frozen baseline and one fingerprint fo…
119e1fd 
…r both

A .strategy file is one strategy with the whole utama_core package at one commit, playing in its
own process. evaluate.py --frozen REF plays the champion and opponents from such files, so
a shared-code change moves only the candidate. An identity match, one side bundled, was
byte-for-byte equal to the normal match.

A player's fingerprint is now the closure of its code plus compiled files, with no name in it,
computed the same way for a file and for code, and the ledger is keyed by it. Code behind the
process boundary leaves the environment, so a skills edit replays no match between two files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
bundle: let a .strategy file drive real robots
bcbbe2c 
Send per-robot kick bend and bias to the file's process, apply the kicker-charge step on
the runner side, and let a run name the robots it has. Unit tested; not yet run on the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
tools: one tool per question, stalls in the ledger, a cheap health check
e16649d 
- docs/strategy_tools_plan.md: which tool answers which question, the
  remaining steps, and the screen's proposed pass mark (not yet agreed).
  tools.md opens with a "Which tool?" table; simulator realism tools get
  their own section; debugging starts at triage.py; --reuse counts fixed.
- Ledger entries record each match's stalls; evaluate.py reports them
  with coverage (the watchdog sees config_a only; old entries unknown).
  Verdicts are unchanged.
- round_robin.py --vs-champion: each strategy once against the champion
  (12 matches, not 66), reference noise seed 2 by default so those are
  evaluate.py's round-2 fixtures.
- tools/sim_setup.py: one way to load a strategy (catalog name,
  .strategy file, PATH.py:FUNCTION) and set up an rsim match, used by
  repro_from_replay, moments, debug_match and main.py. match.py is
  untouched, so cached matches and the ledger stay valid.
- main.py: fix its strategy loading (it failed on import).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
strategy: skip short open-play passes, keep one attacker upfield in d…
5b623b0 
…efence

- Open-play passes go only to teammates at least 1.5 m away
  (MIN_OPEN_TEAMMATE_M, was 0.7): 28% of passes under 1.5 m were received
  against ~50% from 1.5-3.5 m; paired drills +1.7 +/- 0.6 points GD.
  GiveAndGoTactic uses the same constant.
- Restarts keep the old 0.7 m minimum (RESTART_MIN_TEAMMATE_M): a free
  kick's receiver waits for a still ball, so short restart passes work.
- split_shape, defending with four or more robots, keeps a second robot
  in attack holding space upfield (_defending_attack_count): +2.6 +/- 0.9
  points scoring in paired drills, no more conceded, but +7.3 +/- 2.5
  points more free kicks given away.

Not yet judged by evaluate.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fc3fwJXC1cohtBMpWGMjt3
@isaac0804 @claude
examples: fix demos broken by the strategy renames
3b5939f 
- Four examples still passed AbstractStrategy(build_kernel_strategy=...),
  renamed to build= on this branch; two imported the removed
  utama_core.strategy.kernel_strategy, and demo_exhibition_road.py the
  long-deleted WanderingStrategy. They use catalog.builder and
  wandering_strategy now.
- test_every_example_imports imports each example, so a removed module
  fails the suite (it failed on three demos before the fix).
- round_robin.py and triage.py no longer describe build_*_kernel_strategy
  factories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: the screen, on trial; fouls move into triage; moments.py trimmed
a3f387e 
- tools/screen.py STRATEGY --base REF: one strategy now against itself at
  REF (a .strategy file) from the same 60 starts of 20 s of live play drawn
  from a run's replays, both against the base under the same noise. Reports
  how many starts fired (diverged), paired differences in goals and ball
  position, and a call by a rule fixed before any change was screened.
  Never writes the ledger; appends to replays/eval/screen_log.jsonl. Same
  code, file against code, never diverged on 3 real starts.
- Pass mark (agreed 2026-11-11): a track record on real changes. The screen
  decides nothing until 8 clear evaluate.py verdicts with no "skip" on a
  better change and at least half of the worse ones skipped.
- triage.py --fouls replaces moments.py fouls (utama_core/analysis/fouls.py).
- moments.py: `new` and its template removed (moments come from replays);
  --against worktrees in <repo>-moments-against/, not a dated folder.
- tools.md (also the five scripts it did not list), STRATEGY_DEVELOPMENT.md,
  the plan note and the glossary updated.

Fingerprints of every strategy and the environment are unchanged, so no
cached match or ledger entry is invalidated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: screen in about 4 minutes; first track-record entry
d4d090f 
- Starts are picked from each replay's referee sidecar, read once, instead
  of rebuilding a scenario for every draw: 83 s -> 3.5 s, the same rule (the
  last referee command at or before t).
- --max-workers defaults to 12 (8 left half of 16 cores idle): 60 starts in
  about 4 minutes.
- strategy_tools_plan.md: measured cost, a track-record table with the first
  real change (split_shape at 5b623b0: fired 60 of 60, WORTH EVALUATING,
  evaluate.py not run yet), and what that run shows, not acted on: every code
  change fires within 0.1 s, and 120 starts hold only 10 goals.
- roadmap.md, tools.md, STRATEGY_DEVELOPMENT.md: the screen and its cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: remove the screen, which could not save evaluations
49568ce 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
replay: one match results store, reuse automatic
89c036b 
The match cache (round-robin --reuse) and the evaluation ledger become one store,
replay/match_results.py, under replays/match_results/<environment>/<fixture>.json.
A match is looked up by its fixture before it is played and stored after; --reuse,
--spot-check and the spot-check are gone. match_key moves into the fixture and the
environment. Old stores are no longer read; deleting the store replays everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: one match set-up in sim_setup for every tool
ea83269 
match.py, repro_from_replay.py, moments.py, debug_match.py and main.py build their
referee and rsim runner through tools/sim_setup.py: sim_referee (kickoff team,
seeded restart fuzzing) and sim_runner (the StrategyRunner). main.py keeps its
dashboard: the referee is built before it attaches, the runner after. load_side
plays a .strategy file with the robots it was built for. The fingerprints of every
match change with this, so the next round-robin replays everything once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: rename triage to find_bugs, goal ledger to goal sources
c44a34e 
tools/triage.py is tools/find_bugs.py (all flags kept; default output RUN_DIR/find_bugs/).
utama_core/analysis/goal_ledger.py is goal_sources.py (python -m utama_core.analysis.goal_sources);
its output is goal_sources.json. Tests, dashboard, docs, the glossary and the match data flow
picture follow. Historical notes keep the old names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
strategy: delete the ten retired strategies
4bc09bc 
Nobody knew whether tiki_taka, counter_press, high_line_zone, three_slot,
switch_of_play, low_block, zone_fluid, score_aware_zone_flow, overload_flow
and shadow_switch still worked, so they go rather than come back. The
RETIRED list goes with them: the catalog is every module in strategy/.
holds_ball (three_slot's only) goes from pickers.py. DefenseTactic stays:
examples/shooting_drill.py (pixi run demos) uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
strategy: name the twelve strategies
a9aee7b 
A name says who a strategy is, not what it does, so it stays true when the
strategy changes; the first line of each module says the style. pendulum
(was split_shape), breakwater (clear_press_plus), avalanche (overload_press),
wolfpack (high_press), mirage (decoy_and_overload), riptide (counter_flow),
chameleon (score_aware_counter_flow), tripwire (press_trigger_flow),
firebreak (clear_danger), terrier (press_and_pass), weaver (tiki_taka_plus),
ricochet (give_and_go_solo). catalog.RENAMED maps the old names on, so old
replays and summaries still load; replay files keep their names, and lines
in docs and comments that name an old replay or match keep the old names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tests: old strategy names load today's strategy
410b5c5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: one match tool, evaluate.py, with a round robin as round 1
897ae88 
evaluate.py now plays every match. With no candidate it plays one round: a
round robin of the core strategies (reference/players.json), two reserve
strategies rotated in by the round's seed (--all: every strategy) and the
reference, the champion at the last release played from a .strategy file
(--at REF, --compare-with NAME). With candidates, later rounds play only
the undecided candidates' matches until the paired verdict is clear.

It reports stalls with how many matches checked them and every signal the
round robin printed, never standings. Each round writes
replays/eval_<id>/round_<k>/ in the old round-robin layout, so find_bugs,
pitch_maps and goal_sources read it; report.json holds the verdicts.

round_robin.py, its --pair/--vs-champion/--both-sides/--no-save, the
held-out opponents and opponents.json (now players.json) are gone.
--strict, --fuzz-restarts and --control-scheme moved over. The dashboard,
list_runs and package_logs find the new run directories as well as the old.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: evaluate.py plays the core only; the reserve joins with --all
89cd19b 
The verdict compares candidate and reference against the same opponents, so it
needs no reserve rotation; the wider field is an occasional --all run. Round 1
is 15 matches (five core and the reference), a later round 9 per candidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: every evaluate.py round is the same round robin
ce7145a 
The match results store already skips every match whose code is unchanged, so
a later round that plays only the candidates' matches saved nothing once the
store is warm: it was one more rule to explain. Every round now plays every
pair once under its seed; only the undecided candidates stay after round 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: evaluate.py keeps three flags: candidates, --at, --all
2278fca 
With every round the same round robin and the store choosing what plays, the
rest had no job: --compare-with (a second way to pick the reference),
--max-rounds (fixed at 6), --max-workers (CPUs - 1, at most 12), --strict (no
CI used it; the report lists every stall), --fuzz-restarts/--fuzz-interval
(referee bug hunting, not evaluation; the fuzzer stays in sim_setup),
--control-scheme (planners are compared by motion_planning_benchmark), and
--explain (explain.py runs on any run dir).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: a round folder holds every match of its round
b5629db 
The store now records where each played match's files are; a run that reuses
a match links them into its own round folder under this run's names, so
find_bugs, explain, pitch_maps and goal_sources see the whole round, not only
what this run played. Two runs started in the same second no longer share a
folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
skills: explain-result, debug-incident, field-session
d4acc2c 
Three playbooks in .claude/skills/ over the existing scripts, which stay in
tools/: how a strategy plays and why it won or lost (find_bugs, goal sources,
pitch maps, explain); one bad moment from symptom to a fixed cause with a test
(replay_trace, repro_from_replay, the planner pictures, moments); a test day on
real robots and its logs (field-check, calibrate, trim, field_physics).
.claude/skills/ is committed; the rest of .claude/ stays ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
cleanup: drop the restart fuzzer and the fixed settings from the envi…
c128d98 
…ronment

Nothing turned the fuzzer on after evaluate.py lost --fuzz-restarts, and the
planner is always fpp, so neither belongs in a match's environment id. The
store is still empty, so changing the id replays nothing. ROUND_ROBIN_ENTRY
becomes MATCH_ENTRY (same value, so no fingerprint moves); the dashboard's
comments and three historical docs point at the strategy renames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
skills: change-strategy, and pointers from the docs
5e760dd 
The fourth skill: branch, change one thing, test, evaluate (ask first), record
and promote. STRATEGY_DEVELOPMENT.md keeps the reasons and points at the
skills for the steps; "Reading a tournament run" becomes "Reading a round";
explain-result writes its outputs into the round folder, not /tmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
pixi: upgrade the lock file to format v7
3cec922 
Same packages; pixi re-solved from the locked content only to rewrite the
format, which silences its v6 warning on every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
fix: a match's replay lands in its run dir, however deep
38c2afc 
run_match named the replay <run_dir.name>/<tag>, right only when run_dir
sat directly under replays/. evaluate.py's round folders are
eval_<id>/round_<k>/, so every run's replays went to replays/round_<k>/,
overwriting each other, and the round folders had none: no restarts or
ball losses, no replay to link, nothing for find_bugs or pitch_maps.
The test drives the real run_match up to the runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
fix: evaluate.py before its first real run
867a530 
- One player per code: right after a release the frozen champion is the
  working-tree champion's code; both played, the reference played itself
  and the store labelled their matches by either name (hash order), so a
  candidate could find no reference matches. A later duplicate is left out.
- The reference is the champion players.json named at the release, so a
  promotion takes effect at the next release instead of breaking runs.
- No release tag stops the run instead of comparing with today's code.
- The noise fit uses rounds 1..k only, like the pairs, so a verdict
  doesn't depend on what earlier runs left in the store.
- A run claims its folder with mkdir, so two runs in one second can't share it.
- A crashed match is reported (summary/report "failed"), not stored, and
  the run finishes.
- The docstring says the stopping-rule numbers come from a simulation of
  six opponents and SD 1.9, not re-checked for today's five and 1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
isaac0804 and others added 27 commits November 12, 2026 19:26
@isaac0804 @claude
evaluate.py: screen on seeds 1-6, accept only on fresh seeds (paired …
2da3601 
…t-test)

The verdict was a Bayesian posterior with three hand-set weights (a prior on the
difference, a prior pulling the diffs' spread toward unpaired noise, head-to-head
counted alongside the diffs) and P >= 0.99. Replaced by a rule with two settings:

- screen, seeds 1..ROUNDS: from round 2, a mean paired diff <= 0 is dropped;
  the screen never accepts
- confirm, seeds 101..100+ROUNDS, on their own diffs: the same drop rule, then
  better if the one-sided paired t-test gives p <= ALPHA, else undecided

Only fresh seeds accept, so a candidate selected or fitted on seeds 1-6 is not
accepted on that luck; --confirm is gone (confirm is automatic). Head-to-head is
reported, not counted. ROUNDS = 6 and ALPHA = 0.05 are provisional, to be set
from the store (spread of real diffs; base rate of better edits).

The rule lives in analysis/ratings.py (paired_test, decide), so verdict_sim.py
plays the same code without importing the simulator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: a first-strategy tutorial; delete research tools and stale docs
b9ee10f 
For onboarding new members:
- docs/first_strategy.md: watch a match, write a toy strategy by hand, test it,
  evaluate it (read the verdict and the mean diff), then let an agent write one.
  tests/strategy/test_first_strategy_doc.py builds its code so it stays runnable.
- README quick start: `pixi run main` and the tutorial; tools.md opens with the
  four commands a beginner needs.

Deleted (questions answered; nothing runs them):
- tools/metric_correlation.py and the five offline signals only it computed
- tools/signal_report.py, docs/signal_report.md and its figures; the palette the
  figure tools shared moves to tools/figure_style.py, and pitch_maps orders
  panels by name instead of points per match (no rankings)
- docs/strategy_tools_plan.md and the two investigation docs (code comments
  that cite them point into git history)

evaluate.py's docstring is cut to usage and a pointer to STRATEGY_DEVELOPMENT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
structure: drop the src/ layer in skills, motion_planning, team_contr…
f0312b2 
…oller, rsoccer_simulator

Moved everything under each src/ up one level and deleted the empty src/:
- utama_core/skills/src/* -> utama_core/skills/
- utama_core/motion_planning/src/* -> utama_core/motion_planning/
- utama_core/team_controller/src/* -> utama_core/team_controller/
- utama_core/rsoccer_simulator/src/* -> utama_core/rsoccer_simulator/

The empty src/__init__.py in team_controller was dropped; team_controller/__init__.py
already existed (both empty). Compiled robosim/fastpathplanner extensions (gitignored)
moved with their directories. Dotted and slash references updated across the repo.

Co-Authored-By: Claude Haiku <noreply@anthropic.com>
@isaac0804 @claude
structure: shared/ and global_utils/ become geometry/
5fda003 
Geometry helpers move into utama_core/geometry/ (the old shared/ package, __init__ included):
- shared/pass_and_score_geometry.py, tolerance.py, field_scaling.py -> geometry/
- global_utils/math_utils.py -> geometry/math_utils.py

Other global_utils modules go next to their users, not into geometry/:
- global_utils/git_utils.py -> replay/git_utils.py (revision stamp, used by tools/)
- global_utils/mapping_utils.py -> entities/game/mapping_utils.py (friendly/enemy to yellow/blue
  mapping; used by run/, replay/ and data_processing/, and entities/ is the one layer all of
  them already import, so it adds no import cycle)

shared/ and global_utils/ are deleted. Tests: tests/shared/ and tests/global_utils/ move into
tests/geometry/ (test_mapping_utils.py too, as one folder).

Dotted and slash references updated across code, tools, examples, docs and agent skills,
including the fingerprint PLAYER_SIDE entry for pass_and_score_geometry. docs/architecture.md
graph nodes were renamed by hand; its edge counts are stale until tools/package_graph.py is
rerun.

Co-Authored-By: Claude Haiku <noreply@anthropic.com>
@isaac0804 @claude
tests: test_mapping_utils next to its module in entities/
b60af11 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tests: mirror the package names under utama_core/tests
a223b1e 
- controller/ -> team_controller/
- referee/ -> custom_referee/ (one __init__.py, kept in custom_referee/)
- receivers/ and refiners/ -> data_processing/
- abstract_strategy/test_assertions.py -> engine/test_abstract_strategy_assertions.py
- strategy_runner/ -> run/
- every *_test.py renamed to test_*.py (no basename clashes)

Path and module references updated in docs, AGENTS.md, code comments and
the tests that import helpers from the moved modules.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tools: group scripts by who uses them
3e64169 
- tools/debug/: find_bugs, debug_match, match_timeline, replay_trace,
  repro_from_replay, restart_picture, plan_trace, plan_snapshot, pitch_maps,
  moments, figure_style
- tools/field/: field_check, field_physics, vision_check, package_logs,
  trim_idle_log, robot_calibration, field_sessions.json, ssl-vision-lab-field.patch
- tools/realism/: realism_scorecard, convert_ssl_log
- tools/dev/: setup_worktree.sh, package_graph, check_strategy_branch, build_strategy
- tools/motion_planning_benchmark.py stays at the top level
- tools/sim_setup.py is a library: now utama_core/run/sim_setup.py

Scripts resolve the repo root one level deeper (parents[2]); field_check's
sibling import now points at tools/field. moments.py's worktree snapshot copies
itself to tools/debug/ and creates the directory first. CI and CODEOWNERS point
at tools/dev/check_strategy_branch.py. tools/ and each group get an empty __init__.py.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
examples: move the three field-session scripts to tools/field
4238f90 
- examples/calibrate_robot.py, shooting_drill.py, ball_trials.py -> tools/field/

These are the field-day tools (pixi run calibrate, the field test rig), not
dashboard demos. Imports of them are now tools.field.<name>; the other examples,
demos.py and the tests under utama_core/tests/examples use those paths.
ball_trials.py still takes take_ball_facing from examples/pass_receive.py,
imported as examples.pass_receive. The moved scripts resolve the repo root
one level deeper (parents[2]). Text references and pixi's calibrate task
point at the new paths.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: run as utama_core.dashboard; pixi tasks for evaluate and d…
8a3e0de 
…ashboard

- dashboard_server.py -> utama_core/dashboard/__main__.py (python -m utama_core.dashboard).
  The script imported only utama_core and needed no path logic.
- pixi.toml: `evaluate` (python tools/evaluation/evaluate.py, so `pixi run evaluate NAME`)
  and `dashboard` (python -m utama_core.dashboard), next to `main`.
- README, docs/tools.md, docs/first_strategy.md, docs/strategies.md,
  docs/real_robot_setup.md, change-strategy's SKILL.md, AGENTS.md and the
  dashboard's live.js comment use the new commands.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
tests: the field scripts moved out of examples/ keep their import check
70c2138 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
structure: custom_referee/ becomes referee/
663ced6 
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
structure: rsoccer_simulator/ becomes simulator/, subpackages lowercase
0aee61e 
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: stale script references; demo_pendulum_match
e6e517a 
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: the Tournament tab becomes the Strategies tab
5e3022b 
Every catalog strategy with its description, file, role, tactics, verdicts, per-match signals
beside the other strategies' average, and its matches with replay links; every evaluation run with
its verdicts first, then the health check and the matches per round. Standings are gone: the
verdict is the only ranking. Frozen .strategy files and pre-rename names show as the catalog name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: runs send each match's stall flag, not its full stats (12 …
026d9b4 
…MB to 0.7 MB)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
fix: paths one folder off after dropping src/; colours by seed; stale…
43c70a8 
… tests

- robot_profile, find_robots, robosim_wrapper counted folders up from where they sat
  before the src/ layer went: profiles loaded blank (no kick bend or carry caps on real
  robots), the robot pairing and the robosim build env pointed outside the repo
- evaluate.py: colours follow the seed, not the round, so a confirm seed has the same
  colours whatever ROUNDS was; identical at ROUNDS=6, so no stored match changes
- tests: explain's helper deleted with the research tools; fingerprint audit re-sorted
  after the renames

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: regenerate the package graph and data-flow picture after the re…
3b31867 
…names

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: a strategy page with less text
5161e40 
The summary without the name and boilerplate, tactics as chips, verdict and code on
one line each (details on hover), signals from a real round not an exhibition match,
matches of the newest run only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: runs newest first by date; the frozen reference as 'name (…
26681d0 
…last release)'

Runs were sorted by id, so every tournament_ run came before any eval_ run and a
strategy's page showed old signals and matches. evaluate.py records the reference as
name@commit: now mapped to its catalog name, marked in match lists and kept out of the
signals' comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: light theme, position and shot maps per strategy, newest m…
f39f2d0 
…atches only

- Every tab light, sans-serif; the field and timeline colours redrawn for a light page.
- A strategy's page: with-ball and without-ball position maps and its shots, drawn from
  `pitch_maps.py --data` (run in the background once per round, kept as pitch_maps.json);
  "how it plays" as this-against-others dots; the newest run's matches only.
- Runs: evaluate.py runs first; older round robins in one collapsed group.
- Fix: the frozen release file's matches were listed as today's strategy's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
dashboard: no test strategy in the list; the release file's matches l…
62bb494 
…abelled and playable

- `catalog.player_names()`: every strategy but the tests' minimal `default`; the match tool, the
  strategy tests and the dashboard use it instead of each filtering the name.
- Matches against the last release's file read "pendulum (last release)" in runs too, and find
  their replay under the name evaluate.py saves it as (split_shape@<commit>_vs_...).
- Today's strategy against its own last release is listed on its page.
- Replay: no robot coordinate list; a tactic tag only when it differs from the label; robot ids
  readable on the light pitch. Match cards never wrap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: the tutorial looks around the dashboard first, with the replays…
c1799af 
… zip of the release

- first_strategy.md step 2: download replays-v2.6.0.zip (core round 1 of the health check),
  open the Strategies tab, read the maps and numbers, watch a replay; two screenshots.
- Step 9 points back to the dashboard for the new strategy's verdict and maps.
- A strategy with no matches says how it would get some.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
check_strategy_branch: docs/strategies.md is no longer on a strategy …
7b02b5d 
…branch's allowlist

The catalog doc is deleted (next commit): the dashboard reads every strategy, its description and
its results from the code and replays/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: 27 files down to 10; merge what is still used, delete history, …
e7ff86a 
…plans and the catalog

Kept: first_strategy, STRATEGY_DEVELOPMENT, tactic_model_design_decisions, signals, referee,
motion_planning_comparison, real_robot_setup, architecture, tools, README (a 9-line index).

Merged: pitch_zones into signals; referee_integration, referee_gui and referee_design_decisions
into referee; field_logs and setup_external into real_robot_setup; pipeline_method into
architecture; contributing into the top-level README. References updated to the new sections.

Deleted (in git history): strategies and visual_guide (the dashboard shows every strategy, its
maps and matches), sim_realism, roadmap, in_house_vision, testing_gaps, field_log_replay,
scheduling_math_model, motion_planning_results, and the pictures only they used. The rsim tick
diagram moves to vendor/rSim/FORK_NOTES.md. Code comments citing a deleted doc keep pointing at
history, as docs/README.md says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: tutorials/ for step-by-step pages, docs/ for reference; consist…
6d24055 
…ent names; stale facts fixed

- tutorials/first_strategy.md (moved, with its screenshots) and tutorials/field_day.md (the
  setup and test-day steps of real_robot_setup.md). The rest of that file is docs/real_robots.md.
- Renamed: STRATEGY_DEVELOPMENT.md -> strategy.md, tactic_model_design_decisions.md ->
  strategy_design.md, motion_planning_comparison.md -> motion_planning.md. Section numbers kept.
- docs/README.md points at tutorials/ first; AGENTS.md keeps the split.
- Stale: the referee's "ball placement before a free kick" gap is fixed in code (moved to
  resolved; the open lists are one Known gaps list); convert_ssl_log keeps raw logs with
  --keep-raw; `pixi run runs` lists evaluation rounds too; strategy.md says where to see a
  result (the dashboard).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
docs: what was learned, where everyone can read it
e5fbd8b 
Notes that only the agent's private memory held, now in the docs:
- strategy.md: "Tried and rejected" (eight paired-drill experiments and one shipped), "Known open
  bugs" (the keeper ignores a slow ball rolling through its box, left open as a first task), what
  the 118 real matches showed, and why evaluation needs many matches.
- real_robots.md: "Our robots": one kick power, kick bend, weak dribbler and carry caps, motion
  delay and acceleration, lid vs radio, vision over Wi-Fi.
- referee.md: the lone-defender gap and the shadow run over real matches.
- AGENTS.md: evaluation at 2 m/s and one kick power, on purpose; rsim differs across CPUs.
- tools.md: worktrees share replays/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 @claude
skills: in .agents/skills for any agent; .claude/skills is a symlink
ebbf523 
The playbooks were already plain Markdown; only their place tied them to Claude Code. AGENTS.md,
which every agent reads, now lists them with when to use each, and the tutorial's last step names
no particular agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@isaac0804 isaac0804 added the release:minor Minor changes to main label Nov 13, 2026
@isaac0804
isaac0804 merged commit 3698edc into main Nov 13, 2026
4 of 5 checks passed
@isaac0804
isaac0804 deleted the infra/evaluate-dashboard-onboarding branch November 13, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

release:minor Minor changes to main

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@isaac0804