diff --git a/edgebench/tasks/warehouse_forklift_routing/NOTES.md b/edgebench/tasks/warehouse_forklift_routing/NOTES.md index 5d8acd13..9f56feec 100644 --- a/edgebench/tasks/warehouse_forklift_routing/NOTES.md +++ b/edgebench/tasks/warehouse_forklift_routing/NOTES.md @@ -1,26 +1,38 @@ # warehouse_forklift_routing CodeChef WAREHOUS, in ALE-Bench style. The agent writes `solution.py` (stdin -> stdout). The -judge runs it on 50 fixed inputs (each with T=5 grids, 6 ≤ R, C ≤ 20), 8 at a time with a 60 s +judge runs it on 50 fixed inputs (each with T=5 grids, 6 ≤ R, C ≤ 20), 8 at a time with a 150 s wall-clock limit each, scores each output with `tools/bin/tester` (the average over the 5 grids of `(S+2)/(R+C−1) − 2RC + 20`, S = command length, printed with one decimal), and sums the 50 per-input scores. If any input is not `OK` the total is `inf`. Raw score = that sum (minimize). The rescale is the v1 one (`piecewise_log_min`). +## Changes + +### 1.1.0 + +`TIME_LIMIT` goes from 60 s to 150 s per input, the instruction's 30 s per test case for T=5. A +solver taking 60 to 150 s on any input scored `inf`; it now scores. The formula and pool are +unchanged; tested valid submissions moved at most 0.15 raw between judge means. Nominal grading time +is at most 7 rounds of (150 s + 30 s tester) = 1260 s, excluding overhead, inside `timeout_sec`. It +can outlast the 900 s final wait in `sforge run`: there, a 140 s deadline grade was still running at +that wait and was lost, though an earlier valid score survives. + ## Source - Workspace (READMEs, `tools/bin/{gen,tester}`, `tools/src/`) and the 50 judge inputs: the task authors' repository (not public), identical by sha256 and mode to the published v1 images. The tools are Python scripts. - `environment/workspace/solution.py` (starter) and `judge/eval_warehouse_manager.py`: - byte-identical to v1's inline scripts and to the published v1 images. `judge/grade.py` is new and + byte-identical to v1's inline scripts and to the published v1 images, except `TIME_LIMIT`, changed + in 1.1.0. `judge/grade.py` is new and writes `result.json` with v1's scoring. - `instruction.md`: v1's prompt, verbatim. Base: `bases/python310`. ## Deviations from v1 1. Root only: no `agent` user and no `sudo` (SCHEMA rule 3). -2. The eval script moved from `/tmp/` to `/judge/` (content unchanged). +2. The eval script moved from `/tmp/` to `/judge/` (content unchanged until 1.1.0, see Changes). 3. **`inf` becomes invalid.** When any input fails (including the empty submission, whose starter prints empty lines), v1 reported score `inf`, which was not rankable; the new grader returns `valid=false`. Same ranking effect. @@ -51,18 +63,17 @@ Kept as in v1, so scores stay comparable. Fixing any of them changes scores and 1. **The scored inputs are reproducible.** They come from the public generator with a fixed seed range; the instruction says to use seeds 0..10000 only, but nothing enforces it. -2. **Time limit mismatch.** The instruction and README say 30 s per test case; the judge allows - 60 s per input (5 test cases), and the tester gets its own 30 s. -3. **Host-dependent score.** Inputs run 8 in parallel with no CPU pinning, so time-budgeted +2. **Host-dependent score.** Inputs run 8 in parallel with no CPU pinning, so time-budgeted searches score differently with host speed and load, and a slow host can turn a near-limit - solution into a timeout. -4. **One failed input makes the whole submission unrankable**, including a timeout caused by host + solution into a timeout. At the official 4 CPUs each input gets about half a CPU, so a solver + doing fixed work above a computed 75 s of CPU per input still times out. +3. **One failed input makes the whole submission unrankable**, including a timeout caused by host load. -5. **No judge resource limits.** The instruction's 1 GB memory limit is not enforced. -6. Per-input scores are rounded to one decimal before summing. -7. The score parser does not accept negative numbers; no real submission comes close, so this is +4. **No judge resource limits.** The instruction's 1 GB memory limit is not enforced. +5. Per-input scores are rounded to one decimal before summing. +6. The score parser does not accept negative numbers; no real submission comes close, so this is theoretical. -8. **The submission can tamper with grading.** Solutions run as root next to the grader and the +7. **The submission can tamper with grading.** Solutions run as root next to the grader and the tester (the cross-task issue in `tasks/SCHEMA.md`). -9. **The starter does not pass.** The instruction calls `solution.py` a baseline to improve, but it +8. **The starter does not pass.** The instruction calls `solution.py` a baseline to improve, but it prints empty lines that the tester rejects. diff --git a/edgebench/tasks/warehouse_forklift_routing/judge/Dockerfile b/edgebench/tasks/warehouse_forklift_routing/judge/Dockerfile index c5b2baad..d025a4dd 100644 --- a/edgebench/tasks/warehouse_forklift_routing/judge/Dockerfile +++ b/edgebench/tasks/warehouse_forklift_routing/judge/Dockerfile @@ -15,7 +15,7 @@ # 9 rm -rf .git && git init && git config user.* && add && commit -m init # -> kept, author/dates pinned to v1's commit # 10 heredoc > /tmp/eval_warehouse_manager.py; chmod +x -> COPY judge/eval_warehouse_manager.py -# /judge/ (byte-identical, moved out of /tmp) +# /judge/ (moved out of /tmp; TIME_LIMIT changed in 1.1.0) # v1 eval_cmd + parser score_sum -> judge/grade.py (task.toml [judge].cmd) # (v1 judge image has no tools/src/ and no tools/README.md; neither is needed: tester is self-contained.) diff --git a/edgebench/tasks/warehouse_forklift_routing/judge/eval_warehouse_manager.py b/edgebench/tasks/warehouse_forklift_routing/judge/eval_warehouse_manager.py index 560ee779..f5166424 100644 --- a/edgebench/tasks/warehouse_forklift_routing/judge/eval_warehouse_manager.py +++ b/edgebench/tasks/warehouse_forklift_routing/judge/eval_warehouse_manager.py @@ -6,7 +6,7 @@ WORK_DIR = "/home/workspace/warehouse_manager" CASES_DIR = os.path.join(WORK_DIR, "cases") TESTER = os.path.join(WORK_DIR, "tools", "bin", "tester") -TIME_LIMIT = 60 +TIME_LIMIT = 150 def run_case(args): i, case_file = args diff --git a/edgebench/tasks/warehouse_forklift_routing/judge/grade.py b/edgebench/tasks/warehouse_forklift_routing/judge/grade.py index 829bf0f9..607ec60a 100644 --- a/edgebench/tasks/warehouse_forklift_routing/judge/grade.py +++ b/edgebench/tasks/warehouse_forklift_routing/judge/grade.py @@ -5,7 +5,7 @@ v1 eval_cmd : cd /home/workspace/warehouse_manager && python3 /tmp/eval_warehouse_manager.py v1 parser : score_sum (sforge/harness/log_parsers/score_sum.py) v1 score : first `TOTAL_SCORE ` in the eval output (sforge/harness/grading.py:extract_score) -/judge/eval_warehouse_manager.py is byte-identical to the v1 /tmp/eval_warehouse_manager.py. +/judge/eval_warehouse_manager.py is the v1 /tmp/eval_warehouse_manager.py, except TIME_LIMIT (NOTES.md). This wrapper only runs it and translates its stdout into /opt/sforge/result.json. stdout = logs only. """ diff --git a/edgebench/tasks/warehouse_forklift_routing/task.toml b/edgebench/tasks/warehouse_forklift_routing/task.toml index 7a584b56..0c9fb4b0 100644 --- a/edgebench/tasks/warehouse_forklift_routing/task.toml +++ b/edgebench/tasks/warehouse_forklift_routing/task.toml @@ -4,7 +4,7 @@ name = "Warehouse Forklift Routing" category = "Combinatorial Optimization" language = "Python" description = "Route a forklift in a grid warehouse to receive goods arriving in random order, store them, and dispatch them in sequential order, minimizing total movement." -version = "1.0.0" +version = "1.1.0" status = "migrated" [environment] @@ -26,7 +26,7 @@ timeout_sec = 3600 cpus = 4 memory = "8g" # Official leaderboard limits (v1 kept them only in the experiment YAMLs). -# eval_warehouse_manager.py runs the 50 cases 8 at a time, each `python3 solution.py` under a 60 s +# eval_warehouse_manager.py runs the 50 cases 8 at a time, each `python3 solution.py` under a 150 s # wall-clock timeout. Scores of time-limited (anytime) solutions depend on host CPU speed and # load; see NOTES.md.