Skip to content

Merge main and update for 1.0.0 (into #1631) - #1663

Closed
ekjotmultani wants to merge 5 commits into
fix/dart-codegen-format-outputfrom
fix/dart-codegen-format-output-update
Closed

ekjotmultani wants to merge 5 commits into
fix/dart-codegen-format-outputfrom
fix/dart-codegen-format-output-update

Conversation

@ekjotmultani

Copy link
Copy Markdown
Contributor

This merges main into #1631 so it can land in Dart 1.0.0. #1574, #1578, #1599 and #1481 merged after this branch was cut, and all of them regenerated the Dart goldens.

  • 58feeaf: merge origin/main. Every Dart golden conflicted, so I took main's side and regenerated all of them with this branch's formatter (REGENERATE_FIXTURES=1 dart test test/golden_file_test.dart). Fixtures 44 and 45 are new on main and are now formatted too. Fixture 35 stays a codegen-error.txt golden. The only source conflict was the CHANGELOG: the formatting bullet sits at the top of ### Other changes, and nothing from ## 0.2.0 down changed.
  • b244937: scripts/sink-guards/codegen.test.mjs pinned the toString writeln in generator.dart at 2168, which matched no site after the merge. The same line is now at 2175.
  • 8b1ddea: open_union_test.dart and open_enum_test.dart (from feat(native-codegen): decode enum values and union arms a client doesn't know #1481) matched one-line snippets that the formatter splits. They now use dart_source_matchers.dart's contains, like the other tests here.
  • 4027e1d: dependency review failed because the dependency graph reports dart_style and pub_semver as lowercase bsd-3-clause, which the action rejects as an invalid SPDX id. This adds both to allow_dependencies_licenses, as fix(dart): drop the analyzer cap, accept build 4 and 5, and allow build's lowercase license #1474 did for build. The "could not detect a license" line for package_config is informational; http gets the same line on main and passes.

Swift and Kotlin goldens are unchanged. This merges cleanly with #1575 in either order.

You can merge this into your branch as is, or pull your local changes first and then merge it. Drop or reword anything you like.

cadivus and others added 5 commits November 19, 2026 00:26
The generator returned raw StringBuffer output from every sink, so
generated clients failed `dart format`.

generate() now formats the complete unit with dart_style using the target
package language version from package_config for build_runner and CLI,
with Dart 3.11 fallback. It prepends `// dart format width=80` so output
is stable regardless of consumer page_width. Dart goldens were regenerated.
Take main's Dart goldens and regenerate them with this branch's formatter.
@github-actions

Copy link
Copy Markdown
Contributor
📖 Glossary & notes — scoring, indicators, per-metric thresholds (click to expand)
  • N = 1 — one rep per cell, so a small delta may be model variance, not a real change; re-run for certainty.
  • Indicators (change vs baseline, per metric): ✨ noticeably improved · ✅ OK / no significant change (within noise band) · ⚠️ slight regression (beyond 1× threshold) · ❌ serious regression (beyond 2× threshold) · 🆕 new — no baseline value yet (current value shown, tagged (new)) · — nothing to show this run · 🗑️ cell gone since the baseline. Each cell shows <indicator> <value> (<Δ vs main>) inline.
  • Columns: Tests (pass/denom) · Judge (overall 0-10 score with signed delta; per-dimension breakdown lives in the judge artifact JSON) · Cost · Turns (agent cycles) · Score · Stop reason.
  • Threshold definitions (per metric, DELTA_THRESHOLDS in overview.mjs):
    • composite/score: ±5 points — a change ≤5 pts is ✅; >5 improving is ✨; >5 worsening is ⚠️; >10 worsening is ❌.
    • judge (overall): ±0.3 — a shift ≤0.3 is ✅; >0.3 improving is ✨; >0.3 worsening is ⚠️; >0.6 worsening is ❌.
    • tests: ±1 pass — a ≤1 pass change is ✅; >1 improving is ✨; >1 worsening is ⚠️; >2 worsening is ❌.
    • cost: ±10% of baseline (floor $0.02) — within 10% is ✅; >10% cheaper is ✨; >10% more expensive is ⚠️; >20% more expensive is ❌.
    • turns: ±3 cycles — a ≤3 turn change is ✅; >3 fewer is ✨; >3 more is ⚠️; >6 more is ❌.
  • Directions: higher is better for tests, judge, and score; lower is better for cost and turns.
  • Composite (0-100) = round(60·test_rate + 4·judge·min(1, 4·test_rate), 1) — 60% objective pass-rate + 40% judge, the judge term gated below a 25% pass-rate.
  • Cost = builder token spend at Bedrock Claude Opus 4.8 rates ($5 in / $25 out per 1M tokens; BUILDER_PRICING in scoring.mjs).
  • SCORE = composite ÷ cost — composite points per dollar (higher = better). Cost, not raw token volume, is the denominator, so tokens can't dominate; a broken (composite 0) cell scores 0 no matter how cheap. Flip SCORE_PER_DOLLAR in scoring.mjs for cost-per-point (lower = better).
  • Baseline = the most recent main-branch bench (bench/runs/latest-main.json), NOT the PR base commit — the PR always diffs against the current state of main.
  • Excluded from the mean: harness_error cells (infra failures) and gradeable cells that ran no tests. agent_fail (agent produced no app in budget) and dead_server (built app never served / crashed) are BOTH included, as composite 0.
  • Run artifacts — per-cell source + full agent traces · 🕒 Last run: 2026-11-19T09:53:04Z · run #44495355663

Results — PR vs main baseline

Each metric shows its current value colored by the change vs the latest main baseline 4d81f11. ✨ improved · ✅ within noise · ⚠️ slight regression · ❌ serious regression · 🆕 no baseline yet (value shown, tagged (new)).

  • Mean composite 95.5/100 across 10 scored cell(s) — ✅ -0.9 vs main · judge mean 9.12/10.
  • Verdicts: 9 pass · 1 partial · 0 fail · 0 harness_error.
  • Totals: cost $156.71 · tokens 30.4M in / 186K out · 401 turns.
  • Biggest gains: auth-notes/demo (+4.8), observability-api/backend (+2.4), cognito-profile/auth (+1.6).
  • Biggest drops: file-gallery/bare (-16.4), sql-kb-catalog/nextjs (-2.4), collab-presence-board/default (-0.8).
  • Config: baseline 4d81f11.
  • ✅ Composite band 🟢 across 10 scored cell(s). Threshold 65 — pass.
Task Template Tests Judge Cost Turns Score Stop reason
async-word-counter bare ✅ 13/13 (0) ✅ 9.6 (0) ❌ $8 (+$4.54) ❌ 36 (+18) ❌ 12.3 (-16.1) endTurn
auth-notes demo ✅ 11/11 (0) ✨ 9.6 (+1.2) ❌ $34.76 (+$8.24) ❌ 69 (+9) ✅ 2.8 (-0.7) endTurn
cognito-profile auth ✅ 10/10 (0) ✨ 9 (+0.4) ✨ $19.78 (-$6.77) ✨ 57 (-10) ✅ 4.9 (+1.3) endTurn
collab-presence-board default ✅ 11/11 (0) ✅ 9.4 (-0.2) ⚠️ $4.01 (+$0.65) ⚠️ 20 (+5) ✅ 24.3 (-4.9) endTurn
email-digest demo ✅ 12/12 (0) ✅ 9.2 (+0.2) ✨ $4.17 (-$0.93) ✨ 19 (-8) ✅ 23.2 (+4.4) endTurn
file-gallery bare ⚠️ 10/12 (-2) ❌ 7.8 (-1.6) ✨ $2.9 (-$0.95) ✅ 14 (-3) ✅ 28 (+2.6) endTurn
kb-chat-agent demo ✅ 9/9 (0) ✅ 8.8 (0) ✨ $23.06 (-$7.75) ✨ 47 (-10) ✅ 4.1 (+1) endTurn
observability-api backend ✅ 18/18 (0) ✨ 9.6 (+0.6) ❌ $25.73 (+$17.59) ❌ 54 (+8) ⚠️ 3.8 (-8) endTurn
oidc-dsql-notes react ✅ 11/11 (0) ✅ 9.4 (+0.2) ✨ $27.1 (-$18.17) ✨ 59 (-33) ✅ 3.6 (+1.5) endTurn
sql-kb-catalog nextjs ✅ 9/9 (0) ⚠️ 8.8 (-0.6) ✨ $7.21 (-$3.37) ✨ 26 (-8) ✅ 13.2 (+4) endTurn

Merge verdict

✅ This bench run does not block the PR.

  • ✅ Absolute floor — mean composite 95.5 >= floor 65.
  • ✅ Regression vs main — mean composite delta -0.9 pts vs main is within the 10-pt limit.

The Δ vs base ball above is a fixed ±5-pt display band; the regression gate uses BENCH_MAX_REGRESSION. The ball is cosmetic — the gate threshold is what blocks.

@cadivus
cadivus force-pushed the fix/dart-codegen-format-output branch from db9a94e to b397b92 Compare November 19, 2026 13:03

This branch was previously deployed

1 inactive deployment
agent-bench — 4027e1de Deployed Nov 19, 2026 by ekjotmultani via bench / summary #6360
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants