Skip to content

Add max_turns_budget experiment to go-logger and support expression-safe Claude max-turns compilation#36853

Draft
Copilot wants to merge 2 commits into
mainfrom
copilot/ab-advisor-experiment-max-turns
Draft

Add max_turns_budget experiment to go-logger and support expression-safe Claude max-turns compilation#36853
Copilot wants to merge 2 commits into
mainfrom
copilot/ab-advisor-experiment-max-turns

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Jun 4, 2026

This change introduces an A/B experiment for go-logger to compare conservative/standard/generous turn budgets and measure impact on file throughput, compliance, and cost. It wires variant-driven max-turn selection into the workflow while keeping compilation valid and lock output deterministic.

  • Workflow experiment wiring

    • Added experiments.max_turns_budget to .github/workflows/go-logger.md with:
      • variants: conservative, standard, generous
      • metric + secondary metrics + guardrails
      • weighted split, start date, analysis type, tags
    • Added variant-driven top-level max-turns expression mapping:
      • conservative -> 20
      • standard -> 35
      • generous -> 55
  • Compiler/runtime safety for Claude max-turn expressions

    • Updated pkg/workflow/claude_engine.go to avoid embedding raw GitHub expressions directly in shell run: commands when max-turns is expression-based.
    • For expression values, --max-turns is appended via shell expansion from GH_AW_MAX_TURNS, preserving expression support without template-injection regression.
  • Lock regeneration + coverage

    • Recompiled .github/workflows/go-logger.lock.yml to include experiment outputs/spec and mapped max-turn behavior.
    • Added integration coverage in pkg/workflow/max_turns_test.go for Claude top-level expression-based max-turns compilation.
# .github/workflows/go-logger.md
max-turns: "${{ needs.activation.outputs.max_turns_budget == 'conservative' && '20' || needs.activation.outputs.max_turns_budget == 'generous' && '55' || '35' }}"
experiments:
  max_turns_budget:
    variants: [conservative, standard, generous]
    metric: files_successfully_logged_per_run

…n handling

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add A/B test for max_turns in go-logger campaign Add max_turns_budget experiment to go-logger and support expression-safe Claude max-turns compilation Jun 4, 2026
Copilot AI requested a review from pelikhan June 4, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ab-advisor] Experiment campaign for go-logger: A/B test max_turns

2 participants