Repository navigation
feat: Add role-aware Scenario target-attempt accounting (#3043) - #3062
Nimit Jain (Nimit3418) wants to merge 10 commits into
Conversation
Interpret saved outcome counts through an async SDK with shared loop-bound admission, cooperative cancellation, and deterministic cleanup. Match bounded Unicode profile aggregation to the complete SQL fallback, with exact drilldowns, focused compatibility coverage, and SDK lifecycle documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve cancellation when admission races with a slot grant on Python 3.11. Release unscheduled operation capacity and close rejected coroutines, keep shutdown retryable, and log failures that race with caller cancellation. Add deterministic lifecycle regressions and verify raw-result analytics remains distinct from scenario-unit statistics and explicit result-role policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Expose decided-only and all-outcome success rates through one calculator and shared model. Keep raw saved-result selection distinct from scenario latest-unit selection while reusing count validation, rates, shares, and percentage formatting. Preserve legacy defaults and carry both rates through scenario progress and JSON projections. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reject contradictory scenario totals and outcome breakdowns, including mutated inputs at aggregation boundaries. Expose an explicit read-only success_rate_decided alias in shared statistics and JSON. Add typed include_outcome_statistics opt-ins to maintained result and async technique analytics while preserving the six-field default and existing deprecation schedules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@microsoft-github-policy-service agree |
|
Hi Roman Lutz (@romanlutz), thanks for the guidance and the heads-up on the overlapping changes! I've updated the PR to coordinate with the shared OutcomeStatistics calculation from #3059 and fully addressed the role-aware accounting constraints you outlined. Here’s a breakdown of how it was implemented in this update:
Let me know if there's anything else you'd like adjusted. Otherwise, it should be good for another look whenever you have time! |
Description
This PR implements the role-aware target-attempt accounting projection requested in #3043.
Key Changes:
memory_interface.pyto extractresult_roleand aggregate attempt metrics by role (TARGET_FACING,ORCHESTRATION, andUNKNOWN_ROLE) directly via SQL, ensuring SQLite and Azure SQL parity.UNKNOWN_ROLEto protect legacy runs and prevent silent redefinitions.ScenarioProducerCounts. Target-facing attempts now strictly represent physical requests, while orchestration envelopes complete logical units without inflating raw attempt metrics.ScenarioProducerCountsis now properly typed and mapped seamlessly intoScenarioRunListItemandScenarioRunSummary.Closes #3043
Testing
make unit-testlocally and verified that all 24,000+ unit tests pass 100% green.result_rolecorrectly fallback toUNKNOWN_ROLEwithout crashing.