Repository navigation
fix(combos): resolve devin context windows for combo input admission #8575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
7551fad
6954439
c6f650d
95e27f6
7627e1a
ad56a8c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,7 +58,11 @@ export const DEVIN_STATIC_MODELS = [ | |
| * number over from the model's original vendor. | ||
| */ | ||
| export const DEVIN_MODEL_CONTEXT_WINDOWS: Record<string, number> = { | ||
| "swe-2": 262_000, | ||
| // 261112: live catalog context_window 850_000; a real turn completed at 754,971 total. | ||
| "swe-2": 850_000, | ||
| // 261112: live catalog also advertises 850_000, but observed turns terminate at | ||
| // totalTokens 262,140 -- the serving bound is ~256KiB, not the catalog figure. | ||
| "swe-2-flash": 262_000, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| "swe-1-7": 262_000, | ||
| "swe-1-7-lightning": 202_752, | ||
| "swe-1-6": 200_000, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -498,11 +498,19 @@ error naming a limit the model never approached — and by then output has commi | |
| target may be tried. When every failed target of an exhausted combo was refused this way with a send count of 0, `src/server/responses/core-combo.ts` keeps the `context_length_exceeded` overflow envelope that clients compact on, but replaces the provider wording from `context-overflow.ts` with a proxy-owned message saying the turn was refused locally, carrying each tried target's estimate and window (a target skipped for cooldown is not claimed to be too small). Those attempts, and the final row, keep the code `input_admission_refused` instead of `http_413`. One target that reached a provider keeps the provider wording for the whole combo. | ||
|
|
||
| Scope is deliberately narrow. Direct and single-target requests keep the loose 2.5x | ||
| pathological-input gate, because they have nowhere to hop. Compaction turns stay exempt. Unknown | ||
| context and a caller that declared no output allowance both remain fail-open, so this invents no | ||
| limits for custom providers. Canonical native slugs that the narrower pinned table does not carry | ||
| resolve their window from the generated in-tree bundle, which is what made the gate inert on the | ||
| route where this was first observed; explicit provider and operator caps may only narrow it. | ||
| pathological-input gate, because they have nowhere to hop. Compaction turns stay exempt. A | ||
| combo request that declared no output allowance cannot price headroom, so it falls back to | ||
| the same loose 2.5x pathological-input gate direct requests use rather than the strict | ||
| input-plus-headroom check; only a request whose ceiling never resolves stays fail-open, so | ||
| this invents no limits for custom providers. When the routed provider config declares no window at all, the devin | ||
| adapter falls back to its own static table (`DEVIN_MODEL_CONTEXT_WINDOWS` in | ||
| `src/adapters/devin/live-models.ts`, effort suffixes collapsed), because live catalog windows | ||
| reach the model list but never `provider.modelContextWindows` — without it every devin combo | ||
| target resolved `window: null` and the strict gate admitted turns that could only end in a | ||
|
Comment on lines
+508
to
+509
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| committed-output `max_output_tokens` stop. Canonical native slugs that the narrower pinned | ||
| table does not carry resolve their window from the generated in-tree bundle, which is what | ||
| made the gate inert on the route where this was first observed; explicit provider and | ||
| operator caps may only narrow it. | ||
|
|
||
| Regression coverage: `tests/server/input-admission.test.ts`, | ||
| `tests/helpers/combo-context-headroom-cases.ts` and `tests/responses/responses-context-overflow.test.ts`. | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update the existing
swe-2test assertion.tests/providers/devin-adapter.test.ts, Line 450, still expectsDEVIN_MODEL_CONTEXT_WINDOWS["swe-2"]to equal262_000. This change makes that assertion fail in the full test suite. Update the assertion to match the new value.Script executed:
Repository: lidge-jun/opencodex
Length of output: 15931
Script executed:
git cat-file -e f90d2951f6a7aefc1566afd7e9fe9744b8e4db6f^{commit} && git cat-file -e ee68b45a29da1aeb8a108981bfbe5eb325831fd6^{commit} && git diff f90d2951f6a7aefc1566afd7e9fe9744b8e4db6f ee68b45a29da1aeb8a108981bfbe5eb325831fd6 -- src/adapters/devin/live-models.ts src/providers printf '\n--- old table ---\n' git show f90d2951f6a7aefc1566afd7e9fe9744b8e4db6f:src/adapters/devin/live-models.ts | nl -ba | sed -n '35,80p' printf '\n--- registry bindings and persistence references ---\n' rg -n -F -e 'DEVIN_MODEL_CONTEXT_WINDOWS' -e 'providerConfigSeed(' -e 'projectStartupConfigRepairs(' -e 'resolveContextLimits(' -- src tests | sed -n '1,200p' printf '\n--- migration ---\n' cat -n src/providers/stale-context-window-migration.ts | sed -n '1,115p'Repository: lidge-jun/opencodex
Length of output: 34118
Script executed:
Repository: lidge-jun/opencodex
Length of output: 25769
Script executed:
Repository: lidge-jun/opencodex
Length of output: 35028
Script executed:
Repository: lidge-jun/opencodex
Length of output: 25986
Script executed:
Repository: lidge-jun/opencodex
Length of output: 6748
Script executed:
Repository: lidge-jun/opencodex
Length of output: 3525
Provide an upgrade path for the old
swe-2window.Older Devin configurations can retain the seeded
262_000value. Combo admission then skipsswe-2when estimated input plus output headroom exceeds that value, even if the turn fits the new850_000window.Add an opt-in repair and regression coverage. Preserve intentional operator settings, including settings that match the old seed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 6954439 — the devin-adapter assertion now expects the measured 850_000. On seeded values: provider.modelContextWindows is seeded from this same table, so the correction propagates; operator-set values still win by design.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Analysis chain
Script executed:
Length of output: 99940
Script executed:
Length of output: 30250
@omar391Thanks. I confirmed thattests/providers/devin-adapter.test.ts, Line 452, now expects850_000. I did not run the test.For existing saved configurations, the table update does not always propagate:
providerConfigSeedcopies the window map insrc/providers/derive.ts, Line 267. The provider CLI can then save that copy.mapFillgives a saved entry precedence over the registry insrc/providers/resolved-model-policy-merge.ts, Lines 37–42.swe-2insrc/providers/stale-context-window-migration.ts, Lines 45–56.If an older saved entry contains
262_000, combo admission can still skipswe-2when estimated input plus output headroom exceeds that limit. An intentional262_000override must remain unchanged.For operators who want the new limit, an explicit opt-in update of the
swe-2entry would preserve that choice. Regression tests should cover both the opt-in update and an unchanged intentional override.If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
You are interacting with an AI system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow-up addressed in c6f650d — added { devin, swe-2, 262_000 -> 850_000 } to STALE_CONTEXT_WINDOWS so saved configs seeded with the wrong figure get repaired on upgrade, while values that differ from the stale seed (intentional overrides) are untouched. Covered by the existing seed-repair suite.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Analysis chain
Script executed:
Length of output: 39520
Script executed:
Length of output: 42230
@omar391Thanks. The new entry updates savedswe-2values that match the old seed. One case remains:src/providers/stale-context-window-migration.ts, Line 78, checks the saved value but not its origin. If an operator intentionally set262_000, the repair also changes it to850_000.src/providers/model-rename-startup.ts, Line 19 and Line 101, applies and persists that repair at startup.tests/providers/context-window-seed-repair.test.tstests other models, but it does not test thisswe-2case.Please make the
swe-2repair opt-in, so an intentional262_000setting stays unchanged by default. Add tests for the opt-in upgrade, the unchanged override, and combo admission after the upgrade.I inspected the code but did not run tests. Please run the focused seed-repair and input-admission tests.
You are interacting with an AI system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added focused swe-2 coverage in the latest commit (stale 262_000 -> 850_000 repair + deliberate override untouched; 7/7 pass). On opt-in: the migration's contract is byte-for-byte match on the stale seed — identical for all ten existing entries — because a saved 262_000 is indistinguishable from a seeded one. Making this one row opt-in while the others stay automatic would be inconsistent; an operator who deliberately wants 262k can re-set it after the one-time startup repair.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Analysis chain
Script executed:
Length of output: 40069
@omar391Thanks. I understand the choice to keep this migration consistent with the existing value-based repairs.The new tests cover the
262_000→850_000repair and preserve a300_000override intests/providers/context-window-seed-repair.test.ts, Line 37 and Line 44. I did not run the tests.One correction to the workaround: resetting
swe-2to262_000after the repair will not survive the next startup.src/providers/model-rename-startup.ts, Line 19, runs the repair at startup, andsrc/providers/stale-context-window-migration.ts, Line 78, matches that value again. An operator can use261_999for an almost identical limit that does not match the repair.You are interacting with an AI system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch — noted. An operator wanting approximately that bound should pick a value that does not byte-match the stale seed (e.g. 261_999), since the repair re-matches 262_000 on every startup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@omar391 Thanks for confirming. I understand the decision to keep the value-based repair.
You are interacting with an AI system.