Skip to content

feat(settings): persist a default per-conversation cost budget in ConversationSettings - #6824

Open
neubig wants to merge 4 commits into
mainfrom
feat/conversation-settings-budget
Open

neubig wants to merge 4 commits into
mainfrom
feat/conversation-settings-budget

Conversation

@neubig

@neubig neubig commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

HUMAN:


AGENT:

Why

The agent-server already enforces a per-conversation spend cap: StartConversationRequest.max_budget_per_run (#6257) sets it at creation, and PATCH /api/conversations/{id} changes or removes it on an existing conversation. What is missing is a persisted default for new conversations. ConversationSettings, the settings model that clients render and that create_request() turns into a start request, has max_iterations and no budget. A self-hosted Agent Canvas user therefore cannot set a default cost limit: every conversation started from the UI is unbounded (only the 500-iteration cap applies). This is the SDK half of the cost-control request in #5561 / OpenHands/OpenHands#17691.

Summary

  • ConversationSettings.max_budget_per_run: float | None (USD, must be > 0, default None = no limit), labelled Default budget per conversation (USD). Its description says that the value applies to new conversations and that changing it does not affect existing ones (those use PATCH /api/conversations/{id}). It sits in the general schema section next to max_iterations, so schema-driven clients render it automatically.
  • create_request() forwards it to StartConversationRequest.max_budget_per_run as a default; an explicit kwarg still overrides it. The OpenAI-compatible gateway (openai/service.py), which starts conversations through create_request(), gets the budget the same way.
  • Tests: schema section contents and label, round-trip, create_request forwarding and override (an explicit kwarg beats a different settings value, and None clears it), rejection of 0 and negative budgets; the settings-router expected body includes the new key.

Issue Number

Part of #5561 (SDK side; the Canvas side is OpenHands/OpenHands#17691).

How to Test

Unit tests:

uv run pytest tests/sdk/test_settings.py tests/agent_server/test_settings_router.py -q

End to end with Agent Canvas (npm run dev:minimal from the companion Canvas PR, OH_AGENT_SERVER_LOCAL_PATH pointing at this branch). The LLM is a local fake OpenAI-compatible server that returns token usage, so litellm prices each call at $0.03. The prompt makes the agent run sleep 1 25 times.

  1. GET /api/settings/conversation-schema returns general: ['max_iterations', 'max_budget_per_run'] (main: only max_iterations).
  2. Settings → Conversation limits → Default budget per conversation (USD) = 0.2 → Save. GET /api/settings → conversation_settings.max_budget_per_run == 0.2.
  3. Start the conversation from the Home page and poll GET /api/conversations/<id> once a second:
  0.3s  status=running  cost=$0.0900  max_budget_per_run=0.2
  3.7s  status=running  cost=$0.1200  max_budget_per_run=0.2
  7.9s  status=running  cost=$0.1500  max_budget_per_run=0.2
 13.0s  status=running  cost=$0.1800  max_budget_per_run=0.2
 14.2s  status=error    cost=$0.2100  max_budget_per_run=0.2

The run stops with Agent reached maximum budget limit ($0.2000); accumulated cost $0.2100 after 6 commands. On main the same prompt runs all 25 commands and costs $0.81, with max_budget_per_run=null.

Video/Screenshots

Before (SDK main + Canvas main) After (this PR + companion Canvas PR)
before after

Design Doc

N/A (one settings field).

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

This PR was created by an AI agent (OpenHands) on behalf of Graham Neubig.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:7b5af73-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-7b5af73-python \
  ghcr.io/openhands/agent-server:7b5af73-python

All tags pushed for this build

ghcr.io/openhands/agent-server:7b5af73-golang-amd64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-golang-amd64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-golang-amd64
ghcr.io/openhands/agent-server:7b5af73-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:7b5af73-golang-arm64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-golang-arm64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-golang-arm64
ghcr.io/openhands/agent-server:7b5af73-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:7b5af73-java-amd64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-java-amd64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-java-amd64
ghcr.io/openhands/agent-server:7b5af73-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:7b5af73-java-arm64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-java-arm64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-java-arm64
ghcr.io/openhands/agent-server:7b5af73-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:7b5af73-python-amd64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-amd64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-amd64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-amd64
ghcr.io/openhands/agent-server:7b5af73-python-arm64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-arm64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-arm64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-arm64
ghcr.io/openhands/agent-server:7b5af73-python-minimal-amd64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-minimal-amd64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-minimal-amd64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:7b5af73-python-minimal-arm64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-minimal-arm64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-minimal-arm64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:7b5af73-python-slim-amd64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-slim-amd64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-slim-amd64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:7b5af73-python-slim-arm64
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-slim-arm64
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-slim-arm64
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:7b5af73-golang
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-golang
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-golang
ghcr.io/openhands/agent-server:7b5af73-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:7b5af73-java
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-java
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-java
ghcr.io/openhands/agent-server:7b5af73-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:7b5af73-python-minimal
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-minimal
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-minimal
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-minimal
ghcr.io/openhands/agent-server:7b5af73-python-slim
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python-slim
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python-slim
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime-slim
ghcr.io/openhands/agent-server:7b5af73-python
ghcr.io/openhands/agent-server:7b5af7381afd21bc42e72e07cfae1b1e12649900-python
ghcr.io/openhands/agent-server:feat-conversation-settings-budget-python
ghcr.io/openhands/agent-server:7b5af73-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 7b5af73-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 7b5af73-python-amd64) are also available if needed

…ings

ConversationSettings gains max_budget_per_run (USD, > 0, default None) in
the general section next to max_iterations, and create_request() forwards
it to StartConversationRequest.max_budget_per_run. Clients that render the
conversation-settings schema (Agent Canvas) can now offer the budget, so
conversations started from a UI are no longer unbounded by default.

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/settings
   model.py9376793%351, 369, 611, 623, 633–636, 639, 656, 662, 672, 678, 683, 839, 871, 874–875, 881–884, 889, 893, 908, 911–913, 961, 966–967, 972, 993, 1005, 1054, 1063, 1110, 1440, 1455, 1470, 1472, 1474, 1476, 1478, 1480, 1482, 1484, 1486, 1488, 1747, 1808, 1810, 2126, 2146, 2279, 2416, 2455, 2693, 2703, 2705, 2710, 2728, 2741, 2743, 2745, 2747, 2754
TOTAL56222876484% 

@neubig
neubig marked this pull request as ready for review November 14, 2026 02:02
@neubig
neubig requested review from all-hands-bot and removed request for all-hands-bot November 14, 2026 02:02

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope gate

This change belongs here. The repository owns the Python SDK and Agent Server settings surface, and linked issue #5561 (ready-for-dev, priority:medium) explicitly assigns the SDK/agent-server side of per-conversation cost budgeting to this repo; the Canvas UI side is tracked separately in OpenHands/OpenHands#17691. No maintainer decision needed.

Technical assessment

The diff is small and additive: a persisted max_budget_per_run: float | None (gt=0, default None) on ConversationSettings, forwarded through _start_request_kwargs() via setdefault so explicit kwargs still override, plus tests.

Verified against the current head:

  • Persistence/compatibility: ConversationSettings remains at CONVERSATION_SETTINGS_SCHEMA_VERSION = 1. The change is strictly additive with a nullable default, so a persisted v1 payload without the key validates to None (no limit) — exactly the documented behavior. The Persisted settings CI gate and the repo's check_persisted_settings_compat.py both pass (33 fixtures, 8 PyPI baselines re-validated), including the v1 conversation_settings golden fixture. No migration is required, and none is claimed.
  • Cross-layer trace: the field flows from settings → create_request() → StartConversationRequest.max_budget_per_run → the agent-server enforcement added in #6257 (request field, exclusion from StoredConversation, budget check in the conversation loop, and update support in PATCH via model_fields_set). The openai/service.py gateway inherits it through create_request() as described. The settings router round-trips conversation_settings.model_dump(), so the new key survives GET/PUT /api/settings — confirmed by the updated router test. Schema-driven clients pick the field up automatically via export_schema() (it sits in the general section next to max_iterations, MAJOR prominence), and the generated TypeScript schema already carries max_budget_per_run on the conversation types from the earlier #6257 change.
  • Validation: gt=0 rejects 0 and negative budgets at model construction (covered by the parametrized test), so an unusable budget cannot be persisted or sent to the server.
  • Tests: the full focused suites pass in the workspace — uv run pytest tests/sdk/test_settings.py tests/agent_server/test_settings_router.py -q → 314 passed.

No material correctness, security, or compatibility defect in the code itself.

CI / acceptance

All checks on head aac2da64 pass except Validate PR description (fail): Add a short human-written note between HUMAN: and AGENT: — the HUMAN: section currently contains only HTML comments, and the validator requires ≥20 visible characters of human-written text. Per repository policy, AI agents must not edit the HUMAN: section; only the human author can fix it. This is a hard CI gate, so the PR cannot merge until the author adds the note. The failed check run is 43544053329; all other suites (including Persisted settings, REST API (OpenAPI), Check OpenAPI Schema, and the test matrix) are green.

The code is approvable as-is; the outstanding item is the human-authored note in the PR description.

🔄 CHANGES REQUESTED

Comment thread tests/sdk/test_settings.py Resolved
The value is copied into each new conversation's start request; it does
not edit an existing conversation's budget (PATCH /api/conversations/{id}
does that). Rename the label to "Default budget per conversation (USD)"
and say so in the description.

Co-authored-by: openhands <openhands@all-hands.dev>
…udget

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig neubig changed the title feat(settings): persist a per-conversation cost budget in ConversationSettings feat(settings): persist a default per-conversation cost budget in ConversationSettings Nov 15, 2026
@neubig
neubig requested a review from all-hands-bot November 15, 2026 05:09

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope gate

Still belongs in this repository. The PR adds the settings-model half of per-conversation cost budgeting; linked issue #5561 (labels re-verified now: ready-for-dev, priority:medium, open) owns the SDK/agent-server side, and the Canvas UI side is tracked separately in OpenHands/OpenHands#17691. The prior review on aac2da64 passed the same gate.

What changed since the last review (aac2da6 → 7b5af73)

The head moved via two PR commits plus a merge from main (692604b8; every merged commit was verified to already exist on origin/main):

  1. 3fd41713 relabels the field to Default budget per conversation (USD) and rewrites the description to state that the value applies to new conversations and that changing it does not affect existing ones (which use PATCH /api/conversations/{id}). This correctly documents the semantics against the server's existing create-time and PATCH-time budget handling (#6257), and the schema test now asserts the new label and the "new conversations" description text.
  2. 7b5af738 adds the exact case the previous review requested: ConversationSettings(max_budget_per_run=2.5).create_request(..., max_budget_per_run=5.0) now asserts 5.0, pinning kwarg precedence over a different settings value. The review thread was answered and resolved.

Verification on the current head

  • Focused suites pass in the workspace: uv run pytest tests/sdk/test_settings.py tests/agent_server/test_settings_router.py -q → 314 passed.
  • Persisted-settings compatibility gate passes: uv run python .github/scripts/check_persisted_settings_compat.py → 33 fixtures validated, 8 PyPI baselines (openhands-sdk==1.62.0) migrated and re-validated; ConversationSettings stays at schema version 1, so a persisted v1 payload without the new key loads as None (no limit) with no migration — as the PR claims.
  • The diff against current origin/main is unchanged in substance: the same additive max_budget_per_run field (gt=0, default None) in the general schema section plus the _start_request_kwargs() setdefault forwarding. The settings router round-trips conversation_settings.model_dump(), and the exported schema now carries the corrected label; the generated TypeScript client already mirrors max_budget_per_run from the earlier #6257 server work. No other supported entry point drops or ignores the field.
  • Newest main commits (through 2aae48b6) touch none of this PR's files; no hidden conflicts with the settings surface.
  • The REST API contract diff embedded in the PR body (MCPOAuthStartRequest) originates from merged main commit 2284096d (#6221), not from this PR's own change — consistent with the Check OpenAPI Schema gate's diff-against-base reporting.

No material correctness, security, or compatibility defect on the current head.

CI / acceptance

All checks on 7b5af738 were still pending at review time except Validate PR description, which has already failed with the same error as before: Add a short human-written note between HUMAN: and AGENT: — the HUMAN: section still contains only HTML comments. Per repository policy AI agents must not edit the HUMAN: section, so only the human author can clear this hard merge gate. Everything the repository's acceptance review requires of the code itself is satisfied and green.

The code is ready; the outstanding item remains the human-authored note in the PR description.

🔄 CHANGES REQUESTED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants