Skip to content

Add SubQ backend with per-backend request limits; release 0.10.0 - #30

Merged
DevMortimer merged 2 commits into
mainfrom
carvel/subq-backend/subq-judge-backend
Oct 15, 2026
Merged

DevMortimer merged 2 commits into
mainfrom
carvel/subq-backend/subq-judge-backend

Conversation

@DevMortimer

@DevMortimer DevMortimer commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

What

  • A subq backend for SubQ's decision model. createTypeSafe({ backend: "subq" }) sends judgments to https://api.subq.ai/v1/systemone with Authorization: Bearer $SUBQ_API_KEY and the model subq-snap-preview. The usage ledger prices it at $0.05 per million input tokens. The model list at /v1/models is public, so listModels() does not verify the key there.
  • Per-backend request limits (BackendLimits: maxQuestions, minChoiceOptions, maxChoiceOptions, maxScoreLevels). A backend that declares none keeps the schema's bounds (DEFAULT_BACKEND_LIMITS: 32 questions, 1–64 options, 32 levels). No limit can be above the schema's bound. subq declares 16 questions, at least 2 Choice options, and at most 10 Score levels, per SubQ's Decisions API docs.
  • evaluate (and so evaluateMany and evaluateAll) refuses a request over the backend's limits before it sends it, with the same validation error kind the schema uses. The refused request does not count toward the request budget.
  • evaluateAll chunks to the client's question limit. chunkEvaluationRequest takes a backend option and chunks to that backend's limit. A maxQuestions above the limit is capped.
  • Callers read the limits without a request from client.limits or resolveBackend(spec).limits. A BackendEndpoint can declare limits, and they are validated like the other endpoint fields.
  • Release 0.10.0 (package.json, package-lock.json, CHANGELOG.md), as the maintainer asked.

Why

SubQ serves the same decisions protocol but accepts at most 16 questions in one request. Without limits, any caller that sends 17–32 questions would get a 422 from SubQ.

Behaviour of existing backends

typesafe, openrouter, commandcode, and liquid declare no limits. They keep the schema's bounds, so they accept the same requests as before. There is one change in chunking: chunkEvaluationRequest(request, { maxQuestions: 50 }) now returns chunks of 32, not 50. Chunks of 50 were refused by the schema anyway.

Verification

  • npm run check: build, typecheck, and 169 of 169 tests pass. That includes 12 new tests in tests/subq.test.ts and new limit tests in tests/backends.test.ts and tests/batch.test.ts.
  • A script imported the built dist/ with a dummy SUBQ_API_KEY and sent requests to the real api.subq.ai:
    limits.maxQuestions subq: 16 typesafe: 32
    evaluate: true http 401 TypeSafe returned HTTP 401. Check SUBQ_API_KEY. No automatic retry was made.
    listModels: ["subq-snap-preview"]
    17 on subq: validation Evaluation exceeds the backend's limits: 17 questions, at most 16 per request. evaluateAll splits a request to this limit. | fetch calls: 0
    evaluateAll 17 on subq: ok true chunks [16,1] answers 17
    
    The last two lines used an injected counting transport, so no request went out.
  • Not verified: a real decision with a valid SubQ key. npm run test:live was not run, because no key was available. The transport change is limited to one case: the model-list rewrite now also applies when a backend names modelsField without modelsPath, and only subq does that in the registry. The tests and the live 401 and model-list calls above cover it. Response validation is unchanged.

Add a subq backend for SubQ's decision model: judgments go to api.subq.ai under /v1/systemone with the key from SUBQ_API_KEY and the model subq-snap-preview, priced at $0.05 per million input tokens. Its model list is public, so listModels() there does not verify the key.

SubQ accepts at most 16 questions in one request, a Choice of at least 2 options, and a Score of at most 10 levels, where the schema allows 32, 1, and 32. A registry entry or a caller-supplied endpoint can now declare these limits; a backend that declares none keeps the schema's bounds, and no limit can be above them. evaluate refuses a request over its backend's limits with a validation error before sending it, evaluateAll and chunkEvaluationRequest chunk to the backend's question limit and cap a larger maxQuestions, and client.limits and resolveBackend().limits expose the limits without a request.
@DevMortimer
DevMortimer marked this pull request as ready for review October 15, 2026 01:08
@DevMortimer
DevMortimer merged commit 058a18d into main Oct 15, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant