Repository navigation
feat(providers): add LLMTR to the free directory #7955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
knowhycodata
wants to merge
3
commits into
lidge-jun:dev
Choose a base branch
from
knowhycodata:add-llmtr-provider-dev
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| { | ||
| "object": "list", | ||
| "data": [ | ||
| { | ||
| "id": "qwen/qwen3.8-27b-free", | ||
| "supported_operations": ["CHAT_COMPLETIONS"], | ||
| "supported_endpoints": ["/v1/chat/completions"] | ||
| }, | ||
| { | ||
| "id": "llmtr/qwen3-8-27b", | ||
| "supported_operations": ["CHAT_COMPLETIONS"], | ||
| "supported_endpoints": ["/v1/chat/completions"] | ||
| }, | ||
| { | ||
| "id": "google/gemini-3.8-flash", | ||
| "supported_operations": ["CHAT_COMPLETIONS", "RESPONSES"], | ||
| "supported_endpoints": ["/v1/chat/completions", "/v1/responses"] | ||
| }, | ||
| { | ||
| "id": "openai/gpt-image-2", | ||
| "supported_operations": ["IMAGES_GENERATIONS"], | ||
| "supported_endpoints": ["/v1/images/generations"] | ||
| }, | ||
| { | ||
| "id": "llmtr/embeddingmagibu-200m", | ||
| "supported_operations": ["EMBEDDINGS"], | ||
| "supported_endpoints": ["/v1/embeddings"] | ||
| }, | ||
| { | ||
| "id": "jina/rerank", | ||
| "supported_operations": ["RERANK"], | ||
| "supported_endpoints": ["/v1/rerank"] | ||
| }, | ||
| { | ||
| "id": "google/veo-3.1-generate-001", | ||
| "supported_operations": ["VIDEO_GENERATIONS"], | ||
| "supported_endpoints": ["/v1/video/generations"] | ||
| }, | ||
| { | ||
| "id": "openai/gpt-6-luna", | ||
| "supported_operations": ["RESPONSES"], | ||
| "supported_endpoints": ["/v1/responses"] | ||
| }, | ||
| { | ||
| "id": "vendor/chat-without-endpoint", | ||
| "supported_operations": ["CHAT_COMPLETIONS"], | ||
| "supported_endpoints": ["/v1/responses"] | ||
| }, | ||
| { | ||
| "id": "vendor/endpoint-without-operation", | ||
| "supported_operations": ["RESPONSES"], | ||
| "supported_endpoints": ["/v1/chat/completions"] | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| import { describe, expect, test } from "bun:test"; | ||
| import { readFileSync } from "node:fs"; | ||
| import { fixturePath } from "../helpers/repo-root"; | ||
| import { FREE_PROVIDER_DIRECTORY } from "../../src/providers/free-directory"; | ||
| import { extractProviderModelItems, providerModelDiscoverySpecError, resolveProviderModelDiscovery } from "../../src/providers/model-discovery"; | ||
| import { PROVIDER_REGISTRY } from "../../src/providers/registry"; | ||
|
|
||
| // Sibling of provider-registry-parity.test.ts, which sits at the file-size ratchet threshold. | ||
| describe("free-provider directory: LLMTR", () => { | ||
| test("the LLMTR row bounds its free models and keeps the paid catalog discoverable", () => { | ||
| // The row accepts a key. Live discovery keeps chat rows from /v1/models and drops the rest. | ||
| // Six chat ids were the standing zero-price set on 2026-10-29 (public GET /v1/models and the | ||
| // model cards): three reset on a daily quota and three were listed free with no end date. Dated | ||
| // promotions and EVREN bring-your-own-key rows also list a zero price, so the note must carry | ||
| // those bounds instead of advertising the whole catalog as free. | ||
| const row = FREE_PROVIDER_DIRECTORY.find(entry => entry.id === "llmtr"); | ||
| expect(row).toMatchObject({ | ||
| label: "LLMTR", | ||
| adapter: "openai-chat", | ||
| baseUrl: "https://llmtr.com/v1", | ||
| authKind: "key", | ||
| accessGroups: ["recurring-or-keyless"], | ||
| supportLevel: "experimental", | ||
| verification: "official", | ||
| documentationUrl: "https://llmtr.com/docs/gateway/chat-completions/", | ||
| modelsUrl: "https://llmtr.com/v1/models", | ||
| dashboardUrl: "https://llmtr.com/dashboard/api-keys", | ||
| discovery: "hybrid", | ||
| liveModels: true, | ||
| lastVerified: "2026-10-29", | ||
| }); | ||
| expect(row?.keyOptional).toBeUndefined(); | ||
| expect(row?.models).toEqual([ | ||
| "nvidia/nemotron-3-ultra-550b-a55b", | ||
| "openai/gpt-oss-safeguard-20b", | ||
| "qwen/qwen3.8-27b-free", | ||
| "nvidia/nemotron-3.5-super-vl-free", | ||
| "agnes/agnes-3.0-flash", | ||
| "agnes/agnes-2.5-flash", | ||
| ]); | ||
| const note = row?.note ?? ""; | ||
| expect(note).toContain("qwen/qwen3.8-27b-free"); | ||
| expect(note).toContain("daily quota"); | ||
| expect(note).toContain("A key is still required"); | ||
| expect(note).toContain("dated promotions"); | ||
| expect(note).toContain("EVREN bring-your-own-key"); | ||
| expect(note).toContain("Models that are neither listed above nor covered by those promotions or routes are paid"); | ||
| expect(note).toContain("llmtr/qwen3-8-27b"); | ||
| expect(note).toContain("https://llmtr.com/pricing"); | ||
| expect(row?.modelDiscovery).toEqual({ | ||
| filter: { | ||
| allOf: [ | ||
| { path: ["supported_operations"], containsAll: ["CHAT_COMPLETIONS"] }, | ||
| { path: ["supported_endpoints"], containsAll: ["/v1/chat/completions"] }, | ||
| ], | ||
| }, | ||
| }); | ||
| expect(providerModelDiscoverySpecError(row!.modelDiscovery!)).toBeNull(); | ||
| // Directory-only: it must not become a canonical credential destination. | ||
| expect(PROVIDER_REGISTRY.some(entry => entry.id === "llmtr")).toBe(false); | ||
| }); | ||
|
|
||
| test("live discovery keeps chat rows and drops image, embedding, rerank, video, and responses-only rows", () => { | ||
| const provider = { adapter: "openai-chat", baseUrl: "https://llmtr.com/v1" }; | ||
| const discovery = resolveProviderModelDiscovery("llmtr", provider); | ||
| expect(discovery.spec).toEqual(FREE_PROVIDER_DIRECTORY.find(entry => entry.id === "llmtr")?.modelDiscovery); | ||
| // A renamed row still points at the fixed destination, so it keeps the same filter. | ||
| expect(resolveProviderModelDiscovery("llmtr-gateway", provider).spec).toEqual(discovery.spec); | ||
| expect(resolveProviderModelDiscovery("llmtr", { ...provider, baseUrl: "https://llmtr.com/v1/" }).spec).toEqual(discovery.spec); | ||
| expect(resolveProviderModelDiscovery("llmtr", { adapter: "openai-chat", baseUrl: "https://example.test/v1" }).spec).toBeUndefined(); | ||
| expect(resolveProviderModelDiscovery("llmtr", { ...provider, adapter: "anthropic" }).spec).toBeUndefined(); | ||
| expect(resolveProviderModelDiscovery("llmtr", { ...provider, authMode: "oauth" }).spec).toBeUndefined(); | ||
| // A registry name whose transport does not match stays on that registry result. Pointing the | ||
| // row at LLMTR's host must not pick up the directory chat filter. | ||
| const mismatched = resolveProviderModelDiscovery("nebius", { | ||
| adapter: "anthropic", | ||
| baseUrl: "https://llmtr.com/v1", | ||
| authMode: "key", | ||
| }); | ||
| expect(mismatched.spec).toBeUndefined(); | ||
| expect(mismatched).toEqual(resolveProviderModelDiscovery("nebius", { | ||
| adapter: "anthropic", | ||
| baseUrl: "https://api.tokenfactory.nebius.com/v1", | ||
| authMode: "key", | ||
| })); | ||
|
|
||
| const catalog = JSON.parse(readFileSync(fixturePath("llmtr-model-discovery.json"), "utf8")) as unknown; | ||
| const result = extractProviderModelItems(catalog, discovery); | ||
| expect(result).toMatchObject({ ok: true }); | ||
| if (!result.ok) return; | ||
| expect(result.items.map(item => item.id)).toEqual([ | ||
| "qwen/qwen3.8-27b-free", | ||
| "llmtr/qwen3-8-27b", | ||
| "google/gemini-3.8-flash", | ||
| ]); | ||
| }); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: lidge-jun/opencodex
Length of output: 45100
Script executed:
Repository: lidge-jun/opencodex
Length of output: 42209
Seed LLMTR’s directory models into new provider configs.
The LLMTR row describes its six IDs as seeded, but the provider setup path does not copy
FREE_PROVIDER_DIRECTORY[].modelsinto saved config. For a newly configured provider at the LLMTR destination, if live discovery fails before a cache exists, the configured-model fallback is empty. Seed the savedmodelsfrom the matching directory row for new providers, while preserving explicit model lists and keeping LLMTR out of the canonical registry.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. This applies to every directory row today, not just LLMTR: nothing in the setup path reads FREE_PROVIDER_DIRECTORY[].models (runcrate and sealion behave the same way). I'd rather keep this PR to the LLMTR row and its discovery filter, and handle seeding generically in a follow-up if the maintainers want it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@knowhycodata Agreed. Keep
#7955focused on the LLMTR directory row and its discovery filter. The shared setup-path limitation does not need an LLMTR-specific fix in this PR.If maintainers want generic seeding, the follow-up should preserve explicit model lists and cover discovery failure before a cache exists.
Would you like me to open a follow-up issue for that work?
You are interacting with an AI system.