Skip to content

cuda sampler: add separate repetition kernels and contracts - #3154

Draft
gilp456 wants to merge 3 commits into
ashhart:lead/models-input-typesfrom
gilp456:codex/glm-repetition-kernels
Draft

gilp456 wants to merge 3 commits into
ashhart:lead/models-input-typesfrom
gilp456:codex/glm-repetition-kernels

Conversation

@gilp456

@gilp456 gilp456 commented Nov 13, 2026 •

Copy link
Copy Markdown

What this changes

Adds separate CUDA repetition sampler entry points and registered numerical contracts while preserving the legacy identity sampler source path. This layer has no production engine caller; GLM wiring is supplied by the final stack PR.

Stack order: #3153, this kernel/contract layer, then #3155. All three drafts target lead/models-input-types; this PR's review range is the single commit above.

Checks on this head

  • Head commit: 8ee0d8d23.
  • Lean checker: 345 findings on the target base and head, zero new findings.
  • Tests run:
    • zig build test-kernel-contract --summary all: 34/34 steps, 240/240 tests passed.
    • zig build test test-kernel-contract test-golden fuzz-cortheon --summary all: exit 0 with zero failures and 222 platform skips.
    • zig fmt --check build.zig zig, git diff --check, and tools/pr_check.py: pass; 49 changed files cumulatively and zero new lean findings.
  • CI: pending on this draft head.
  • Not run, and why: CUDA compilation/device execution, compiler-derived resource records, legacy-kernel SASS hashes and the CUDA SUMMARY require an authorized CUDA toolchain/device.
  • Plain zig build: 26/29 steps succeeded; tf-qwen35-cuda hits seven Windows std.c/fabric-network errors. The exact target base reproduces the same seven errors and 26/29 result; this stack does not change that code.
  • Cluster activity: zero SSH, deployment, service, Docker or model-process commands.

Receipt

  • Main against branch: n/a for served inference; this layer has no production engine caller.
  • Boxes it reaches: CUDA sampler source, repetition-only entry points, generated reference fixture, registered contract runner and launch inventory.
  • New switch? Default, accepted values, profile row: n/a.
  • Environment: Windows host, Zig 0.17.0; host contract inventory above; CUDA compiler/device unavailable.
  • Paths and platforms the change reaches, which I ran and which I could not: host registration and fixture checks ran; CUDA SM80/86/89/120/121 execution remains pending.
  • Exactness: source-pinned float32 fixture and host contract wiring pass; CUDA numerical execution remains pending.
  • Main against this branch, same machine and checkpoint: n/a until the final GLM layer supplies a production caller.
  • Decode speed before and after: n/a for this uncalled layer; final-stack served measurements remain pending.
  • Prompt speed before and after: n/a for this uncalled layer.
  • Licence of any new weights or drafter, or "n/a": n/a; no weights or drafter added.
Kernel contract
  • Kernels registered with the contract, and the kernel each one replaces: repetition-only greedy, sampled-token and FP32-probability operations are registered; the legacy identity entry points remain the baseline path.
  • A planted control for each new kernel, and each one fails as planted: ten controls are registered; CUDA red execution remains pending.
  • Each backend's SUMMARY line and exit status: pending CUDA execution; no device SUMMARY is claimed.
  • Chips I could not run: CUDA SM80/86/89/120/121; Metal and HIP are outside this CUDA layer.
  • Resource and SASS evidence: launch geometry is registered with resource usage explicitly pending; legacy-kernel cuobjdump -sass hashes remain pending.

Checklist

  • The title and description match the head commit, receipt and stack.
  • The independent fixture and registry checks pass without a CUDA device.
  • Comments and docstrings are one line. Measurements remain outside source.
  • Publication content contains no personal data, machine names, internal hosts, local paths or AI attribution lines.
  • The production identity path retains its existing ABI and source arithmetic.
  • No precision policy changes are introduced.
  • CUDA planted-control execution and complete SUMMARY.
  • Compiler resource records and identity-kernel SASS hashes.
  • Final-stack model and performance gates.

ashhart and others added 3 commits November 12, 2026 14:04
…ake is a 400

A client could not tell from /v1/models whether a model takes image or
video parts, and a video_url part sent to a server whose model reads video
but which has no libtfvideo (GLM-5.3-Flash with the default image) ended in
HTTP 500 at the first video (ashhart#3004).

Each model entry now ends with "input_types": ["text"], plus "image" with
--vision, plus "video" when the checkpoint offers video and libtfvideo
loads. The keys before it keep their order and bytes. There is no new
option: the server decides it once at startup (input_types.zig), loading
libtfvideo with the loader every video uses, which keeps its answer, and
only when the model offers video. Server.media() takes its image and video
flags from the same Inputs, so every request route (chat, completions,
responses, messages, tokenize) refuses a video_url part on a server that
does not list "video" with the 400 an unsupported part gets
(invalid_request_error). The startup log names the missing library once.
If libtfvideo still fails at a video, that is now a 400 naming the library
instead of a 500.

The 17 server goldens holding a /v1/models reply gain the key on each
entry (the fake engine is text only) and 25 bytes of Content-Length per
entry, and nothing else. New tests in models_input_types_test.zig cover
the listing (text, image, image and video, and video without the library),
the 400 for both kinds of model without video, and the library refusal.
docs/api.md describes the key; docs/vision.md says when libtfvideo loads.

Co-authored-by: WebReflection <85749+WebReflection@users.noreply.github.com>
@gilp456

gilp456 commented Nov 13, 2026

Copy link
Copy Markdown
Author

/atc depends 3153

@ashhart
ashhart force-pushed the lead/models-input-types branch from 0029d0e to eab1e7e Compare November 14, 2026 03:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants