[CerebralCoding]

What this changes

Adds experimental native SAM3/SAM3.1 segmentation through tf-sam3-image and tf-sam3-video, including prompt-seeded tracking. This is a draft for the remaining fidelity, provenance and performance review; it is not ready to merge. Inference uses Zig/custom Metal. No HTTP route is registered.

• Issue: Refs Native SAM3/SAM3.1 segmentation and tracking on Metal #2991.
• Stacked on: main at d6c4cce602eef4aff235625a5325bb617a9674ff, with 18 linear SAM commits and no merge commits. Main advanced during verification to 923acc35caf2102a083b9577a289363f35cd2cb9 with unrelated tooling changes; a merge preview against that tip is clean and its native backend code is unchanged.
• atc overlap --json: final selected check on 2026-11-14 at 16:55 UTC against upstream/main reported complete:true, no limitations and nine clean resolved overlaps: idol: Idol's denoiser as a native Metal family #1766, idol: Idol's denoiser on CUDA (stacked on #1766) #1767, metal nemotron: TF_KV_SHARE keeps prompt states as core pool pages (stacked on #1837) #1850, DeepSeek-V4.1-Flash on Metal: the EXL3 2.9 bpw lane #2373, kernel contract: MiniMax-H3-Image's 19 Metal kernels, 31 registrations and 34 planted controls; the prompt path's attention keeps an infinite last value #2808, h3: MiniMax-H3-Image's edits end to end: Qwen3-VL's vision tower with DeepStack (opt-in), tf-h3-image-vision, and the prompt tool's references #2810, qwen27 Metal prompt attention: threadgroups past the causal edge exit early (#2912) #2926, speech: wav24, wav32f and FLAC replies, a sample_rate and stereo, each opt-in per request #3138, DeepSeek-V4.1 on Metal: M5 prompt rows' FP8 projections on tensor units by default (quality-gated, 17-39% faster prefill) #3354. The full snapshot covered 364 open PRs, every page of the 23 large file lists and all 121 overlapping heads. It found no added conflict paths: the same 75 raw branch conflicts occur on unchanged main, and all 68 conflict blocks in shared changed files are identical to main after normalizing marker labels. A final head refresh found six new/updated PRs, all with no shared changed files. SAM's reader imports, source/launch registrations and limits filter were isolated to remove its introduced conflicts.

The remaining shared-file conflicts are inherited from main for #1038, #1635, #1773, #1774, #1775, #1780, #1787, #1835, #1858, #1859, #1860, #1861, #1862, #2008, #2020, #2077, #2231, #2260, #2273, #2302, #2410, #2475, #2514, #2522, #2523, #2634, #2944, #3146. No peer implementation was merged into this branch.

The original branch contained adapted SAM-licensed implementations. Commit 89ee04b77 replaced the vision, prompt, decoder, memory and tracker implementations with new Zig modules written from architecture/configuration, checkpoint geometry and external reference measurements, then removed the associated SAM notice under the approved cleanup. That provenance change needs review. The current diff against its upstream base changes no project licence or notice files. No Python reference tools, checkpoints or public movie assets are added in the current PR diff.

Checks on this head

• Head commit: 9217969bdb1c733f2fc4becf42f35d464db3e06d.
• Lean checker, python3 tools/zig/lean_check.py, on base and head: 341 inherited findings on each, zero new. The raw head checker exits 1 for those inherited findings. python3 tools/pr_check.py --base upstream/main exits 0 with one advisory warning about the not-yet-assigned PR number.
• Tests run: zig build test test-golden test-sam3 test-kernel-contract fuzz-cortheon dev-tools -Dcpu=apple_m1 -Doptimize=fast -Dversion=1.0.0 -j4 --summary all succeeds: 1,119/1,119 steps, 2,494/2,538 tests executed with 44 skipped and zero failures; other test results were cached. Server goldens have zero unexpected differences. The plain build, zig fmt --check build.zig zig and git diff --check also pass.
• Additional checks: Drama3 CUDA/Metal host compilation, Drama3 host/quantization tests, residency compilation and slide host tests pass. The shared Torch archive reader is reused through explicit module imports.
• CI: the pushed fork head reports zero check runs. No upstream PR CI has run yet.
• Not run: physical M1-M4/other M5 chips, CUDA/HIP devices, Windows or app integration. The offline Metal toolchain is unavailable. The full Metal runner stops before a SUMMARY with ControlEditMissing, exit 1, on both the exact unchanged base and this head on the same Mac.

Receipt

• Main against branch: main has no SAM implementation. Against the prior SAM checkpoint, the retained exact-FP16-weight path keeps FP32 activations/accumulation and reduces peak Metal allocation from 4,281,778,176 to 3,704,537,088 bytes. The final warm tracking measurement is 1.673183 FPS; the matched PyTorch reference is 1.760044 FPS. The speed target is still open.
• Boxes it reaches: native SAM checkpoint loading, image/prompt/decoder/memory/tracker operations, CLI tools, shared Torch reader imports, tokenizer Lowercase support and guarded Metal tensor products.
• New switch? --fp32, --split-attention, --device-projections, --device-attention and --compensated-normalization are boolean CLI flags, off by default. Default precision follows checkpoint dtype; two-pass attention and compact GEMMs default on. The measured profile enables all five flags. Default checkpoint-precision and other combinations remain unqualified.
• Environment: base d6c4cce602eef4aff235625a5325bb617a9674ff, head above; Zig 0.17.0, Xcode 27.2 (27B52), Metal runtime compilation, macOS 27.2.1, Apple M5 Max, 128 GiB RAM. SAM3.1 checkpoint SHA-256 9ba99c92703c2e8b4f47de2d34a539bb8e18923049e238b780d70dbe6368eb03. Tested executable SHA-256 cd3291c1942dfe1192b009a428e0c3e27e9e5dede5e85c5f1847aa41fa396f52.
• Paths and platforms: SAM3 .pt and SAM3.1 .safetensors on the M5 Max. No other hardware qualification is claimed.
• Exactness, from the committed tf-sam3-compare tool: final SAM3.1 matches all 120 PyTorch masks, including seed, across 39,813,120 pixels; min/mean IoU 1. The checked first nine visible Big Buck Bunny frames and nine Sintel frames match independent FP64 masks exactly. SAM3's two checked point-prompt frames also match FP64 exactly. The Big Buck Bunny object is visible for nine frames and absent after the cut, so this is limited development coverage. Lifecycle exercises pass cancellation, exactly-one duplicate-frame admission, reset/ID reuse, concurrent versus solo byte identity and three closes returning to the warmed allocation baseline.
• Main against this branch, same machine/checkpoint: same-machine full-backend runs both reproduce ControlEditMissing; full non-regression qualification remains blocked. SAM operator gates below pass. The comparison tools are in the diff; selected raw warm-frame receipts are below.
• Decode speed before and after, from tools/bench_openai.py: n/a, SAM produces masks rather than autoregressive tokens. Tracking uses identical checkpoint/input hashes, point prompt ID=42, x=384, y=240, 120 RGB frames of 432×768 and warm frames 20-119. Native: 59.766329 seconds/100 frames, 1.673183 FPS. Meta PyTorch 2.15.0/MPS with official compilation enabled, FP32/highest matmul precision and no CPU fallback/profiler/trace hooks: 56.816776 seconds/100 frames, 1.760044 FPS.
• Prompt speed before and after: n/a for text-server TTFT; SAM image and prompt stage timings are recorded by its native receipt. A full public text/point/box prompt matrix has not been qualified.
• Licence of any new weights or drafter: n/a, no weights or drafter are distributed. Users supply checkpoints under their own terms. Public local fixtures were derived from Blender Foundation's Big Buck Bunny and Sintel, used under CC BY 3.0; those assets are not distributed here.

Reproduce the native tracking/profile comparison with supplied checkpoint, frames and reference masks:

zig build sam3 -Dcpu=apple_m1 -Doptimize=fast -Dversion=1.0.0
zig-out/bin/tf-sam3-video "$CHECKPOINT" "$FRAMES" "$OUT" --point 42 384 240 --frames 120 --fp32 --split-attention --device-projections --device-attention --compensated-normalization --reference "$REFERENCE" --raw-masks "$OUT.safetensors" --benchmark "$OUT.json" --warmup 19
zig-out/bin/tf-sam3-compare "$REFERENCE" "$OUT.safetensors"

The public-frame tensor SHA-256 is 6c527e3ea851138c0069955ed78a49f92200cec3e5cfb62e50a2efe33862b7af; prompt SHA-256 9b0baef9e917c59c2478bfae2e4a1ad1cb5843e66fc46c680dee4580f7bfb770; final mask SHA-256 01aae0116ebcfd9b579b911e06667f4ebf2d96c1de10fba3151b8296e3724ec6.

Kernel contract

• Registration: zig/src/core/contract/sam3_records.zig, the shared runner and limits mapping cover 104 SAM launches. Groups cover FP32 MMA/GEMM, projections, normalization, attention, deconvolution, resizing and sparse mask memory.
• Replacements include exact-half-weight device projection for FP32-expanded-weight projection (22 shapes byte-identical), four-output deconvolution for the single-output baseline, small compensated projection for its separate baseline, and D256 attention for separate value halves. Compensated arithmetic changes reduction order to retain operand/residual bits; it is assessed independently against FP64, not claimed identical to an older approximate path.
• Planted controls: 31, all red as planted, including FP16 output rounding, omitted residuals, reduced precision sums and subnormal flushing.
• Metal on M5 Max, tf-kernel-contract --only sam3, exit 0:

SUMMARY 80 kernels: 80 pass gates 1-4, 0 fail, 0 known findings; 31 controls: 31 red as planted, 0 missed; 0 limit problems; gates 276560 ms, limits 179 ms

• Full Metal: tf-kernel-contract --rotary-accuracy FILE, exit 1 with ControlEditMissing before SUMMARY on both base and branch. CUDA/HIP: not implemented/run for SAM.
• Chips not run: M1-M4 and all other M5 variants. Static declared limits across M1-M5 passed; that is not physical-chip verification.

Remaining qualification

Before a confirmatory run, agree and freeze the public image/clip/prompt set and stage thresholds in this description. Proposed strict gate: report worst/mean absolute FP64 error for image encoder, prompt encoder, decoder logits and memory attention, with neither metric worse than matched PyTorch FP32 against the same FP64 tensors; require exact final binary masks (min/mean IoU 1) for every admitted prompt/frame/object. These development receipts do not complete that gate.

• Provenance review of the rewritten implementations.
• Fixed public image set and both clips across text/point/box prompts, both model versions and admitted profiles; complete per-stage FP64 and per-frame fidelity receipts.
• Full backend non-regression once the inherited control failure is resolved.
• Matched alternating performance runs meet the user's requirement to beat PyTorch without losing correctness.
• Default precision/profile and every admitted hardware path are qualified.

Checklist

• Title and description match this head and describe its incomplete qualification.
• New-test red/green and missing-dependency behavior are fully qualified. The rebased bundle failed shader compilation before its shared-helper fix; GPU gates now pass and controls are caught. Other-chip/missing-dependency execution remains unrun.
• Comments/docstrings are one line; no measurements or task history were added to source comments.
• No private paths, machine names, internal hosts or AI attribution lines in the outgoing diff/commits; public no-reply authorship is used.
• Every token goes through lane rounds: n/a, these are segmentation tools.
• Drafted/resumed/concurrent token equality: n/a for tokens; SAM concurrent/solo and reset/ID reuse checks pass.
• No precision traded for speed across every admitted profile: the reported FP32 path preserves source weight bits, but other profiles and full fidelity remain unqualified.
• Decode/prompt processing no slower than main at all streams: n/a for autoregressive serving; the SAM speed target remains unmet.

Raw warm-frame timing receipts (seconds, frames 20-119)

{
  "native": {
    "frame_span": [
      20,
      119
    ],
    "frames": 100,
    "seconds": 59.76632904156577,
    "fps": 1.6731829042143924,
    "executable_sha256": "cd3291c1942dfe1192b009a428e0c3e27e9e5dede5e85c5f1847aa41fa396f52",
    "checkpoint_sha256": "9ba99c92703c2e8b4f47de2d34a539bb8e18923049e238b780d70dbe6368eb03",
    "frames_tensor_sha256": "6c527e3ea851138c0069955ed78a49f92200cec3e5cfb62e50a2efe33862b7af",
    "prompts_sha256": "9b0baef9e917c59c2478bfae2e4a1ad1cb5843e66fc46c680dee4580f7bfb770",
    "masks_sha256": "01aae0116ebcfd9b579b911e06667f4ebf2d96c1de10fba3151b8296e3724ec6",
    "peak_metal_bytes": 3704537088,
    "frame_seconds": [
      0.5919139583711512,
      0.5922420416609384,
      0.5923399166204035,
      0.5917733333189972,
      0.5924582500010729,
      0.7441658333409578,
      0.5923722083098255,
      0.5910909583326429,
      0.5919806250021793,
      0.5924113333458081,
      0.5914085000404157,
      0.5920498333289288,
      0.5921072500059381,
      0.592106333351694,
      0.5924505416187458,
      0.5912811667076312,
      0.5912718749605119,
      0.5915340416831896,
      0.5911799583700486,
      0.5922861250001006,
      0.592280708369799,
      0.5913557499879971,
      0.592216249962803,
      0.5918011666508391,
      0.591631083341781,
      0.5915659583406523,
      0.5922717500361614,
      0.5918840416707098,
      0.5922671250300482,
      0.5920510833384469,
      0.5924375833128579,
      0.5924320833291858,
      0.5920182500267401,
      0.5921778333140537,
      0.5919254999607801,
      0.5914441249915399,
      0.5926622083061375,
      0.5920631249900907,
      0.5914935000473633,
      0.5921495417132974,
      0.5912027916638181,
      0.5923241666750982,
      0.5928216666798107,
      0.5921447916189209,
      0.6298293332802132,
      0.5924456249922514,
      0.5929655833169818,
      0.5920900000492111,
      0.5916650833096355,
      0.5920624583377503,
      0.5923740416765213,
      0.5919499166775495,
      0.5926912083523348,
      0.5929296666290611,
      0.592253000009805,
      0.592470166622661,
      0.768484250002075,
      0.5926339167053811,
      0.5922748750308529,
      0.5930943333078176,
      0.592944833333604,
      0.5970283333444968,
      0.7458425833028741,
      0.6007488333270885,
      0.5931639583432116,
      0.5939067083527334,
      0.5937398750102147,
      0.5933290832908824,
      0.5952571666566655,
      0.5940133749973029,
      0.5945644166786224,
      0.5943043333245441,
      0.5920394583372399,
      0.5928653333103284,
      0.5924415000481531,
      0.5922284583211876,
      0.5923957499908283,
      0.5924989582854323,
      0.5923729166388512,
      0.5927252083201893,
      0.5917805833742023,
      0.5920252916403115,
      0.5923807500512339,
      0.5917107083369046,
      0.5925568749662489,
      0.592621166666504,
      0.5917967916466296,
      0.592874749971088,
      0.593405875028111,
      0.5928326666471548,
      0.5925112083205022,
      0.5925571250263602,
      0.592628541693557,
      0.5926207916345447,
      0.5916230832808651,
      0.5921819999930449,
      0.5921534167136997,
      0.59162416664185,
      0.5930162916774862,
      0.5917562500108033
    ]
  },
  "pytorch": {
    "torch": "2.15.0",
    "precision": "fp32",
    "compile": true,
    "official_compile_activated": true,
    "float32_matmul_precision": "highest",
    "source_tree_sha256": "2aa1297728e62f573b457da9ac9d8a78727750e849711baa1cd1802daef28232",
    "checkpoint_sha256": "9ba99c92703c2e8b4f47de2d34a539bb8e18923049e238b780d70dbe6368eb03",
    "frames_tensor_sha256": "6c527e3ea851138c0069955ed78a49f92200cec3e5cfb62e50a2efe33862b7af",
    "cpu_fallback": false,
    "timing_has_operator_profiler": false,
    "timing_has_trace_hooks": false,
    "frame_span": [
      20,
      119
    ],
    "frame_seconds": [
      0.5733130410080776,
      0.573931458988227,
      0.5630721249908675,
      0.5638667499879375,
      0.5679205000051297,
      0.5735725830018055,
      0.5727364589984063,
      0.5623515839979518,
      0.5725884170096833,
      0.5620870419952553,
      0.5620957920036744,
      0.5635768749925774,
      0.5638538750063162,
      0.5625732079788577,
      0.5648257919820026,
      0.5749587909958791,
      0.572497207991546,
      0.5707089999923483,
      0.5647555420000572,
      0.5736642919946462,
      0.5617579160025343,
      0.5625704579870217,
      0.5741876249958295,
      0.5743779999902472,
      0.5735428750049323,
      0.5640011659997981,
      0.5622612079896498,
      0.5735218329937197,
      0.5733619579987135,
      0.5745894580031745,
      0.571523792023072,
      0.5663066249981057,
      0.5629537499917205,
      0.5636856670025736,
      0.5628070829843637,
      0.5625562079949304,
      0.5631047500064597,
      0.5744571250106674,
      0.562346416991204,
      0.5617533749900758,
      0.5626138329971582,
      0.5622399999992922,
      0.5621195000130683,
      0.5623165419965517,
      0.5621300420025364,
      0.5729943749902304,
      0.5722236250003334,
      0.574486124998657,
      0.5641136250051204,
      0.5628831250069197,
      0.565113000018755,
      0.5647269169858191,
      0.5631341669941321,
      0.5632127499848139,
      0.5631808749749325,
      0.5716714999871328,
      0.5638546669797506,
      0.5667885419970844,
      0.5718263329763431,
      0.5734690829995088,
      0.5745444589992985,
      0.5746815419988707,
      0.5713490410125814,
      0.565189167013159,
      0.5671860000002198,
      0.5729924159822986,
      0.5731413750036154,
      0.5721130830061156,
      0.5678040839848109,
      0.5704034579975996,
      0.5675551659951452,
      0.5657692500099074,
      0.56260766700143,
      0.5732414170051925,
      0.5638802080065943,
      0.5754325419839006,
      0.5666987920121755,
      0.5733301659929566,
      0.5639890420134179,
      0.5695827080053277,
      0.573164875007933,
      0.5719806669803802,
      0.5740309169923421,
      0.574049874994671,
      0.5712822079949547,
      0.5627801670052577,
      0.5696755409881007,
      0.5729474580148235,
      0.5627550409990363,
      0.564799249987118,
      0.5636244160123169,
      0.563427999994019,
      0.5630292919813655,
      0.5670140830043238,
      0.5687557920173276,
      0.5752169999759644,
      0.575529458001256,
      0.5761394590081181,
      0.5742907079984434,
      0.5710735419997945
    ],
    "frames": 100,
    "seconds": 56.816775581770344,
    "fps": 1.7600435606571976
  }
}