Phase 1: refactor engine to one shared context with batched decode

[FuJacob]

Follow-up to #3. The Phase 0 spike measured a 1.43x-2.35x throughput win for batched decode on M-series Metal; this PR makes the engine actually use it.

(Re-opening because #4 closed when its base branch feat/batched-decode-bench was deleted on merge of #3. Same content, rebased onto main.)

Summary

Replaces the per-sequence llama_context architecture with a single shared context (n_seq_max = MAX_SEQUENCES) and a dedicated decoder thread that coalesces sample-step requests from multiple sequences into one llama_decode call. Public C++ API (CotabbyInferenceEngine.h) is unchanged; Cotabby's Swift code does not need to be modified.

Design

• One shared llama_context, n_ctx = configured * MAX_SEQUENCES. Each SequenceState owns a llama_seq_id slot (0..MAX_SEQUENCES-1).
• Decoder thread loop: wait for a request, wait an additional BATCH_WINDOW_MICROS (200 µs) for siblings, build one llama_batch carrying all pending tokens with their seq_ids, llama_decode once, sample each sequence's next token using its own sampler chain at its assigned batch index, resolve every request's std::promise.
• sampleNext fast path: deliver the seed token sampled at decodePrompt time. No decoder round-trip on the first sample after a prompt.
• sampleNext steady-state path: queue a PendingRequest and wait on the promise.
• decodePrompt holds decode_mutex for its chunk decodes and takes the seed sample inline while the prompt's logits are still resident.
• trimKV holds decode_mutex, calls llama_memory_seq_rm, and invalidates pending seed/input so the caller re-primes via decodePrompt.

The 200 µs window is the throughput knob. Multi-sequence workloads naturally fall into lockstep because each sequence resubmits as soon as its sample returns. Single-sequence callers pay one window per token (~2% of a ~10 ms decode).

Validation

swift build
# Build complete!

COTABBY_TEST_MODEL_PATH=...Qwen3-0.6B-Q4_K_M.gguf swift test
# Executed 12 tests, with 0 failures

Two new tests gated on COTABBY_TEST_MODEL_PATH:

• testInterleavedMultiSequenceSampling: alternates sampleNext between two sequences with greedy sampling and identical prompts; asserts identical token output. Validates the seed-token / feedback-decode handoff and per-sequence sampler isolation in the shared context.
• testCancellationStopsSamplingPromptly: verifies sampleNext after cancelSequence returns was_cancelled=true without model work.

Existing testEndToEndWithModel passes unchanged — verifies the API stayed source-compatible for the single-sequence flow Cotabby uses today.

Follow-ups

• Bench scenario c_engine_threaded that exercises the engine via its public API from two threads, for end-to-end throughput validation.
• README update: the "no shared decode mutex, no contention" line is no longer accurate. The new design has one decode_mutex serializing the single llama_context. The contention is productive (it enables batching), but the docs should reflect the new model.
• Cotabby side (tabby-4): no code changes needed for correctness; SPM ref pinning + comment touch-ups in a small follow-up PR.

Risk / rollout notes

• Memory footprint goes up: n_ctx is multiplied by MAX_SEQUENCES = 4, so KV cache memory ~4x. For Gemma 3 1B with 2048 ctx, that's ~107 MB — tolerable on M-series. Worth checking on lower-spec hardware.
• The seed-token-in-decodePrompt path is new. decodePrompt holds decode_mutex for its entire body, which serializes against the decoder thread. Verified with the interleaved-multi-sequence test.
• Cancellation latency is now bounded by one batch (~10 ms) rather than zero — a cancelled token's decode slot is still consumed even though the sampler is skipped. Acceptable trade-off.