Fix/issue#503: FunctionGemma now usable

[lennartvoelz]

This PR fixes issue #503
After trying overflow prevention in the residual stream, I realized that the issue is the logit sampling. Sampling
directly in FP16 causes numerical breakdown: the softmax and probability normalization lose too much precision across the 256K+ vocabulary, leading to degenerate token selection (repetition, gibberish, wrong/hallucinated tool calls).

I introduced a new sampling kernel, that reads FP16 logins and performs all internal math in fp32 (I previously tried to use the existing cactus_sample_f32 kernel, but the upcasting introduced too much overhead for additional fp32 buffer allocs). This kernel is only used, when a flag (fp32_accumulate in OpParams) is set, so other models that do not need the higher precision in sampling math are not affected.

This is the rough strategy:

• Fused widening convert with scaling and running max acc
• Reduce to a smaller active token list to reduce working set
  -> Remaining work on the really small list
• Allocations: all buffers are static thread_local and reused across tokens
• RNG: static thread_local std::mt19937, seeded once per thread

The kernel is a bit slower but the overhead vs the pure f16 kernel is dominated by BW in the full vocab pass, so basically unavoidable (at least I think so, correct me if I'm wrong).

The result is a much better working model with roughly 370ms average latency (MacBook M3) on the Hackathon dataset. Before the changes, fine tuned models were unusable (described in #503). The new kernel also make the base weight model much better.

Here are the results from the "benchmark":
Fine-tuned & new sampling:

Fine-tuned & old sampling:

Base weights & new sampling:

Base weights & old sampling:

Please note, that this is not a real benchmark in terms of the model latency, because the hackathon benchmark setup returns 0ms on model failure -> the times for the old sampling are heavily skewed towards smaller execution times.
I did not have time to properly benchmark the execution times. If you want me to do that, let me know

[lennartvoelz]

With all the latest fixes applied (see #510), the fine-tuned model now matches the outputs from the Transformers library's reference implementation. This allows all Gemma family models to be fine-tuned and deployed in Cactus :)
This was not possible before the chain of fixes.
Benchmark now with the fine tuned model:

Benchmark now with the base FunctionGemma model:

Who needs cloud fallback now...

[HenryNdubuaku]

@lennartvoelz thanks, you wanna resolve the conflicts

[lennartvoelz]

Replaced with #510