feat(server): place --target-device auto by GPU availability and multi-GPU technique

[Graffioh]

Summary

--target-device auto now places a model by what the GPUs are doing at startup, and by how the model can use more than one GPU.

• Busy GPUs are skipped. On AMD, a GPU is busy when another process has KFD queues there or more than 64 MiB in vram_<gpu_id> (/sys/class/kfd/kfd/proc). Free memory alone cannot tell: Strix Halo reports ~121 GiB free while holding DeepSeek V4 or half of a layer-split Qwen. A server single-GPU or split shows up on every GPU it spans, about a second after it starts and before its weights load. The startup log and --list-devices name it: busy=pid 1154989 luce_server Qwen3.8-27B-IQ4_XS-pure.gguf on hip:0+hip:1.
• Free memory sizes each GPU (HIP reports Strix Halo's total as 1.25 TiB).
• Placement order:
  1. a discrete GPU that holds the model alone;
  2. the architecture's multi-GPU technique over the free GPUs, when together they hold the model: DeepSeek V4/V4.1 expert parallel (--expert-device), layer-split families by layer (weights ∝ free memory). This targets models only Strix Halo can hold, where the discrete GPU's help beats Strix alone;
  3. one free GPU that holds it, discrete first;
  4. the roomiest free GPU for backends with expert offload;
  5. otherwise stop before loading, naming the busy processes.
• With --profile, auto picks only the device, since a profile is qualified for one topology. --target-device auto --profile ds4-strix behaves as before.
• Fix (separate commit): DeepSeek4's uncapped hybrid expert budget reserved 0.5 GiB on the main GPU, while check_device_headroom() requires 1.5 GiB on a discrete GPU. Every R9700 + Strix expert-parallel launch without LUCE_EXPERT_BUDGET_MB failed at 0.99 GiB free. The qualified profiles pin a lower budget, so they never hit it.

qwen4exp (Qwen3.8-Flash-Next) has no layer-split or expert-device path, so auto places it on one free GPU: the R9700 when it fits there, otherwise Strix Halo.

Validation (R9700 hip:0 + Strix Halo hip:1, rebased on c320db9)

DeepSeek-V4-Flash-0731 ROCMFPX-MIX-STRIX (91.5 GiB), --target-device auto --max-ctx 8192, no profile:

R9700	Strix Halo	Placement	Result
free	free	expert parallel hip:0,hip:1, peer access on	loaded, 2.05 GiB free on R9700; answered "Paris"; routed calls 33% R9700 / 67% Strix
busy (Qwen 27B server)	free	Strix Halo alone	answered "Paris"; queues only on Strix, Qwen undisturbed
free	busy	none: R9700 cannot hold it, DS4 cannot offload	refused before loading
busy	busy	none: every GPU is busy	refused before loading

Why a model that fits the discrete GPU stays there: Qwen3.8-27B IQ4_XS (13.5 GiB) forced into an auto layer split ([0,13) R9700, [13,64) Strix Halo) decoded 15.1 tok/s over 128 tokens, against 36.2 tok/s on the R9700 alone. Both were warm, 2 runs each. With this order, auto keeps it on hip:0. The split path itself loaded and answered correctly.

Detection was checked against a Qwen 27B layer split (--target-devices hip:0,hip:1), a single-GPU Qwen server, and a server 2 s into loading.

• test_server_unit 630/630
• test_feature_gate, including new test_launch_policy cases for the availability policy
• test/test_entrypoint.sh PASS

Not covered

• Two servers launched within the same ~1 s can both see a GPU as free.
• CUDA builds have no busy signal and fall back to free memory.
• DS4 expert parallel picked by auto runs without the ds4-r9700-strix tuning. Its 33.7 tok/s on a 2-token reply is not a benchmark. ds4-strix on Strix alone is documented at 42 tok/s with 131K context. Whether DS4 should prefer Strix alone when both GPUs are free is open.

🤖 Generated with Claude Code

[Graffioh]

Review findings addressed

• Exhausted devices: zero queried free bytes previously fell back to total VRAM and could select a full GPU. Live placement now excludes exhausted devices, including offload fallback.
• Feature/topology compatibility: automatic layer splitting could reject a valid paged Qwen launch or discard monolithic-only features. Auto now preserves the requested features and limits paged DeepSeek expert parallelism to the supported HIP gfx1201/gfx1151 pair.
• KFD ownership mapping: matching only PCI domain/bus/device missed or aliased nonzero PCI functions/GPU partitions. Discovery now uses the full HIP PCI address; filesystem-backed regression coverage protects distinct functions and domains.
• All-cold DeepSeek: the hot-budget admission check required an expert round even when zero experts belong to the primary. That requirement is now skipped for all-cold placement, while primary KV accounting and the final headroom check remain intact.

Updated the server reference, including the stale --target-device table and availability limitations.

Verification

• HIP server build succeeded.
• ctest --test-dir server/build-hip-dual --output-on-failure -R '^test_(feature_gate|server_unit)\.': 634/634 passed.
• CPU policy binary compiled with LUCE_BACKEND_CUDA=1: 4/4 suites passed. This is not CUDA hardware validation.
• bash server/test/test_entrypoint.sh server/scripts/entrypoint.sh: PASS.
• Qwen 27B, auto + paged attention + 1,048,576 max context + F16 KV: before, chose a layer split and failed admission; after, selected hip:1, loaded and answered Paris.
• Live --list-devices identified that Qwen process/model on hip:1; auto refused the 91.5 GiB DeepSeek model rather than using the busy GPU.
• DeepSeek all-cold with LUCE_EXPERT_BUDGET_MB=1: before, failed with expert budget is smaller than one uniform expert round; after, loaded all routed experts on hip:1 and answered Paris.
• Uncapped DeepSeek auto expert parallel: loaded hip:0,hip:1, retained 2.05 GiB primary free memory against the 1.50 GiB floor, and answered Paris.
• Final integrated-primary DeepSeek launch on hip:1: retained UMA-aware budget reporting, reserved the 4 GiB floor, loaded and answered Paris.

Availability remains a startup snapshot, not a reservation; simultaneous launches can race. CUDA uses free memory without a busy-process signal. KFD host-PID labels/self-exclusion are not namespace-safe in containers using a separate PID namespace; this is documented. Nonzero PCI-function mapping was verified with filesystem fixtures, not partitioned GPU hardware.

[Graffioh]

Production-readiness follow-up at 38d1ea5c

Ready for review is not production sign-off. The initial verification covered startup and short inference. Additional checks now establish sampled API-output parity and bounded serving stability, not bitwise logits/tensor parity or equivalence against a pre-PR binary.

Checks performed

Same model files/build, temperature=0, seed=1234, thinking disabled, no drafter/profile, prefix cache disabled, maximum 128 output tokens. Four prompts: arithmetic, JSON sorting, Python code generation, and record retrieval (1,446 prompt tokens on DeepSeek; 1,965 on Qwen), each repeated twice.

Comparison	Result
DeepSeek explicit hip:0 + expert hip:1 + peer access vs auto expert parallel	Generated message bytes, finish reason, and completion-token counts matched on all four prompts; repeats stable
DeepSeek Strix-only hip:1 vs auto expert parallel	Same equality on all four prompts; repeats stable
Qwen explicit hip:1 vs auto, F16 KV, two concurrent paged slots	Same equality on all four prompts; repeats stable

Qwen was configured with a 1,048,576-token logical context and a 4,096-token physical pool to exercise the auto-placement feature constraint. This did not test million-token inference.

For each auto configuration, also sent 12 requests through two simultaneous HTTP clients, interrupted a streamed generation after 12 SSE events, then checked two recovery requests. DeepSeek's single-slot backend queued the overlapping requests; Qwen used a two-slot paged backend. No request errors or output drift. Both ended with zero active/parked requests. Total: 68 completed requests, 2,432 generated tokens, plus two intentional stream disconnects.

Strict requested output formatting was not universally obeyed: DeepSeek added a sentence around the correct arithmetic answer and Markdown fences around JSON; Qwen also fenced JSON. The baselines did the same. This is not a broad model-quality pass.

Reasons not to give blanket production approval

• Untuned auto expert parallel was slower than Strix-only in this sample: the DeepSeek retrieval request took 84.2–84.7 s, versus 66.9–67.5 s on Strix alone. This is not a controlled performance qualification, but the automatic preference for both GPUs is not established as a performance win.
• No long-context/long-duration soak, raw-logit oracle, multimodal/drafter coverage, memory-pressure fault campaign, or end-to-end CUDA auto-placement qualification was performed.
• Simultaneous-start placement races and container PID-namespace limitations remain as documented.

The latest Linux/Windows builds, workspace/commit checks, and R9700/Strix/GB10 GPU CI jobs pass. The speed-profile job is skipped.

Recommendation: keep production pinned to a qualified profile/explicit topology; treat the new automatic multi-GPU policy as a controlled canary until workload-specific performance and long-running qualification are complete. No additional source changes or deployment were made in this validation pass.

[davide221]

@cubic

[cubic-dev-ai]

@cubic

@davide221 cubic can't start this review because your workspace has reached its monthly review limit. cubic has reviewed 161,326 of the 160,000 allowed lines of code this month. Reviews resume on 24 October 2026 (in 17 days). Enable flex capacity to cover overages automatically and resume reviews now. Learn how flex capacity works.

To help optimise your usage, you can tune cubic to get the most out of your usage limits:

• Stop reviewing test files
• Turn off incremental reviews
• Use the cubic CLI locally
• Upgrade to a pro plan

Learn more →

[davide221]

The code review bot stopped.

The review failed, so nothing was posted.

Maintainers: see the lucebox_bots run log.

[davide221]

The code review bot stopped.

The review failed, so nothing was posted.

Maintainers: see the lucebox_bots run log.

[davide221]

The code review bot stopped.

The review failed, so nothing was posted.

Maintainers: see the lucebox_bots run log.

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥80% confidence · add the code-review label to review again}

[davide221]

The code review bot stopped.

An unexpected error occurred.

Maintainers: see the lucebox_bots run log.

[davide221]

The code review bot stopped.

The code-review label was removed before the bot started.

Maintainers: see the lucebox_bots run log.

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

Note: the diff reviewer gave no answer.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥80% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥80% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥80% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Code review

Found 1 issue:

1. Multi-GPU auto placement checks only the summed free memory, not each GPU's share (server/src/placement/device_select.h:235-260)

_{Reviewed 38d1ea5c · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 0 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 2 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed 38d1ea5c · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again}