feat(slurm): support hosted models through native DD provider settings

fern/versions/latest/pages/slurm/overview.mdx

@@ -1,6 +1,6 @@
 # Data Designer on Slurm
 
-Data Designer Slurm runs Data Designer clients and vLLM model servers in batch allocations. The optional extension owns profile selection, image inspection, submission, status, retry, result collection, and benchmark workflows.
+Data Designer Slurm runs Data Designer clients and optional vLLM model servers in batch allocations. Recipes can combine local models with external providers, or use CPU allocations for hosted-only generation. The optional extension owns profile selection, image inspection, submission, status, retry, result collection, and benchmark workflows.
 
 Start with [Getting Started](/slurm/getting-started). Use [Dependencies and tools](/slurm/dependencies-and-tools) for plugins, MCP providers, and secrets, and keep the [CLI reference](/slurm/cli-reference) nearby for automation.
 
@@ -12,7 +12,7 @@ The submit environment needs:
 - `sbatch`, `squeue`, `sacct`, `scancel`, `scontrol`, and `sinfo`
 - Access to a shared workspace that is visible from the submit host and every eligible compute node
 
-Compute nodes need access to the same shared Python environment at the exact path used by the submit command. Enroot 3.5 or later is required for both digest-pinned OCI imports and existing SQSH inspection. Serving images must contain vLLM.
+Compute nodes need access to the same shared Python environment at the exact path used by the submit command. Local serving requires Enroot 3.5 or later for both digest-pinned OCI imports and existing SQSH inspection, and serving images must contain vLLM. Hosted-only runs require access to their external endpoints and do not need serving images or Enroot.
 
 Your Slurm administrator can confirm the account, GPU partition, CPU image-build partition, GPU request mode, and shared filesystem paths for your cluster.
 
@@ -34,6 +34,6 @@ Runtime bundle extraction, the client dependency overlay, Data Designer home and
 
 ## Supported in v1
 
-The public v1 surface supports named cluster profiles, serving image aliases, one or more vLLM deployments, independent replicas, multi-node replicas, Slurm arrays, bounded array concurrency, durable status, cancellation, sparse retry, winner-driven collection, dependencies, plugins, MCP providers, and benchmarks.
+The public v1 surface supports named cluster profiles, serving image aliases, vLLM deployments, external model providers, hosted-only CPU runs, independent replicas, multi-node replicas, Slurm arrays, bounded array concurrency, durable status, cancellation, sparse retry, winner-driven collection, dependencies, plugins, MCP providers, and benchmarks. See [External model providers](/slurm/run-configuration#external-model-providers) for mixed and hosted-only configuration.
 
 Interactive allocations, code-sandbox sidecars, NeMo Run, SGLang, Dynamo, Kubernetes, Ray, and platform-specific orchestration are not part of the v1 Slurm extension.

fern/versions/latest/pages/slurm/profiles.mdx

@@ -85,7 +85,9 @@ scheduler:
 
 The CLI also accepts `--partition gpu-a,gpu-b` during profile initialization. Slurm chooses a partition when scheduling the job; the CPU `image_build.partition` remains a single partition.
 
-`gpus_per_node` is both the GPU count used for deployment topology and, in `gres` mode, the count requested from Slurm on each job node. `auto` queries eligible nodes and uses their reported count, so an eight-GPU partition produces `--gres=gpu:8`. Set an explicit smaller count, such as `1`, for partial-node allocations when the cluster permits them. Use `auto` only when requesting every GPU on each selected node is intended, and use an explicit count when eligible nodes are heterogeneous.
+For runs with local deployments, `gpus_per_node` is both the GPU count used for deployment topology and, in `gres` mode, the count requested from Slurm on each job node. `auto` queries eligible nodes and uses their reported count, so an eight-GPU partition produces `--gres=gpu:8`. Set an explicit smaller count, such as `1`, for partial-node allocations when the cluster permits them. Use `auto` only when requesting every GPU on each selected node is intended, and use an explicit count when eligible nodes are heterogeneous.
+
+[Hosted-only runs](/slurm/run-configuration#hosted-only-cpu-runs) skip GPU discovery and ignore the profile's GPU count and `mem_per_gpu`. Select a CPU partition with `submission.partition` in the run file. Each task requests one node, `--gres=none`, and `client.cpus` CPUs.
 
 Multi-node generation jobs request exclusive nodes, including in `gres` mode. This keeps serving ports isolated because Slurm reports batch-job GPU IDs for the first node only. A partial-node `gpus_per_node` setting still limits the GPUs used per node, but other jobs cannot share those nodes during the run.
 

fern/versions/latest/pages/slurm/run-configuration.mdx

@@ -36,9 +36,9 @@ submission:
 
 `builder` accepts exactly one complete serialized Data Designer config through `source` or `inline`. Relative source paths resolve from the run file directory. `invocation.run_config` accepts public `RunConfig` fields except `display_tui`, which is disabled for Slurm jobs. `input_bindings.seed_path` and `managed_assets_path` accept absolute compute-visible paths.
 
-Every `model_alias` in `model_concurrency` and the builder must have one deployment. Deployment aliases must be unique. `model` selects the Hugging Face model ID or absolute model path loaded by the serving container. `served_model_name` is the name exposed to the client and defaults to `model`; set it explicitly when `model` is a container-visible path but the builder uses a stable model name. Shard and collected output files are Parquet. Choose the destination and partition count with `slurm merge` after the run completes.
+Each builder model alias listed in `deployments` is served locally. Every remaining alias must reference a provider in DD's normal `model_providers.yaml` settings. `model_concurrency` accepts any declared alias, including external models. Deployment aliases must be unique. `model` selects the Hugging Face model ID or absolute model path loaded by the serving container. `served_model_name` is the name exposed to the client and defaults to `model`; set it explicitly when `model` is a container-visible path but the builder uses a stable model name. Shard and collected output files are Parquet. Choose the destination and partition count with `slurm merge` after the run completes.
 
-The client runs from the same shared Python environment as the CLI on the first allocated GPU node. Its step requests no GPUs; the serving steps use the GPUs and their registered images. The `client` block is optional; `client.cpus` defaults to 16. It sets the client step's CPU request and contributes to the batch allocation's CPU request.
+The client runs from the same shared Python environment as the CLI on the first allocated node. Its step requests no GPUs; local serving steps use the GPUs and their registered images. The `client` block is optional; `client.cpus` defaults to 16. It sets the client step's CPU request and contributes to the batch allocation's CPU request.
 
 Use `submission.partition` to override the profile's GPU partition selection for one run. It accepts a comma-separated value or a YAML list, such as:
 
@@ -47,7 +47,87 @@ submission:
   partition: [gpu-a, gpu-b]
 ```
 
-Slurm schedules the job in one of the selected partitions. With `gpus_per_node: auto`, every selected partition must report GPU resources with one consistent count.
+Slurm schedules the job in one of the selected partitions. For runs with local deployments and `gpus_per_node: auto`, every selected partition must report GPU resources with one consistent count.
+
+## External model providers
+
+Configure hosted endpoints in DD's normal provider settings on the host where you run the Slurm submission command. The default file is `~/.data-designer/model_providers.yaml`.
+
+`DATA_DESIGNER_HOME` is an optional environment variable that changes DD's settings directory. If it is unset, DD uses `~/.data-designer`. To use a different directory, set it before launching the Slurm CLI:
+
+```bash
+export DATA_DESIGNER_HOME="$HOME/dd-settings"
+mkdir -p "$DATA_DESIGNER_HOME"
+```
+
+With this override, the provider file is `$DATA_DESIGNER_HOME/model_providers.yaml`. See [Default Model Settings](/concepts/models/default-model-settings#how-default-model-providers-and-configurations-work) for more about DD's settings files.
+
+Add the hosted provider to the `providers` list in that file. Provider definitions are separate from the recipe's `model_configs` and the Slurm run file:
+
+```yaml
+providers:
+  - name: hosted
+    endpoint: https://models.example/v1
+    provider_type: openai
+    api_key: MODEL_API_KEY
+```
+
+Export `MODEL_API_KEY` in the submit environment. Keep its value out of configuration files. The recipe references the provider through its normal `ModelConfig`:
+
+```yaml
+data_designer:
+  model_configs:
+    - alias: generator
+      model: example/generator
+      provider: openai
+    - alias: judge
+      model: example/hosted-judge
+      provider: hosted
+  # Keep the recipe's columns and inference parameters here.
+```
+
+The Slurm run deploys only `generator`. There is no separate hosted-provider block in the run file:
+
+```yaml
+invocation:
+  num_records: 1000
+  dataset_name: judged-output
+  model_concurrency:
+    generator: 64
+    judge: 16
+```
+
+Submission captures only the required hosted providers in the resolved run inputs, using DD's native provider representation. Later edits to submit-side settings do not change that run, and compute nodes do not need access to the submit host's settings directory. Missing or ambiguous providers and collisions with generated names such as `slurm-generator` fail before submission. Only locally deployed aliases receive generated endpoints; hosted model names and inference settings stay intact. An explicit `model_concurrency` value overrides the selected alias's request concurrency.
+
+API keys must reference environment variables. Slurm forwards their values only to native client steps through runtime credential transport, including retries. Values stay out of persisted plans and generated scripts. Plaintext API keys, sensitive headers, and custom secret resolvers are not supported for hosted Slurm providers. Non-secret `extra_body` parameters and static headers use the normal DD provider fields.
+
+Both local and external requests use DD's existing async engine. Ready requests for different records and models can overlap. Async overlap does not guarantee continuous GPU utilization: slow external calls and the final judging tail can still retain an idle GPU allocation. Measure utilization against the cluster's policy when deciding whether to use separate recipe stages.
+
+### Hosted-only CPU runs
+
+Slurm runs require at least one explicitly configured model. Sampler-only recipes are not currently supported.
+
+For a builder whose models all reference configured hosted providers, omit local deployments or set `deployments: []`. Select the cluster's CPU partition:
+
+```yaml
+schema_version: 1
+name: hosted-judging
+builder:
+  source: hosted-builder.yaml
+invocation:
+  num_records: 1000
+  dataset_name: hosted-judging
+client:
+  cpus: 4
+deployments: []
+submission:
+  partition: cpu
+  memory: 16G
+```
+
+Each array task requests one node with `--gres=none` and `--cpus-per-task` set to `client.cpus`. It does not discover GPUs, reserve exclusive nodes, require serving images, or start serving containers and proxies. The profile's GPU facts and GPU memory request are unused for this run. Sharding, status, retry, resume, and collection use the same commands as local-model runs.
+
+Set `submission.memory` to the memory needed per node, such as `16G`. It emits `--mem` for generation and collection jobs and takes precedence over the profile's `mem_per_gpu` request for generation. If omitted, local deployments retain the profile's GPU memory request; otherwise Slurm's memory default applies. Some clusters default to reserving all memory on a node, even for a small CPU-only job.
 
 ## Configure vLLM
 
@@ -169,7 +249,7 @@ Increasing `resources.nodes` at TP=1 creates more independent one-GPU servers; i
 
 ### Multiple models and images
 
-Add one deployment per builder model alias. Each deployment may use a different registered serving image:
+Add one deployment per locally served builder model alias. Each deployment may use a different registered serving image:
 
 ```yaml
 deployments:

packages/data-designer-slurm/examples/hosted-only.yaml

@@ -0,0 +1,25 @@
+schema_version: 1
+name: hosted-judging
+builder:
+  inline:
+    data_designer:
+      model_configs:
+        - alias: judge
+          model: example/hosted-judge
+          provider: hosted
+      columns:
+        - name: review
+          column_type: llm-text
+          model_alias: judge
+          prompt: Explain one property of a useful product description.
+invocation:
+  num_records: 100
+  dataset_name: hosted-judging
+  model_concurrency:
+    judge: 16
+client:
+  cpus: 4
+deployments: []
+submission:
+  partition: cpu
+  memory: 16G

packages/data-designer-slurm/examples/mixed-providers.yaml

@@ -0,0 +1,36 @@
+schema_version: 1
+name: local-generation-hosted-review
+builder:
+  inline:
+    data_designer:
+      model_configs:
+        - alias: generator
+          model: example/generator
+          provider: openai
+        - alias: judge
+          model: example/hosted-judge
+          provider: hosted
+      columns:
+        - name: description
+          column_type: llm-text
+          model_alias: generator
+          prompt: Write a short description of an everyday household product.
+        - name: review
+          column_type: llm-text
+          model_alias: judge
+          prompt: "Review this product description for clarity: {{ description }}"
+invocation:
+  num_records: 100
+  dataset_name: judged-descriptions
+  model_concurrency:
+    generator: 32
+    judge: 16
+deployments:
+  - model_alias: generator
+    model: example/generator
+    server:
+      type: vllm
+      image:
+        name: vllm
+    topology:
+      tensor_parallel: 8

packages/data-designer-slurm/examples/model_providers.yaml

@@ -0,0 +1,7 @@
+# Copy this provider into your normal DATA_DESIGNER_HOME/model_providers.yaml.
+# Export MODEL_API_KEY in the submit environment; do not store its value here.
+providers:
+  - name: hosted
+    endpoint: https://models.example/v1
+    provider_type: openai
+    api_key: MODEL_API_KEY

packages/data-designer-slurm/src/data_designer/slurm/_errors.py

@@ -42,6 +42,13 @@
             "shared_node requires GRES GPU request mode",
             "shared_node requires the same resolved serving image",
             "deployment placement and GPU ownership must match authored resources",
+            "model concurrency references undeclared aliases",
+            "external provider names must not collide with managed providers",
+            "model provider names must be unique",
+            "client-only runs must resolve zero GPUs",
+            "server deployments require a positive GPU count",
+            "resolved external model aliases do not match the inline builder",
+            "external model aliases require external providers",
         )
     },
     "Value error, builder content digest does not match the resolved input": (