Support external model providers in Slurm runs #1079

Activity

1. dhruvnathawani

   added a commit that references this issue on Oct 23, 2026

   feat(slurm): support external model providers

   ...

   This issue will close once commit 6cef606 is merged into the 'main' branch.

   6cef606
2. nabinchha commented on Oct 23, 2026

   

   nabinchha

   on Oct 23, 2026

   Contributor

   More actions

   Could we keep hosted-provider configuration entirely in DD’s existing model-provider settings, with the Slurm config describing only the models we deploy locally? That would preserve hosted-only and mixed runs without introducing invocation.model_providers or a parallel provider schema.

   Concretely, I’m thinking of the following flow:

   1. Users configure hosted providers through DD as usual.

   Their normal model_providers.yaml contains the endpoint, provider type, request settings, and credential reference:

   providers:
     - name: hosted-judge
       endpoint: https://models.example/v1
       provider_type: openai
       api_key: MODEL_API_KEY

   The recipe’s ModelConfig references provider: hosted-judge. Users export MODEL_API_KEY in the submit environment before running the Slurm command; they don’t repeat the provider or credential configuration in the Slurm YAML.

   2. Submission resolves which models Slurm owns.

   Aliases listed in deployments get managed servers and generated runtime endpoints. For every remaining alias, submission resolves its provider through DD’s normal settings. We validate missing providers, ambiguous assignments, and provider-name collisions before submitting a job.

   3. Capture provider settings as run inputs.

   Submission snapshots only the hosted providers required by the recipe into the internal run bundle, keeping credential references rather than values. That makes the run independent of later edits to the user’s provider settings and avoids assuming their home directory is available on compute nodes.

   This also matters because the client currently uses an isolated DATA_DESIGNER_HOME per attempt.

   4. Reuse the existing credential transport.

   Discover the required environment-variable references from those providers, validate their availability, and forward their values only to the client steps. Values should stay out of the persisted plans and generated scripts.

   I’d initially support environment-backed API keys explicitly. Plaintext keys, sensitive headers, and custom secret resolvers need a defined policy so we don’t accidentally persist credentials. Any additional secret-reference support should belong to DD’s provider contract and be reusable outside Slurm.

   5. Combine providers at execution time.

   The compute-node client loads the captured hosted providers, adds the generated local providers, and passes the combined list to DD. Hosted model identities and inference settings remain intact; only locally deployed aliases get endpoint overrides.

   With no deployments, this becomes a CPU-only run. With local deployments, it remains a GPU run using the existing async execution, sharding, retry/resume, and collection machinery.

   The mixed-run utilization tradeoff still exists: GPUs remain allocated until the shard finishes, including hosted requests. We should document that clearly and recommend separate stages when hosted processing is substantial. This proposal simplifies configuration ownership; it doesn’t claim to solve that scheduling tradeoff.
3. nabinchha

   added 2 commits that reference this issue on Oct 27, 2026

   feat(slurm): support hosted models through native DD provider settings (

   ...

   This issue will close once commit f109e32 is merged into the 'main' branch.

   f109e32

   feat(slurm): support hosted models through native DD provider settings (

   ...

   This issue will close once commit e6d3277 is merged into the 'main' branch.

   e6d3277
4. andreatnvidia commented on Oct 28, 2026

   

   andreatnvidia

   on Oct 28, 2026

   Contributor

   More actions

   Implemented in #1080 on feat/slurm-execution, with hosted-only and mixed-provider runs verified. Closing ahead of the feature branch's upcoming merge into main via #1040.
5. andreatnvidia

   closed this as completedon Oct 28, 2026