bundle: don't set edition/channel defaults for ingestion pipelines#5418
Draft
shreyas-goenka wants to merge 3 commits into
Draft
bundle: don't set edition/channel defaults for ingestion pipelines#5418shreyas-goenka wants to merge 3 commits into
shreyas-goenka wants to merge 3 commits into
Conversation
Ingestion pipelines (those with `ingestion_definition`) use serverless compute by default. The Databricks API rejects `edition` and `channel` fields as "cluster settings" when a pipeline runs on serverless compute. Previously, DABs unconditionally set `edition: ADVANCED` and `channel: CURRENT` as defaults for all pipeline types. This caused creation of LakeFlow Connect ingestion pipelines to fail with: "cannot create pipeline: You cannot provide cluster settings when using serverless compute" The fix conditionally applies these defaults only to pipelines without `ingestion_definition`. The test `change-ingestion-definition` is restructured to use per-engine output files since the TF provider independently adds these defaults for all pipeline types. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
|
Commit: 9c7c82c |
Replace the dyn.Value pattern walk with a direct loop over b.Config.Resources.Pipelines to conditionally apply edition/channel defaults. Also remove the now-unused setPipelineDefault helper. Add acceptance/bundle/resources/pipelines/ingestion-defaults, a cloud test (direct engine) that deploys an ingestion pipeline and asserts edition and channel are absent from the POST /api/2.0/pipelines request body. The deploy fails with "connection not found" — not "cluster settings" — confirming the fix. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
changed the title
Extends the ingestion-defaults acceptance test to run both direct and terraform engines, showing the contrast: - Direct engine: edition/channel absent from POST (DABs fix working) - Terraform engine: edition/channel present (provider-level SetDefault, companion fix in terraform-provider-databricks#5783) Badness clause documents the known terraform engine gap. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ingestion pipelines (
ingestion_definition) use serverless compute. Settingeditionorchannelon them causes:This PR skips those defaults for ingestion pipelines. Regular DLT pipelines are unaffected.
Note: The Terraform provider applies the same defaults unconditionally at the provider level (
resource_pipeline.go:253), so a companion fix is needed there for TF-engine users.