Repository navigation
Add the testing property of pipelines to bundles - #7300
tharunbhuk-db wants to merge 3 commits into
Conversation
cf934e0 to
cc2e61a
Compare
52e5f6b to
9c9d9c8
Compare
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 4 tests: regression=2 unreleased=2 coverage=0 failing=0
TestAccept/bundle/environments/dependencies/DMS=: branch ✅ | main (0f6d067) ❌ | latest (v1.26.0) ❌main (0f6d067): latest (v1.26.0): same output as main (0f6d067) TestAccept/bundle/environments/dependencies/DMS=true: branch ✅ | main (0f6d067) ❌ | latest (v1.26.0) ❌main (0f6d067): latest (v1.26.0): same output as main (0f6d067) TestAccept/bundle/resources/pipelines/testing/DMS=: branch ✅ | main (0f6d067) ✅ | latest (v1.26.0) ❌latest (v1.26.0): TestAccept/bundle/resources/pipelines/testing/DMS=true: branch ✅ | main (0f6d067) ✅ | latest (v1.26.0) ❌latest (v1.26.0): |
✅ Integration test reportCommit: 115d175
Top 3 slowest tests (at least 2 minutes):
|
cc2e61a to
b4df318
Compare
9c9d9c8 to
742ef87
Compare
e038a13 to
e895d35
Compare
b4df318 to
7c61f68
Compare
| } | ||
| var diags diag.Diagnostics | ||
| pipelines := b.Config.Resources.Pipelines | ||
| for _, key := range slices.Sorted(maps.Keys(pipelines)) { |
There was a problem hiding this comment.
pipeline_development_deprecated.go right above this in initialize.go does the same walk with dyn.MapByPattern on resources.pipelines.*.development, which gives the path and locations directly. Can we do the same here for resources.pipelines.*.testing instead of iterating the typed map and calling dyn.GetByPath per key?
There was a problem hiding this comment.
Done in 13c0eb2. The check now uses dyn.MapByPattern on resources.pipelines.*.testing, like the development check.
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| func TestPipelineTestingEnabled(t *testing.T) { |
There was a problem hiding this comment.
Do we need this unit test? The acceptance test already shows the error with its location and path (the env -u ... bundle validate step) and the enabled case, and we try not to unit test mutators that an acceptance output covers.
There was a problem hiding this comment.
Removed in 13c0eb2. The acceptance test covers the error with its location and path, and the enabled case.
| "file_path", | ||
| }, | ||
| "pipelines": { | ||
| // The experimental testing property is not deployed yet. |
There was a problem hiding this comment.
If this lands before #7438, a bundle with the variable on and testing set deploys fine and testing is silently dropped (PrepareState copies only CreatePipeline and CascadeOnDestroy). Fine for an experimental flag while the stack merges together, or do we want validate to say the property isn't deployed yet?
There was a problem hiding this comment.
This is out of date. The field now comes from the Go SDK's CreatePipeline, so PrepareState copies it and this PR already deploys testing; main's direct engine sends it too. The knownMissingInStateType entry is gone.
4c21f8b to
33085b7
Compare
33085b7 to
86997dc
Compare
2a9184c to
24212b4
Compare
24212b4 to
76d7d05
Compare
86997dc to
5a146d7
Compare
76d7d05 to
a19ae14
Compare
5a146d7 to
13c0eb2
Compare
commented
Nov 17, 2026
|
Status of the notes from the review above:
|
a19ae14 to
9253416
Compare
13c0eb2 to
2c5d853
Compare
9253416 to
3409615
Compare
2c5d853 to
c14224e
Compare
| "$fields": | ||
| "glob": | ||
| "description": |- | ||
| A `.py` test file, or a folder followed by `/**`. Other wildcards are rejected, and the sources must not overlap the pipeline's `libraries`. A relative path is relative to the YAML file that defines it. |
There was a problem hiding this comment.
We don't need the override; the description comes from upstream.
There was a problem hiding this comment.
Agreed, removed. The descriptions now come from the SDK.
| @@ -0,0 +1 @@ | |||
| trace $CLI bundle validate -o json | jq .resources.pipelines.orders.testing | |||
There was a problem hiding this comment.
What is this test testing now that the property comes from the SDK spec?
There was a problem hiding this comment.
On its own it covered nothing beyond the SDK type. It now lives in #7437, where it checks the translated test paths and the uploaded wheel.
Add resources.pipelines.<key>.testing, with the test sources and test-only dependencies that databricks pipelines test runs with. The field comes from the Go SDK's CreatePipeline. The CLI adds its own descriptions for it. The property is experimental. It needs DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true. While the variable is off, bundle validate rejects testing and names the variable. Bundles without testing are not changed. Later PRs translate its paths. Co-authored-by: Isaac <no-reply@databricks.com>
The check walks resources.pipelines.*.testing with dyn.MapByPattern, like the check of the development property, which gives the path and the locations directly. The acceptance test covers the error and the enabled case, so the unit test is removed. Co-authored-by: Isaac <no-reply@databricks.com>
The Go SDK now has the testing field, so the property needs no separate opt-in. Co-authored-by: Isaac <no-reply@databricks.com>
c14224e to
115d175
Compare
🥞 Stacked PR
Adds
resources.pipelines.<key>.testing(sources,environment.dependencies), with the type from the Go SDK. Adds descriptions fortesting,dependenciesandglob, and regenerates the JSON schema and Python models.Bundles without
testingdo not change.Tests: acceptance
bundle/resources/pipelines/testing.This pull request and its description were written by Isaac.