Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
bundle:
name: pipeline-testing

resources:
pipelines:
orders:
name: orders
libraries:
- glob:
include: ./transformations/**
testing:
sources:
- glob:
include: ./tests/**
environment:
dependencies:
- -e .[test]
- -r ./tests/requirements.txt
- ./dist/*.whl
2 changes: 2 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["", "true"]
18 changes: 18 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

>>> [CLI] bundle validate -o json
{
"environment": {
"dependencies": [
"-e .[test]",
"-r ./tests/requirements.txt",
"./dist/*.whl"
]
},
"sources": [
{
"glob": {
"include": "./tests/**"
}
}
]
}
6 changes: 6 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[project]
name = "orders"
version = "0.1"

[project.optional-dependencies]
test = ["faker"]
1 change: 1 addition & 0 deletions acceptance/bundle/resources/pipelines/testing/script
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
trace $CLI bundle validate -o json | jq .resources.pipelines.orders.testing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this test testing now that the property comes from the SDK spec?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On its own it covered nothing beyond the SDK type. It now lives in #7437, where it checks the translated test paths and the uploaded wheel.

4 changes: 4 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Runs only against the local test server.
Cloud = false
RecordRequests = true
Ignore = [".databricks"]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
faker
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def test_orders():
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from pyspark import pipelines as dp
14 changes: 14 additions & 0 deletions bundle/internal/schema/annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1465,6 +1465,20 @@ resources:
"user_name":
"description": |-
The name of the user granted the permission level.
"testing":
"description": |-
The test configuration of the pipeline. Only test-only updates, such as those started by `databricks pipelines test`, use it.
"$fields":
"environment":
"$fields":
"dependencies":
"description": |-
A list of files that declare the packages the tests need: a project folder, a requirements file, or a wheel, in the workspace or a UC volume. A project folder can take `-e` and an extra, for example `-e ./project[test]`. A relative path, such as `-e ./project[test]` or `-r ./tests/requirements.txt`, is relative to the YAML file that defines it. Package names and URLs are rejected.
"sources":
"$fields":
"glob":
"description": |-
A `.py` test file, or a folder followed by `/**`. Other wildcards are rejected, and the sources must not overlap the pipeline's `libraries`. A relative path is relative to the YAML file that defines it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need the override; the description comes from upstream.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed. The descriptions now come from the SDK.

"trigger":
"deprecation_message": |-
Use continuous instead
Expand Down
6 changes: 3 additions & 3 deletions bundle/schema/jsonschema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/pipelines/_models/pipeline.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading