Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/release-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,3 +191,71 @@ jobs:
with:
name: wheel
path: python/dist/*

# Wraps the signed binaries from the cli job into PyPI wheels (project
# "databricks"). Kept as a separate artifact from "wheel" (databricks-bundles)
# so each artifact maps to one PyPI project in the databricks-cli release workflow.
cli-wheels:
needs: cli
runs-on:
group: databricks-protected-runner-group-large
labels: linux-ubuntu-latest-large

permissions:
id-token: write
contents: read

steps:
# The script and LICENSE come from the workflow's own ref, like the
# local actions in the other jobs, so building an older tag still works.
- name: Checkout workflow ref
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.sha }}
sparse-checkout: |
.github
tools/build_cli_wheels.py
tools/validate_cli_wheels.py
tools/requirements.txt
LICENSE

- name: Setup JFrog
uses: ./.github/actions/setup-jfrog

- name: Install uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
with:
version: "0.8.9"

- name: Download signed CLI binaries
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: cli
path: cli-artifact

- name: Build wheels
run: python3 tools/build_cli_wheels.py cli-artifact cli-wheels

- name: Check wheel metadata
run: uvx twine@6.2.0 check --strict cli-wheels/*

- name: Validate wheels
run: uv run -q -p ">=3.11" --no-project --with-requirements tools/requirements.txt python tools/validate_cli_wheels.py cli-wheels

# uv picks the wheel for the runner's platform from the directory, as it would
# from PyPI. --no-index keeps it from resolving "databricks" on the index.
- name: Smoke test the wheel for this runner
run: |
cli_version=$(basename cli-artifact/databricks_cli_*_SHA256SUMS)
cli_version=${cli_version#databricks_cli_}
cli_version=${cli_version%_SHA256SUMS}
uv venv "$RUNNER_TEMP/venv"
uv pip install --python "$RUNNER_TEMP/venv" --no-index --find-links cli-wheels databricks
"$RUNNER_TEMP/venv/bin/databricks" --version | tee /dev/stderr | grep -F "$cli_version"

- name: Upload CLI wheels
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: cli-wheels
path: cli-wheels/*.whl
if-no-files-found: error
8 changes: 8 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,13 @@ tasks:
cmds:
- 'uv run -p ">=3.11" --no-project python tools/validate_codeowners.py'

test-cli-wheels:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intentionally not used and different from the script in release-build.yml?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used: test-cli-wheels is in checks (Taskfile.yml, the checks list), which runs in check.yml. It is different on purpose: the task unit-tests the builder and validator on fake input, while release-build runs the two scripts on the real release binaries. Clarified that in the task, and both now use the same uv run options and the same requirements file.

# release-build.yml runs the scripts themselves on the release binaries.
desc: Unit-test the scripts that build and validate the CLI wheels for PyPI
dir: tools
cmds:
- 'uv run -q -p ">=3.11" --no-project --with-requirements requirements.txt python -m unittest -q test_build_cli_wheels test_validate_cli_wheels'

changelog-preview:
desc: Print the CHANGELOG.md section the next release would add from .nextchanges/
env:
Expand Down Expand Up @@ -314,6 +321,7 @@ tasks:
- task: check-codeowners
- task: check-lockfiles
- task: check-agents-setup
- task: test-cli-wheels

install-pythons:
desc: Install Python 3.9-3.13 via uv
Expand Down
40 changes: 40 additions & 0 deletions acceptance/cmd/pypi-wheel/make_archives.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Fake goreleaser output for the CLI wheel builder.

Only the archive for the platform running the test holds the CLI binary. The
others hold a placeholder that cannot be executed, so running the installed
`databricks` fails if uv picks another platform's wheel.
"""

import io
import platform
import sys
import tarfile
import zipfile
from pathlib import Path

GOOS = {"Linux": "linux", "Darwin": "darwin", "Windows": "windows"}
GOARCH = {"x86_64": "amd64", "amd64": "amd64", "aarch64": "arm64", "arm64": "arm64"}

cli, out = Path(sys.argv[1]), Path(sys.argv[2])
host = (GOOS[platform.system()], GOARCH[platform.machine().lower()])
placeholder = b"placeholder for a binary of another platform\n"

out.mkdir()
for goos in ("linux", "darwin", "windows"):
for goarch in ("amd64", "arm64"):
data = cli.read_bytes() if (goos, goarch) == host else placeholder
if goos == "windows":
path = out / f"databricks_cli_1.2.3_{goos}_{goarch}.zip"
with zipfile.ZipFile(path, "w") as z:
z.writestr("databricks.exe", data)
else:
path = out / f"databricks_cli_1.2.3_{goos}_{goarch}.tar.gz"
# Compression only costs test time: the builder reads the archive either way.
with tarfile.open(path, "w:gz", compresslevel=0) as t:
info = tarfile.TarInfo("databricks")
info.size = len(data)
info.mode = 0o755
t.addfile(info, io.BytesIO(data))
# Only the file name is used: it carries the version.
(out / "databricks_cli_1.2.3_SHA256SUMS").touch()
1 change: 1 addition & 0 deletions acceptance/cmd/pypi-wheel/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
114 changes: 114 additions & 0 deletions acceptance/cmd/pypi-wheel/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@

>>> python3 make_archives.py [CLI] archives

>>> ls archives
databricks_cli_1.2.3_SHA256SUMS
databricks_cli_1.2.3_darwin_amd64.tar.gz
databricks_cli_1.2.3_darwin_arm64.tar.gz
databricks_cli_1.2.3_linux_amd64.tar.gz
databricks_cli_1.2.3_linux_arm64.tar.gz
databricks_cli_1.2.3_windows_amd64.zip
databricks_cli_1.2.3_windows_arm64.zip

>>> python3 [TESTROOT]/../tools/build_cli_wheels.py archives wheels [TESTROOT]/../LICENSE
wheels/databricks-1.2.3-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.musllinux_1_1_x86_64.whl
wheels/databricks-1.2.3-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl
wheels/databricks-1.2.3-py3-none-macosx_13_0_x86_64.whl
wheels/databricks-1.2.3-py3-none-macosx_13_0_arm64.whl
wheels/databricks-1.2.3-py3-none-win_amd64.whl
wheels/databricks-1.2.3-py3-none-win_arm64.whl

>>> uv run -q --no-project --with packaging python [TESTROOT]/../tools/validate_cli_wheels.py wheels
valid: wheels/databricks-1.2.3-py3-none-macosx_13_0_arm64.whl
valid: wheels/databricks-1.2.3-py3-none-macosx_13_0_x86_64.whl
valid: wheels/databricks-1.2.3-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl
valid: wheels/databricks-1.2.3-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.musllinux_1_1_x86_64.whl
valid: wheels/databricks-1.2.3-py3-none-win_amd64.whl
valid: wheels/databricks-1.2.3-py3-none-win_arm64.whl

>>> python3 show_wheels.py wheels
== databricks-1.2.3-py3-none-macosx_13_0_arm64.whl
100755 databricks-1.2.3.data/scripts/databricks
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-macosx_13_0_arm64
== databricks-1.2.3-py3-none-macosx_13_0_x86_64.whl
100755 databricks-1.2.3.data/scripts/databricks
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-macosx_13_0_x86_64
== databricks-1.2.3-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl
100755 databricks-1.2.3.data/scripts/databricks
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-manylinux_2_17_aarch64
Tag: py3-none-manylinux2014_aarch64
Tag: py3-none-musllinux_1_1_aarch64
== databricks-1.2.3-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.musllinux_1_1_x86_64.whl
100755 databricks-1.2.3.data/scripts/databricks
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-manylinux_2_17_x86_64
Tag: py3-none-manylinux2014_x86_64
Tag: py3-none-musllinux_1_1_x86_64
== databricks-1.2.3-py3-none-win_amd64.whl
100755 databricks-1.2.3.data/scripts/databricks.exe
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-win_amd64
== databricks-1.2.3-py3-none-win_arm64.whl
100755 databricks-1.2.3.data/scripts/databricks.exe
100644 databricks-1.2.3.dist-info/METADATA
100644 databricks-1.2.3.dist-info/WHEEL
100644 databricks-1.2.3.dist-info/licenses/LICENSE
100644 databricks-1.2.3.dist-info/RECORD
Wheel-Version: 1.0
Generator: databricks-cli build_cli_wheels.py
Root-Is-Purelib: false
Tag: py3-none-win_arm64
== METADATA (same in every wheel)
Metadata-Version: 2.4
Name: databricks
Version: 1.2.3
Summary: Databricks CLI
Project-URL: Source, https://github.com/databricks/cli
Project-URL: Documentation, https://docs.databricks.com/dev-tools/cli/
License-File: LICENSE
Description-Content-Type: text/markdown

# Databricks CLI

Command-line interface for Databricks. This package installs the prebuilt `databricks` binary.

Run it with `uvx databricks` or `pipx run databricks`, or install it with `uv tool install databricks`.

Documentation: https://docs.databricks.com/dev-tools/cli/
Source: https://github.com/databricks/cli

>>> uvx -q --offline --no-index --find-links wheels --from databricks==1.2.3 databricks --version
Databricks CLI v[CLI_VERSION]
15 changes: 15 additions & 0 deletions acceptance/cmd/pypi-wheel/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Package the CLI under test the way release-build does, then run it through uvx
# to check that a wheel built by build_cli_wheels.py is usable by Python tooling.
trace python3 make_archives.py $CLI archives
trace ls archives
trace python3 $TESTROOT/../tools/build_cli_wheels.py archives wheels $TESTROOT/../LICENSE
# Offline, so packaging comes from the vendored packages rather than the
# tools/requirements.txt pin (they can differ after a Dependabot bump).
trace uv run -q --no-project --with packaging python $TESTROOT/../tools/validate_cli_wheels.py wheels
trace python3 show_wheels.py wheels

# uv must pick the wheel for this platform: the other wheels hold a placeholder
# instead of the binary (see make_archives.py), so a wrong pick fails here.
trace uvx -q --offline --no-index --find-links wheels --from databricks==1.2.3 databricks --version

rm -fr archives wheels
23 changes: 23 additions & 0 deletions acceptance/cmd/pypi-wheel/show_wheels.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env python3
"""Print what each wheel in a directory installs, in a platform-independent form.

RECORD is not printed: its hashes cover the CLI binary, which differs per build.
"""

import sys
import zipfile
from pathlib import Path

wheels = sorted(Path(sys.argv[1]).glob("*.whl"))
for whl in wheels:
print(f"== {whl.name}")
with zipfile.ZipFile(whl) as z:
for info in z.infolist():
print(f"{info.external_attr >> 16:o} {info.filename}")
wheel_file = next(n for n in z.namelist() if n.endswith(".dist-info/WHEEL"))
print(z.read(wheel_file).decode(), end="")

with zipfile.ZipFile(wheels[0]) as z:
metadata_file = next(n for n in z.namelist() if n.endswith(".dist-info/METADATA"))
print("== METADATA (same in every wheel)")
print(z.read(metadata_file).decode(), end="")
5 changes: 5 additions & 0 deletions acceptance/cmd/pypi-wheel/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Cloud = false

# uvx may need to create a Python environment.
Timeout = '180s'
TimeoutWindows = '240s'
Loading
Loading