Repository navigation
Conversation
…ub is unreachable
## Changes
`databricks ssh connect` uploads the Linux build of the CLI, at the client's version, to the workspace as the SSH server. Until now it could only download that build from the GitHub release. With this change, when the GitHub download fails, it takes the binary from the CLI's Linux wheel in the `databricks` package on a Python package index.
```text
UploadTunnelReleases
--releases-dir set -> local archives (unchanged)
otherwise -> withPyPIFallback(getGithubRelease, getPyPIRelease)
GitHub OK -> GitHub archive
GitHub fails, dev version -> GitHub error (dev builds are not published)
GitHub fails -> getPyPIRelease
index URL: PIP_INDEX_URL > index-url in per-user pip config > https://pypi.org/simple
GET <index>/databricks/ (PEP 503 HTML page)
pick databricks-<version>-py3-none-<tags>.whl whose tags include a manylinux/musllinux tag for the machine
download it and check its SHA-256
repack <name>.data/scripts/databricks as `databricks` at the root of a zip, the GitHub archive layout
```
- `experimental/ssh/internal/client/pypi.go` (new): finds, downloads and checks the wheel, then builds the release archive from it. The binary keeps its compressed form in the new archive, so it isn't decompressed and compressed again.
- `experimental/ssh/internal/client/releases.go`: `withPyPIFallback` wraps the GitHub provider. If both sources fail, the error includes both causes and lists the hosts to allow (GitHub or PyPI) or the setting to change (`PIP_INDEX_URL` or `index-url`).
- `libs/localenv/uv.go`: exports `pipConfIndexURL` as `PipConfIndexURL`, so this code reads the per-user pip config the same way the uv bootstrap does. Only the name changes. System-wide pip config files are still not read.
- `experimental/ssh/README.md`: new "Network Requirements" section. A changelog fragment is under `.nextchanges/cli/`.
How the wheel is verified:
- A wheel is accepted only if its link on the project page has a `#sha256=<hex>` URL fragment. The downloaded bytes must match that digest. The comparison ignores case.
- A link with no fragment, or with only another hash such as `#md5=`, is refused with an error. Without a SHA-256 digest the wheel can't be verified.
- No other source of hashes is read. That includes the PEP 691 JSON API, which is why the request sends `Accept: text/html`.
How credentials in the index URL are handled (for example `https://user:token@mirror.example/simple`):
- When a wheel link has no userinfo of its own, the index credentials are added only if the link has the index URL's scheme and host. Relative links get them anyway, because they resolve against the index page. A wheel on another host, such as `files.pythonhosted.org`, is fetched without credentials.
- A custom redirect policy compares each redirect target with the first URL of its request chain:
- Same scheme and host: the userinfo is kept, and added back if an absolute redirect left it out.
- Any other scheme or host, including a subdomain or another port: the userinfo and the `Authorization` header are removed. Go's default policy forwards `Authorization` to subdomains, and this one doesn't.
- If the first URL has credentials, a redirect from HTTPS to plain HTTP is refused, even on the same host.
- `Referer` is removed on every redirect, because the previous URL's query can carry a token.
- The 10-redirect limit of Go's default policy still applies.
- Error messages name only the host. The URL that `net/http` adds to its errors is removed, and a parse error doesn't echo the URL.
Requests time out after 10 minutes. At most 16 MiB is read for the project page and 256 MiB for a wheel.
## Why
Some networks block GitHub but allow PyPI, or send pip to a PyPI mirror through `PIP_INDEX_URL` or `pip.conf`. On those networks `ssh connect` failed before it could start the server, even though the same binary is published to PyPI in every CLI release.
## Tests
- `pypi_test.go` runs `getPyPIRelease` against an in-process fake index. It covers:
- absolute links, relative links, and a project page that redirects
- credentials sent to the index host for absolute and relative links
- credentials kept on an absolute redirect back to the index host
- credentials kept off a separate files host and off a redirect to another host
- an upper-case digest
- errors: version not listed, missing project page, digest mismatch, no digest, only an MD5 digest, wheel without the binary, and a file that isn't a wheel
- rejected credentials, an unreachable index and a download that gets cut off, each with an index URL that carries credentials. The test checks that none of these errors contains the username or password.
- `TestCheckPyPIRedirect` checks the redirect policy for the same origin, another host, a subdomain, another port, a target URL that carries credentials, HTTPS to HTTP on the same host and on another host, and the redirect limit. It also checks that a redirect from HTTPS to HTTP is followed when the index URL has no credentials.
- `TestPyPIIndexURL` checks the order `PIP_INDEX_URL`, then pip config, then PyPI. `TestIsLinuxWheel` checks wheel file names with compressed platform tag sets for both machines.
- `releases_test.go`:
- `TestWithPyPIFallback` covers GitHub working, GitHub failing, both failing (both errors are wrapped), and a development version (no fallback).
- `TestWithPyPIFallbackNamesWhatToAllow` checks the remediation text.
- `TestUploadReleasesFromPyPI` uploads the archives built from the wheels to the fake workspace server and reads back the same binaries for amd64 and arm64.
- `go test ./experimental/ssh/internal/client/ ./libs/localenv/` passes.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: c71b79c
Top 3 slowest tests (at least 2 minutes):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks ssh connectuploads the Linux build of the CLI, at the client's version, to the workspace as the SSH server. Until now it could only download that build from the GitHub release. With this change, when the GitHub download fails, it takes the binary from the CLI's Linux wheel in thedatabrickspackage on a Python package index.experimental/ssh/internal/client/pypi.go(new): finds, downloads and checks the wheel, then builds the release archive from it. The binary keeps its compressed form in the new archive, so it isn't decompressed and compressed again.experimental/ssh/internal/client/releases.go:withPyPIFallbackwraps the GitHub provider. If both sources fail, the error includes both causes and lists the hosts to allow (GitHub or PyPI) or the setting to change (PIP_INDEX_URLorindex-url).libs/localenv/uv.go: exportspipConfIndexURLasPipConfIndexURL, so this code reads the per-user pip config the same way the uv bootstrap does. Only the name changes. System-wide pip config files are still not read.experimental/ssh/README.md: new "Network Requirements" section. A changelog fragment is under.nextchanges/cli/.How the wheel is verified:
#sha256=<hex>URL fragment. The downloaded bytes must match that digest. The comparison ignores case.#md5=, is refused with an error. Without a SHA-256 digest the wheel can't be verified.Accept: text/html.How credentials in the index URL are handled (for example
https://user:token@mirror.example/simple):files.pythonhosted.org, is fetched without credentials.Authorizationheader are removed. Go's default policy forwardsAuthorizationto subdomains, and this one doesn't.Refereris removed on every redirect, because the previous URL's query can carry a token.net/httpadds to its errors is removed, and a parse error doesn't echo the URL.Requests time out after 10 minutes. At most 16 MiB is read for the project page and 256 MiB for a wheel.
Why
Some networks block GitHub but allow PyPI, or send pip to a PyPI mirror through
PIP_INDEX_URLorpip.conf. On those networksssh connectfailed before it could start the server, even though the same binary is published to PyPI in every CLI release.Tests
pypi_test.gorunsgetPyPIReleaseagainst an in-process fake index. It covers:TestCheckPyPIRedirectchecks the redirect policy for the same origin, another host, a subdomain, another port, a target URL that carries credentials, HTTPS to HTTP on the same host and on another host, and the redirect limit. It also checks that a redirect from HTTPS to HTTP is followed when the index URL has no credentials.TestPyPIIndexURLchecks the orderPIP_INDEX_URL, then pip config, then PyPI.TestIsLinuxWheelchecks wheel file names with compressed platform tag sets for both machines.releases_test.go:TestWithPyPIFallbackcovers GitHub working, GitHub failing, both failing (both errors are wrapped), and a development version (no fallback).TestWithPyPIFallbackNamesWhatToAllowchecks the remediation text.TestUploadReleasesFromPyPIuploads the archives built from the wheels to the fake workspace server and reads back the same binaries for amd64 and arm64.go test ./experimental/ssh/internal/client/ ./libs/localenv/passes.