Skip to content

ssh: download the server binary from a Python package index when GitHub is unreachable - #7852

Open
anton-107 wants to merge 2 commits into
mainfrom
ssh-server-binary-pypi-fallback
Open

anton-107 wants to merge 2 commits into
mainfrom
ssh-server-binary-pypi-fallback

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

databricks ssh connect uploads the Linux build of the CLI, at the client's version, to the workspace as the SSH server. Until now it could only download that build from the GitHub release. With this change, when the GitHub download fails, it takes the binary from the CLI's Linux wheel in the databricks package on a Python package index.

UploadTunnelReleases
  --releases-dir set   -> local archives (unchanged)
  otherwise            -> withPyPIFallback(getGithubRelease, getPyPIRelease)
    GitHub OK                  -> GitHub archive
    GitHub fails, dev version  -> GitHub error (dev builds are not published)
    GitHub fails               -> getPyPIRelease
      index URL: PIP_INDEX_URL > index-url in per-user pip config > https://pypi.org/simple
      GET <index>/databricks/ (PEP 503 HTML page)
      pick databricks-<version>-py3-none-<tags>.whl whose tags include a manylinux/musllinux tag for the machine
      download it and check its SHA-256
      repack <name>.data/scripts/databricks as `databricks` at the root of a zip, the GitHub archive layout
  • experimental/ssh/internal/client/pypi.go (new): finds, downloads and checks the wheel, then builds the release archive from it. The binary keeps its compressed form in the new archive, so it isn't decompressed and compressed again.
  • experimental/ssh/internal/client/releases.go: withPyPIFallback wraps the GitHub provider. If both sources fail, the error includes both causes and lists the hosts to allow (GitHub or PyPI) or the setting to change (PIP_INDEX_URL or index-url).
  • libs/localenv/uv.go: exports pipConfIndexURL as PipConfIndexURL, so this code reads the per-user pip config the same way the uv bootstrap does. Only the name changes. System-wide pip config files are still not read.
  • experimental/ssh/README.md: new "Network Requirements" section. A changelog fragment is under .nextchanges/cli/.

How the wheel is verified:

  • A wheel is accepted only if its link on the project page has a #sha256=<hex> URL fragment. The downloaded bytes must match that digest. The comparison ignores case.
  • A link with no fragment, or with only another hash such as #md5=, is refused with an error. Without a SHA-256 digest the wheel can't be verified.
  • No other source of hashes is read. That includes the PEP 691 JSON API, which is why the request sends Accept: text/html.

How credentials in the index URL are handled (for example https://user:token@mirror.example/simple):

  • When a wheel link has no userinfo of its own, the index credentials are added only if the link has the index URL's scheme and host. Relative links get them anyway, because they resolve against the index page. A wheel on another host, such as files.pythonhosted.org, is fetched without credentials.
  • A custom redirect policy compares each redirect target with the first URL of its request chain:
    • Same scheme and host: the userinfo is kept, and added back if an absolute redirect left it out.
    • Any other scheme or host, including a subdomain or another port: the userinfo and the Authorization header are removed. Go's default policy forwards Authorization to subdomains, and this one doesn't.
    • If the first URL has credentials, a redirect from HTTPS to plain HTTP is refused, even on the same host.
    • Referer is removed on every redirect, because the previous URL's query can carry a token.
    • The 10-redirect limit of Go's default policy still applies.
  • Error messages name only the host. The URL that net/http adds to its errors is removed, and a parse error doesn't echo the URL.

Requests time out after 10 minutes. At most 16 MiB is read for the project page and 256 MiB for a wheel.

Why

Some networks block GitHub but allow PyPI, or send pip to a PyPI mirror through PIP_INDEX_URL or pip.conf. On those networks ssh connect failed before it could start the server, even though the same binary is published to PyPI in every CLI release.

Tests

  • pypi_test.go runs getPyPIRelease against an in-process fake index. It covers:
    • absolute links, relative links, and a project page that redirects
    • credentials sent to the index host for absolute and relative links
    • credentials kept on an absolute redirect back to the index host
    • credentials kept off a separate files host and off a redirect to another host
    • an upper-case digest
    • errors: version not listed, missing project page, digest mismatch, no digest, only an MD5 digest, wheel without the binary, and a file that isn't a wheel
    • rejected credentials, an unreachable index and a download that gets cut off, each with an index URL that carries credentials. The test checks that none of these errors contains the username or password.
  • TestCheckPyPIRedirect checks the redirect policy for the same origin, another host, a subdomain, another port, a target URL that carries credentials, HTTPS to HTTP on the same host and on another host, and the redirect limit. It also checks that a redirect from HTTPS to HTTP is followed when the index URL has no credentials.
  • TestPyPIIndexURL checks the order PIP_INDEX_URL, then pip config, then PyPI. TestIsLinuxWheel checks wheel file names with compressed platform tag sets for both machines.
  • releases_test.go:
    • TestWithPyPIFallback covers GitHub working, GitHub failing, both failing (both errors are wrapped), and a development version (no fallback).
    • TestWithPyPIFallbackNamesWhatToAllow checks the remediation text.
    • TestUploadReleasesFromPyPI uploads the archives built from the wheels to the fake workspace server and reads back the same binaries for amd64 and arm64.
  • go test ./experimental/ssh/internal/client/ ./libs/localenv/ passes.

…ub is unreachable

## Changes

`databricks ssh connect` uploads the Linux build of the CLI, at the client's version, to the workspace as the SSH server. Until now it could only download that build from the GitHub release. With this change, when the GitHub download fails, it takes the binary from the CLI's Linux wheel in the `databricks` package on a Python package index.

```text
UploadTunnelReleases
  --releases-dir set   -> local archives (unchanged)
  otherwise            -> withPyPIFallback(getGithubRelease, getPyPIRelease)
    GitHub OK                  -> GitHub archive
    GitHub fails, dev version  -> GitHub error (dev builds are not published)
    GitHub fails               -> getPyPIRelease
      index URL: PIP_INDEX_URL > index-url in per-user pip config > https://pypi.org/simple
      GET <index>/databricks/ (PEP 503 HTML page)
      pick databricks-<version>-py3-none-<tags>.whl whose tags include a manylinux/musllinux tag for the machine
      download it and check its SHA-256
      repack <name>.data/scripts/databricks as `databricks` at the root of a zip, the GitHub archive layout
```

- `experimental/ssh/internal/client/pypi.go` (new): finds, downloads and checks the wheel, then builds the release archive from it. The binary keeps its compressed form in the new archive, so it isn't decompressed and compressed again.
- `experimental/ssh/internal/client/releases.go`: `withPyPIFallback` wraps the GitHub provider. If both sources fail, the error includes both causes and lists the hosts to allow (GitHub or PyPI) or the setting to change (`PIP_INDEX_URL` or `index-url`).
- `libs/localenv/uv.go`: exports `pipConfIndexURL` as `PipConfIndexURL`, so this code reads the per-user pip config the same way the uv bootstrap does. Only the name changes. System-wide pip config files are still not read.
- `experimental/ssh/README.md`: new "Network Requirements" section. A changelog fragment is under `.nextchanges/cli/`.

How the wheel is verified:
- A wheel is accepted only if its link on the project page has a `#sha256=<hex>` URL fragment. The downloaded bytes must match that digest. The comparison ignores case.
- A link with no fragment, or with only another hash such as `#md5=`, is refused with an error. Without a SHA-256 digest the wheel can't be verified.
- No other source of hashes is read. That includes the PEP 691 JSON API, which is why the request sends `Accept: text/html`.

How credentials in the index URL are handled (for example `https://user:token@mirror.example/simple`):
- When a wheel link has no userinfo of its own, the index credentials are added only if the link has the index URL's scheme and host. Relative links get them anyway, because they resolve against the index page. A wheel on another host, such as `files.pythonhosted.org`, is fetched without credentials.
- A custom redirect policy compares each redirect target with the first URL of its request chain:
  - Same scheme and host: the userinfo is kept, and added back if an absolute redirect left it out.
  - Any other scheme or host, including a subdomain or another port: the userinfo and the `Authorization` header are removed. Go's default policy forwards `Authorization` to subdomains, and this one doesn't.
  - If the first URL has credentials, a redirect from HTTPS to plain HTTP is refused, even on the same host.
  - `Referer` is removed on every redirect, because the previous URL's query can carry a token.
  - The 10-redirect limit of Go's default policy still applies.
- Error messages name only the host. The URL that `net/http` adds to its errors is removed, and a parse error doesn't echo the URL.

Requests time out after 10 minutes. At most 16 MiB is read for the project page and 256 MiB for a wheel.

## Why

Some networks block GitHub but allow PyPI, or send pip to a PyPI mirror through `PIP_INDEX_URL` or `pip.conf`. On those networks `ssh connect` failed before it could start the server, even though the same binary is published to PyPI in every CLI release.

## Tests

- `pypi_test.go` runs `getPyPIRelease` against an in-process fake index. It covers:
  - absolute links, relative links, and a project page that redirects
  - credentials sent to the index host for absolute and relative links
  - credentials kept on an absolute redirect back to the index host
  - credentials kept off a separate files host and off a redirect to another host
  - an upper-case digest
  - errors: version not listed, missing project page, digest mismatch, no digest, only an MD5 digest, wheel without the binary, and a file that isn't a wheel
  - rejected credentials, an unreachable index and a download that gets cut off, each with an index URL that carries credentials. The test checks that none of these errors contains the username or password.
- `TestCheckPyPIRedirect` checks the redirect policy for the same origin, another host, a subdomain, another port, a target URL that carries credentials, HTTPS to HTTP on the same host and on another host, and the redirect limit. It also checks that a redirect from HTTPS to HTTP is followed when the index URL has no credentials.
- `TestPyPIIndexURL` checks the order `PIP_INDEX_URL`, then pip config, then PyPI. `TestIsLinuxWheel` checks wheel file names with compressed platform tag sets for both machines.
- `releases_test.go`:
  - `TestWithPyPIFallback` covers GitHub working, GitHub failing, both failing (both errors are wrapped), and a development version (no fallback).
  - `TestWithPyPIFallbackNamesWhatToAllow` checks the remediation text.
  - `TestUploadReleasesFromPyPI` uploads the archives built from the wheels to the fake workspace server and reads back the same binaries for amd64 and arm64.
- `go test ./experimental/ssh/internal/client/ ./libs/localenv/` passes.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: c71b79c

Run: 44786029315

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 5:05
💚​ aws-windows-latest-4core-16gb 1 274 17 4:12
💚​ azure linux-2core-8gb 1 271 19 5:27
💚​ azure-windows-latest-4core-16gb 1 273 17 4:21
💚​ gcp linux-2core-8gb 1 272 19 5:04
💚​ gcp-windows-latest-4core-16gb 1 274 17 4:11
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:20 azure-windows-latest-4core-16gb TestAccept
4:09 gcp-windows-latest-4core-16gb TestAccept
3:42 aws-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants