Skip to content

Record the real error in config-remote-sync telemetry - #7720

Open
ilyakuz-db wants to merge 14 commits into
mainfrom
ilyakuz-db/configsync-telemetry-real-error
Open

ilyakuz-db wants to merge 14 commits into
mainfrom
ilyakuz-db/configsync-telemetry-real-error

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Changes

bundle config-remote-sync telemetry now records the first diagnostic error instead of the AlreadyPrinted sentinel. When loading the bundle or its target fails before bundle auth is set up, the event is still sent if a profile was selected explicitly (--profile or DATABRICKS_CONFIG_PROFILE).

Why

Errors already printed to stderr reached telemetry only as the sentinel, and failures before auth (for example a target that assigns an undefined variable) sent no event at all, so many sync failures had no usable message. The fallback uses only the selected profile, so hosts from the bundle config are never contacted. It reads the profile from the config file and leaves the config for the telemetry upload to resolve within its own timeout, with the profile winning over auth env vars and no OAuth token refresh. No profile, an unknown profile, a profile for another host or workspace than the bundle's or the selected target's, a bundle auth error, or disabled telemetry sends nothing. The JSON output keeps the sentinel error, so callers that fall back to stderr are unchanged.

Tests

New acceptance test bundle/telemetry/config-remote-sync-error-message checks the real message after an already printed error, a target-load failure (with -p, with DATABRICKS_CONFIG_PROFILE, with a matching bundle or target host or workspace, with DATABRICKS_HOST set to another host, and for the default target) and a missing bundle with -p, and that nothing is sent, and no request made, without a profile, with an unknown profile, with a profile for another host than the bundle's or the target's or for another workspace on the bundle's host, or on a bundle auth error. With a profile host that never answers, the failing run takes 3s (the upload's own timeout) and returns at once with DATABRICKS_CLI_DISABLE_TELEMETRY=1. The config-remote-sync and telemetry acceptance tests, the configsync/cmd/bundle unit tests and lint pass locally.

Status Check Revision Comments
❌ Local review 4f09cc0 (latest) Codex and auto review, 3 rounds: P1s fixed (profile over DATABRICKS_HOST, a profile for another workspace on the same host); still open: the bundle's workspace_id is not used when the profile has none, the profile file is read outside the upload timeout, a profile with an empty host
⚠️ Design review 4f09cc0 (latest) SIMPLER PATH: drop the bundle host and workspace check, since the profile-only config already keeps telemetry off bundle hosts
➖ CI code review (Isaac) 4f09cc0 (latest) n/a on CLI PRs
✅ Code owners review 4f09cc0 (latest) Sankalp-Mittal approved
⚠️ E2E validation 4f09cc0 (latest) not run on a workspace; acceptance tests run locally
⏳ CI / merge conflicts 4f09cc0 (latest) checks running; no conflicts

✅ success · ⏳ pending · ⚠️ warning · ❌ failed · ➖ n/a

This pull request and its description were written by Isaac.

ilyakuz-db and others added 8 commits November 16, 2026 17:30
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…host

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 17, 2026 11:32
@ilyakuz-db ilyakuz-db added DABs DABs related issues software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 17, 2026
@ilyakuz-db
ilyakuz-db requested a review from denik November 17, 2026 11:32
@ilyakuz-db ilyakuz-db self-assigned this Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 4f09cc008 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (8b7e509) latest (v1.27.0)
TestAccept/bundle/telemetry/config-remote-sync-error-message/DMS= ✅ ❌ ❌
TestAccept/bundle/telemetry/config-remote-sync-error-message/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌

main (8b7e509):

acceptance_test.go:1402: Diff:
        --- bundle/telemetry/config-remote-sync-error-message/output.txt
        +++ /tmp/TestAcceptbundletelemetryconfig-remote-sync-error-messageDMS1666917001/002/output.txt
        @@ -5,7 +5,7 @@
         
         >>> print_requests.py //telemetry-ext
         {
        -  "error_message": "no value assigned to required variable myvar. Variables are usually assigned in databricks.yml, and they can be overridden using \"--var\", the BUNDLE_VAR_myvar environment variable, or [REDACTED_REL_PATH](json)",
        +  "error_message": "AlreadyPrinted",
           "error_category": "BUNDLE_LOAD_FAILED"
         }
         
        @@ -14,102 +14,6 @@
         
         
         >>> print_requests.py //telemetry-ext
        -{
        -  "error_message": "failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad -p myprofile
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad -p myprofile
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t own-host -p myprofile
        -Error: failed to perform target override for target=own-host: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t profile-workspace -p workspace2222
        -Error: failed to perform target override for target=profile-workspace: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -p myprofile
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> print_requests.py //telemetry-ext
        -{
        -  "error_message": "failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -{
        -  "error_message": "failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -{
        -  "error_message": "failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -{
        -  "error_message": "failed to perform target override for target=own-host: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -{
        -  "error_message": "failed to perform target override for target=profile-workspace: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -{
        -  "error_message": "failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -
        ->>> musterr [CLI] bundle config-remote-sync -p myprofile
        -Error: Invalid bundle root DATABRICKS_BUNDLE_ROOT="missing": stat missing: no such file or directory
        -
        -The DATABRICKS_BUNDLE_ROOT environment variable must point to an existing directory that contains a bundle configuration file.
        -
        -
        ->>> print_requests.py //telemetry-ext
        -{
        -  "error_message": "Invalid bundle root DATABRICKS_BUNDLE_ROOT=\"missing\": stat missing: no such file or directory",
        -  "error_category": "BUNDLE_LOAD_FAILED"
        -}
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad -p missing
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad -p myprofile
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t bad -p workspace2222
        -Error: failed to perform target override for target=bad: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t other -p myprofile
        -Error: failed to perform target override for target=other: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -p myprofile
        -Error: failed to perform target override for target=other: variable undefined_var is not defined but is assigned a value
        -
        -
        ->>> musterr [CLI] bundle config-remote-sync -t auth-error -p myprofile
        -Error: cannot resolve bundle auth configuration: validate: more than one authorization method configured: oauth and pat. Config: host=[DATABRICKS_URL], token=***, profile=myprofile, config_file=[TEST_TMP_DIR]/.databrickscfg, client_id=x, databricks_cli_path=[CLI]. Env: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_CONFIG_FILE, DATABRICKS_CLI_PATH
        -
        -
        ->>> errcode print_requests.py //telemetry-ext
         File [OUTPUT_DIR]/out.requests.txt not found
         
         Exit code: 1
        
FAIL

latest (v1.27.0): same output as main (8b7e509)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 4f09cc0

Run: 44764650527

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 6:43
💚​ aws-windows-latest-4core-16gb 1 274 18 5:16
💚​ azure linux-2core-8gb 1 271 20 5:25
💚​ azure-windows-latest-4core-16gb 1 273 18 4:56
💚​ gcp linux-2core-8gb 1 272 20 6:23
💚​ gcp-windows-latest-4core-16gb 1 274 18 5:27
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
4:14 azure-windows-latest-4core-16gb TestAccept
4:03 aws-windows-latest-4core-16gb TestAccept
3:12 gcp-windows-latest-4core-16gb TestAccept
2:00 gcp linux-2core-8gb TestAccept

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Comment thread bundle/configsync/telemetry.go Resolved
Comment thread bundle/configsync/telemetry.go Resolved
Comment thread bundle/configsync/telemetry.go Resolved
ilyakuz-db and others added 3 commits November 20, 2026 11:53
…heck the target's host

Co-authored-by: Isaac <no-reply@databricks.com>
…st, and never refresh tokens for it

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added factory-generated and removed software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No blocking or major issues found.

Design, could be simpler: resolveProfile resolves the profile in a goroutine under its own 3s budget, but telemetry.Upload already resolves the config it is given within its timeout (libs/telemetry/logger.go:152). Put the unresolved Workspace{Profile}.Config(ctx) on the context, as cmd/root/auth.go:404 does, and read the profile's host with profile.WithName for the comparison. resolveProfile, profileAuthBudget and the abandoned goroutine go away.

@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

Agree, profile management is too complex

Factory, let's simplify as suggested; make sure that telemetry in config sync is fail-open.

ilyakuz-db and others added 3 commits November 20, 2026 14:52
…ofile's host from the config file

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… bundle's host

Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

[AI] Simplified as suggested in 6277854: resolveProfile, profileAuthBudget and the goroutine are gone. The fallback reads the profile with profile.WithName and puts the unresolved Workspace{Profile} config on the context, so the upload resolves it within its own timeout. It stays fail-open: a profile host that never answers still costs only the upload's 3s, and a panic in the fallback is recovered. Review follow-ups: 7eb3860 lets the profile win over DATABRICKS_HOST, as bundle auth does for an explicit profile, and 4f09cc0 skips a profile for another workspace on the bundle's host. A few edge cases are still open and wait on a decision (Local review row in the description).

🤖 Factory-generated comment. I still review the code and double-check every claim myself.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

Earlier findings: 4 fixed.

No new issues found.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants