Skip to content

Report config-remote-sync selector misses under their own telemetry category - #7693

Open
ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/configsync-no-matching-selector-telemetry
Open

ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/configsync-no-matching-selector-telemetry

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

bundle config-remote-sync --select-ids now records a new NO_MATCHING_SELECTOR telemetry error category when none of the selected resources is in the deployed state. The JSON output carries the same errorCode next to status: skipped.

Why

Those runs are reported as skipped, not failed, but telemetry had no category for them, so they fell back to BUNDLE_LOAD_FAILED and counted toward the sync failure rate. With their own category, they can be left out of it.

Tests

New acceptance test selects a job id that isn't deployed and checks the JSON output and the telemetry payload; updated the json_status_selectors golden. The config-remote-sync acceptance tests, the configsync/cmd/bundle unit tests and lint pass locally.

Status Check Revision Comments
⚠️ Local review 25ca011 Codex: one P1 deferred (stale selectors within an otherwise matching batch, out of scope here)
➖ CI code review (Isaac) d742b69 (latest) n/a on CLI PRs
⏳ Code owners review d742b69 (latest) Sankalp-Mittal commented on d742b69 (no issues found), no approval yet
⚠️ E2E validation beaf1d0 not run on a workspace; acceptance tests run locally
✅ CI / merge conflicts d742b69 (latest) 18 pass, 9 skipping; no conflicts

✅ success · ⏳ pending · ⚠️ warning · ❌ failed · ➖ n/a

This pull request and its description were written by Isaac.

ilyakuz-db and others added 2 commits November 16, 2026 16:01
…ategory

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 16, 2026 16:11
@ilyakuz-db ilyakuz-db added DABs DABs related issues software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 16, 2026
@ilyakuz-db ilyakuz-db self-assigned this Nov 16, 2026
@ilyakuz-db ilyakuz-db added DABs DABs related issues software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Commit: d742b6989 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

3 tests: regression=3 unreleased=0 coverage=0 failing=0
test branch main (1511899) latest (v1.27.0)
TestAccept/bundle/config-remote-sync/json_status_selectors/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true ✅ ❌ ❌
TestAccept/bundle/telemetry/config-remote-sync-no-matching-selector/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=: branch ✅ | main (1511899) ❌ | latest (v1.27.0) ❌

main (1511899):

acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:704
        	Error:      	Not equal: 
        	            	expected: "Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...\nCreated jobs.test_job\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n=== A selector that matches nothing is reported as skipped, exit 0\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]\",\n  \"errorCode\": \"NO_MATCHING_SELECTOR\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n=== A malformed selector is reported as failed, exit 0\n{\n  \"status\": \"failed\",\n  \"error\": \"invalid --select-ids value \\\"no-such-id-999\\\", expected \\u003ctype\\u003e:\\u003cid\\u003e (e.g. jobs:123456789)\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> [CLI] bundle destroy --auto-approve\nThe following resources will be deleted:\n  delete resources.jobs.test_job\n\nAll files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default\n\nDestroy: 1 deleted\n"
        	            	actual  : "Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...\nCreated jobs.test_job\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n=== A selector that matches nothing is reported as skipped, exit 0\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n=== A malformed selector is reported as failed, exit 0\n{\n  \"status\": \"failed\",\n  \"error\": \"invalid --select-ids value \\\"no-such-id-999\\\", expected \\u003ctype\\u003e:\\u003cid\\u003e (e.g. jobs:123456789)\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> [CLI] bundle destroy --auto-approve\nThe following resources will be deleted:\n  delete resources.jobs.test_job\n\nAll files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default\n\nDestroy: 1 deleted\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	   "error": "no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]",
        	            	-  "errorCode": "NO_MATCHING_SELECTOR",
        	            	   "files": null,
        	Test:       	TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=
        	Messages:   	bundle/config-remote-sync/json_status_selectors/output.txt vs /tmp/TestAcceptbundleconfig-remote-syncjson_status_selectorsDMS=2663278179/002/output.txt

latest (v1.27.0): same output as main (1511899)

TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true: branch ✅ | main (1511899) ❌ | latest (v1.27.0) ❌

main (1511899):

acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:704
        	Error:      	Not equal: 
        	            	expected: "Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...\nCreated jobs.test_job\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n=== A selector that matches nothing is reported as skipped, exit 0\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]\",\n  \"errorCode\": \"NO_MATCHING_SELECTOR\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n=== A malformed selector is reported as failed, exit 0\n{\n  \"status\": \"failed\",\n  \"error\": \"invalid --select-ids value \\\"no-such-id-999\\\", expected \\u003ctype\\u003e:\\u003cid\\u003e (e.g. jobs:123456789)\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> [CLI] bundle destroy --auto-approve\nThe following resources will be deleted:\n  delete resources.jobs.test_job\n\nAll files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default\n\nDestroy: 1 deleted\n"
        	            	actual  : "Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...\nCreated jobs.test_job\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n=== A selector that matches nothing is reported as skipped, exit 0\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n=== A malformed selector is reported as failed, exit 0\n{\n  \"status\": \"failed\",\n  \"error\": \"invalid --select-ids value \\\"no-such-id-999\\\", expected \\u003ctype\\u003e:\\u003cid\\u003e (e.g. jobs:123456789)\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> [CLI] bundle destroy --auto-approve\nThe following resources will be deleted:\n  delete resources.jobs.test_job\n\nAll files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default\n\nDestroy: 1 deleted\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	   "error": "no deployed jobs resource with id no-such-id-999; deployed jobs ids in state: [TEST_JOB_ID]",
        	            	-  "errorCode": "NO_MATCHING_SELECTOR",
        	            	   "files": null,
        	Test:       	TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true
        	Messages:   	bundle/config-remote-sync/json_status_selectors/output.txt vs /tmp/TestAcceptbundleconfig-remote-syncjson_status_selectorsDMS=t2454170156/002/output.txt
FAIL

latest (v1.27.0): same output as main (1511899)

TestAccept/bundle/telemetry/config-remote-sync-no-matching-selector/DMS=: branch ✅ | main (1511899) ❌ | latest (v1.27.0) ❌

main (1511899):

acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:704
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/config-remote-sync-no-matching-selector/default/files...\nCreated jobs.foo\nFiles: 1 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n>>> [CLI] bundle config-remote-sync --select-ids jobs:999999 -o json\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]\",\n  \"errorCode\": \"NO_MATCHING_SELECTOR\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> cat [OUTPUT_DIR]/out.requests.txt\n{\n  \"error_message\": \"no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]\",\n  \"error_category\": \"NO_MATCHING_SELECTOR\"\n}\n"
        	            	actual  : "\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/config-remote-sync-no-matching-selector/default/files...\nCreated jobs.foo\nFiles: 1 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\n>>> [CLI] bundle config-remote-sync --select-ids jobs:999999 -o json\n{\n  \"status\": \"skipped\",\n  \"error\": \"no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]\",\n  \"files\": null,\n  \"changes\": {}\n}\n\n>>> cat [OUTPUT_DIR]/out.requests.txt\n{\n  \"error_message\": \"no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]\",\n  \"error_category\": \"BUNDLE_LOAD_FAILED\"\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -11,3 +11,2 @@
        	            	   "error": "no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]",
        	            	-  "errorCode": "NO_MATCHING_SELECTOR",
        	            	   "files": null,
        	            	@@ -19,3 +18,3 @@
        	            	   "error_message": "no deployed jobs resource with id 999999; deployed jobs ids in state: [FOO_ID]",
        	            	-  "error_category": "NO_MATCHING_SELECTOR"
        	            	+  "error_category": "BUNDLE_LOAD_FAILED"
        	            	 }
        	Test:       	TestAccept/bundle/telemetry/config-remote-sync-no-matching-selector/DMS=
        	Messages:   	bundle/telemetry/config-remote-sync-no-matching-selector/output.txt vs /tmp/TestAcceptbundletelemetryconfig-remote-sync-no-matching-selec3554389877/002/output.txt

latest (v1.27.0): same output as main (1511899)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: d742b69

Run: 44772825920

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 275 20 7:54
💚​ aws-windows-latest-4core-16gb 1 274 19 7:48
💚​ azure linux-2core-8gb 1 274 20 6:21
💚​ azure-windows-latest-4core-16gb 1 273 19 4:06
💚​ gcp linux-2core-8gb 1 275 20 6:57
💚​ gcp-windows-latest-4core-16gb 1 274 19 5:00
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 5 slowest tests (at least 2 minutes):
duration env testname
3:25 aws-windows-latest-4core-16gb TestAccept
3:23 azure-windows-latest-4core-16gb TestAccept
3:19 gcp-windows-latest-4core-16gb TestAccept
2:41 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:29 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Comment thread cmd/bundle/config_remote_sync.go Resolved
Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Earlier findings: 1 fixed.

No new issues found.

@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

@Sankalp-Mittal can you stamp?

@denik denik added factory-generated and removed software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
@ilyakuz-db
ilyakuz-db enabled auto-merge November 20, 2026 14:32

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No issues found.

@ilyakuz-db
ilyakuz-db disabled auto-merge November 20, 2026 16:57
@ilyakuz-db
ilyakuz-db enabled auto-merge November 20, 2026 16:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants