Skip to content

Add authenticated AIR feedback command - #7254

Closed
delbalso wants to merge 2 commits into
mainfrom
codex/air-agent-feedback
Closed

delbalso wants to merge 2 commits into
mainfrom
codex/air-agent-feedback

Conversation

@delbalso

@delbalso delbalso commented Oct 29, 2026 •

Copy link
Copy Markdown

Changes

Add databricks air feedback with workspace authentication, human/agent attribution, optional run and agent context, contact opt-in, and a JSON receipt. Include command help, acceptance coverage, and a changelog entry.

Why

Let users and coding agents report actionable AIR product friction through their existing workspace login, with enough context for product and engineering teams to investigate.

Tests

The source is unchanged at 6af851a. GitHub CI passed Linux, macOS, and Windows tests in both deployment modes, lint, generated-file validation, and regression reporting. External integration tests passed across AWS, Azure, and GCP on Linux and Windows. The regression report confirms the new feedback/help tests pass on this branch and fail on main/latest, where the command is absent.

Earlier local unit suites passed 12,519 root tests (132 skipped) and 73 tools tests. Focused feedback/help/config-help acceptance tests passed. The broader local acceptance run encountered unrelated AppKit/bundle timeouts; no clean full local acceptance pass is claimed.

Two real staging submissions were traced from CLI receipts to raw text and existing fields. New typed metadata and the full Jira/Insights path remain pending deployment verification.

This PR was written with Codex.

Related PRs and rollout

Component PR Release gate
Schema #2915550 Lumberjack review; deploy to both workspace telemetry handlers and Lumberjack; verify the raw metadata.
Collector / Jira sync #49 After schema deployment, apply the migration and update both jobs.
Ingestion / dashboards #2915780 Fast Feedback/code-owner approval; deploy and verify the full backend path.
CLI #7254 Hold merge until schema, collector, and ingestion are deployed and verified; then publish a CLI release.
Agent skill #2945816 Deco/AIR review; merge and publish after the CLI release.
Public docs #2945815 Docs review/self-review; merge after CLI release and skill publication because docs publish from master automatically.

Employee dashboard: go/air-cli-feedback.
Deployment steps, request-protection evidence, and verification handoff.
Implementation and review readiness do not imply deployment or owner approval.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 6af851af8 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0

test branch main (aed6702) latest (v1.23.0)
TestAccept/air/feedback ✅ ❌ ❌
TestAccept/air/help ✅ ❌ ❌
TestAccept/air/feedback: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- air/feedback/output.txt
        +++ /tmp/TestAcceptairfeedback849732613/002/output.txt
        @@ -1,28 +1,31 @@
         
         === agent report with a receipt
         >>> [CLI] air feedback --message Logs omitted the failure reason --reporter agent --agent-name codex --run-id 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "received": true,
        -    "feedback_id": "[UUID]"
        -  }
        -}
        +Error: unknown flag: --message
         
        -=== human report
        ->>> [CLI] air feedback --message The workaround succeeded
        -Feedback received. Reference: [UUID]
        +Usage:
        +  databricks air [command]
         
        -=== validation errors
        ->>> musterr [CLI] air feedback --message     -o json
        -true
        +Available Commands:
        +  cancel          Cancel one or more runs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
        ->>> musterr [CLI] air feedback --message Logs omitted the failure reason --reporter robot -o json
        -true
        +Flags:
        +  -h, --help   help for air
         
        ->>> musterr [CLI] air feedback --message Logs omitted the failure reason --agent-name codex -o json
        -true
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> musterr [CLI] air feedback unexpected --message Logs omitted the failure reason -o json
        -true
        +Use "databricks air [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

TestAccept/air/help: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp1530177345/002/output.txt
        @@ -11,7 +11,6 @@
         Available Commands:
           cancel          Cancel one or more runs
           convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        -  feedback        Send feedback about AI Runtime to Databricks
           get             Show status, configuration, and timing details for a specified run
           images          Manage container images for AI Runtime
           list            List active runs for the current user
        @@ -214,36 +213,30 @@
         
         === feedback help
         >>> [CLI] air feedback --help
        -Send feedback about AI Runtime using your workspace authentication.
        +Run and manage AI Runtime training workloads on Databricks serverless GPU compute.
         
        -Include the command, observed problem, expected behavior, and any workaround
        -in --message. Use --reporter agent for reports written by an agent.
        +These commands are under active development.
         
        -Sends your message, CLI version, reporter type, and optional run ID and agent
        -name to Fast Feedback. The service associates it with your workspace and
        -authenticated principal. No transcripts, source files, or logs are collected.
        -Do not include credentials or sensitive data. Follow-up contact is opt-in.
        -
        -This is an explicit submission, independent of automatic usage telemetry.
        -Success confirms receipt by the collector; processing into review tools is
        -asynchronous.
        -
         Usage:
        -  databricks air feedback [flags]
        +  databricks air [command]
         
        -Examples:
        -  databricks air feedback --message "air logs omitted the failure reason" --reporter agent --agent-name codex --run-id 123 -o json
        +Available Commands:
        +  cancel          Cancel one or more runs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
         Flags:
        -      --agent-name string   Agent name; requires --reporter agent
        -      --can-be-contacted    Allow follow-up about this feedback
        -  -h, --help                help for feedback
        -      --message string      Feedback message (1–10,000 characters)
        -      --reporter string     Who wrote the feedback: human or agent (default "human")
        -      --run-id string       Related job run ID
        +  -h, --help   help for air
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks air [command] --help" for more information about a command.
        
FAIL

latest (v1.23.0): same output as main (aed6702)

@delbalso
delbalso force-pushed the codex/air-agent-feedback branch from b40f37c to 6af851a Compare October 29, 2026 17:01
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6af851a

Run: 40917487010

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 17 5:09
✅​ aws-windows-latest-4core-16gb 283 15 6:24
✅​ azure linux-2core-8gb 280 17 5:20
✅​ azure-windows-latest-4core-16gb 282 15 5:51
✅​ gcp linux-2core-8gb 281 17 5:28
✅​ gcp-windows-latest-4core-16gb 283 15 4:45
Top 6 slowest tests (at least 2 minutes):
duration env testname
6:16 aws-windows-latest-4core-16gb TestAccept
5:44 azure-windows-latest-4core-16gb TestAccept
4:40 gcp-windows-latest-4core-16gb TestAccept
3:39 azure linux-2core-8gb TestAccept
3:39 gcp linux-2core-8gb TestAccept
3:28 aws linux-2core-8gb TestAccept

@delbalso
delbalso marked this pull request as ready for review October 29, 2026 17:42
@delbalso
delbalso requested review from a team as code owners October 29, 2026 17:42
@delbalso
delbalso requested a review from rclarey October 29, 2026 17:42
@maggiewang-db

Copy link
Copy Markdown
Contributor

This PR depends on backend changes that need to be merged and released before it merges:

  1. https://github.com/databricks-eng/universe/pull/2915550: new FastFeedbackLog fields. Deploy it to the workspace telemetry handler and Lumberjack, and confirm the new fields appear in the staging feedback logs.
  2. https://github.com/databricks-eng/fast-feedback-notebooks/pull/49: collector migration and Jira sync. Run the migration, then deploy the reader and sync.
  3. https://github.com/databricks-eng/universe/pull/2915780: ingestion and dashboard changes. The agent skill and public docs in that PR should land only after a CLI release that includes this command, so it may be worth splitting.

Please hold this PR until those are live.

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the transient authentication failure handling noted inline. The focused AIR feedback unit and local acceptance tests passed for this head.

Two non-blocking questions:

  • Could submitFeedback share the one-shot sender in libs/telemetry/logger.go:200, while keeping explicit submission semantics and its acknowledgement check? Both implementations construct the same request, routing header, timestamp, and empty items workaround. If the sender stays separate, please document why the API requires an empty items array.
  • Could validation reuse the existing AIR INVALID_ARGS/PERMANENT pair? This command introduces INVALID_ARGUMENT/VALIDATION into the common JSON envelope.

This review was generated with GitHub MCP.

Comment thread cmd/air/feedback.go
if err == nil || errors.Is(err, root.ErrAlreadyPrinted) {
return err
}
return renderError(cmd.Context(), cmd, "AUTHENTICATION_ERROR", "AUTHENTICATION", false, err)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2][blocking] root.MustWorkspaceClient can fail on a temporary OAuth connection error before /telemetry-ext is called. This branch marks every such failure retryable: false, so an agent following the JSON response abandons a report that is safe to retry.

I reproduced this with a local OAuth endpoint that closes the token connection: air feedback returned AUTHENTICATION_ERROR/AUTHENTICATION with retryable: false and zero feedback submissions. air get reported the same failure as INTERNAL_ERROR/TRANSIENT with retryable: true.

Please route this branch through the existing authError helper and add a regression case for a transient authentication failure.

# Explicit feedback still submits when automatic usage telemetry is disabled.
Env.DATABRICKS_CLI_DISABLE_TELEMETRY = "1"

[[Repls]]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The built-in UUID replacement runs before config.Repls is appended, so this rule never sees an original UUID and cannot produce [FEEDBACK_ID]. The golden already uses [UUID]; please remove this unused replacement block.

Comment thread cmd/air/feedback.go
if o.message == "" || !utf8.ValidString(o.message) || utf8.RuneCountInString(o.message) > feedbackMessageLimit {
return fmt.Errorf("--message must contain 1 to %d UTF-8 characters", feedbackMessageLimit)
}
if o.reporter != "human" && o.reporter != "agent" {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Reporter values and error code/kind pairs are repeated across validation, flag defaults, and rendering. .agents/rules/style-guide-go.md:40 requires named constants for repeated strings; please extract the shared values.

Comment thread cmd/air/feedback.go
Product: "PRODUCT_AIR",
FeedbackSource: "SOURCE_AIR_CLI",
ReporterType: "REPORTER_" + strings.ToUpper(o.reporter),
CLIVersion: build.GetInfo().Version,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test this? the schema doesn't look aligned to the proto fast_feedback_log event. For example it has no CLIVersion field.

Typically I see this info get inlined into a synthetic user agent.

Comment thread cmd/air/feedback.go
if err == nil || errors.Is(err, root.ErrAlreadyPrinted) {
return err
}
return renderError(cmd.Context(), cmd, "AUTHENTICATION_ERROR", "AUTHENTICATION", false, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every MustWorkspaceClient failure is reported here as AUTHENTICATION_ERROR with retryable: false, including a transient one such as a dropped OAuth token request, where nothing has been sent yet and a retry is safe. air get, logs and pools report that same failure as INTERNAL_ERROR/TRANSIENT with retryable: true, so please route this through authError like they do and add a test for a non-auth failure on this branch.

Comment thread cmd/air/feedback.go
// errors can be rendered, while still rejecting input before authentication.
cmd.PreRunE = func(cmd *cobra.Command, args []string) error {
if err := cobra.NoArgs(cmd, args); err != nil {
return renderError(cmd.Context(), cmd, "INVALID_ARGUMENT", "VALIDATION", false, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bad input is reported as INVALID_ARGUMENT/VALIDATION here, but every other air command uses INVALID_ARGS/PERMANENT for it, so a caller that keys on the envelope's code or kind across subcommands won't recognise these. I'd reuse the existing pair before this ships, since the codes are hard to change once agents depend on them.

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocker inline (feedback.go:66), plus @simonfaltum's open thread on feedback.go:130. Two non-blocking follow-ups inline.

Comment thread cmd/air/feedback.go

// These bindings mirror FastFeedbackLog in universe. Feedback text is sent only
// through that explicitly submitted feedback schema, never a CLI usage event.
type fastFeedbackLog struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the existing libs/telemetry/protos types instead of a second copy of FrontendLog here:

  • Add FastFeedbackLog to protos.FrontendLogEntry.
  • Use typed constants for product, source, and reporter, like protos.AirPackagingMode. That replaces the string concatenation on line 166.
  • Send it through a one-shot helper exported from libs/telemetry that wraps attempt. That also keeps the comment on why Items must be empty.

Comment thread cmd/air/feedback.go
Do not include credentials or sensitive data. Follow-up contact is opt-in.

This is an explicit submission, independent of automatic usage telemetry.
Success confirms receipt by the collector; processing into review tools is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: a few phrases in this help text may be hard for CLI users to follow. Suggested wording:

  • Line 103: "to Fast Feedback" could be "to Databricks", unless users already know the Fast Feedback name.
  • Lines 108-109: "Success confirms receipt by the collector; processing into review tools is asynchronous." could be "A success message means Databricks received your feedback."

trace $CLI air feedback --message "The workaround succeeded"

title "validation errors"
trace musterr $CLI air feedback --message " " -o json | jq -e '.error.code == "INVALID_ARGUMENT"'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: with the jq -e pipe, output.txt records only true, so the error message is never checked. acceptance/air/get/script uses errcode trace $CLI air get ... -o json. The same pattern here would record the full error for all four cases.

@simonfaltum

Copy link
Copy Markdown
Member

This PR has been merged FYI #7724 which adds a databricks feedback command.

@pardis-beikzadeh-db

Copy link
Copy Markdown
Contributor

As Simon brought to my attention, #7724 does the same thing. Can we close this one?

@delbalso

Copy link
Copy Markdown
Author

Superseded by #7724, which implements the shared databricks feedback command with AIR routing on the existing Fast Feedback schema. Closing this duplicate AIR-specific implementation; its history remains available for context. The dashboard and agent/docs guidance are being adapted to the shared command. Direct agent submission without confirmation is tracked separately in #7795.


This comment was generated with GitHub MCP.

@delbalso delbalso closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants