Repository navigation
Conversation
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
8cfc4f1 to
c3634a9
Compare
Adds a CLI-wide feedback command. Feedback is sent as a fast_feedback_log event to /telemetry-ext using only existing schema fields: --command goes in origin and selects the owning ES component for parent_ui, and the agent, run ID and CLI version go in the standard user agent. Feedback from a detected AI agent requires user confirmation. Co-authored-by: Isaac <no-reply@databricks.com>
c3634a9 to
2f69b13
Compare
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
|
@tedjt, this was just a quick agentic review. I'll look into how we can do this. Findings from
The affected unit tests and new feedback acceptance test passed. The cases above were verified locally using fake endpoints, terminal repros, and the repository's changelog validator. This comment was generated with GitHub MCP. |
## Changes Users, and agents acting for them, can't report a CLI problem from the CLI. This adds `databricks feedback [--command "<command>"] MESSAGE`, which sends the message to Databricks from the authenticated workspace. It supersedes databricks#7567 and keeps its behavior. - `--command` names the command the feedback is about, as a free string. Its first word picks the owning team: `air` goes to `ModelTraining.AIRuntime`, `bundle` to `DeveloperEcosystem.CoreDabs`, anything else to `DeveloperEcosystem.DatabricksCLI`. Its words may only use command-name characters, which rejects flags and paths, and it is at most 200 characters. `origin` is the words joined with underscores (`bundle_deploy`). - Each submission is one `fast_feedback_log` event (the in-product feedback widget's) posted to `/telemetry-ext`, using existing fields only; the agent and `--run-id` ride in the user agent. - A detected AI agent needs the user's approval: a prompt when stdin and stderr are terminals, or `--yes` otherwise. The prompt shows the message with invisible characters escaped, the workspace, and every choice sent with it. Declining, including Ctrl-D, prints "Feedback not sent." and exits 1. Humans are not prompted. - The message, flags and approval are checked before authenticating. `--target` is rejected, because the bundle is never loaded. An unquoted message or `--command` gets an error that says what to quote. - The send makes one attempt with SDK retries off and is bounded by 30 s, including a token refresh; the login before it works as for any command. Success prints a reference ID (JSON with `-o json`). A failure names the ID and says whether the feedback was definitely not sent: it failed before a connection was set up or while authenticating, or the server answered 4xx or accepted none of it. - Feedback is sent even with `DATABRICKS_CLI_DISABLE_TELEMETRY` set; the help says so. - `telemetry.Send`, `telemetry.NotSent` and `cmdio.IsErrorTTY` are new, and `DatabricksCliLog` is tagged `omitzero` so a feedback entry carries only `fast_feedback_log` (existing events always set it). ## Why Feedback text is retained centrally, so an agent's text needs the user's approval before it is sent. A retry after an ambiguous failure could file the feedback twice, so the command makes one attempt and reports the ID on failure. ## Decisions to review - `--command` is not looked up in the command tree, which would mean building every command (the tree is lazy since databricks#7487). A typo, an argument or a `databricks` prefix reaches `origin` as typed, and a misspelled `bundle` goes to the CLI team. Keeping only the first word would drop details like `bundle_deploy`. - `Text` is user-authored, which telemetry normally never carries. That is the point of the event, and an agent's text needs the user's approval first. - The prompt reads a plain line and needs only stdin and stderr to be terminals, not `cmdio.IsPromptSupported`, so it also works with `TERM=dumb` and MSYS2 pipes. It escapes invisible characters instead of stripping them like `cmd/air`'s `stripControl`, so the user sees exactly what is sent. - `feedback` is exempt from the aitools install hint, like other one-shot commands. - The single-attempt client repeats the setup in `cmd/air/validateconfig.go`; sharing it would touch `cmd/air` and `experimental/ssh`. ## Tests - `acceptance/cmd/feedback`: routing, the JSON receipt, agent refusal and `--yes`, input errors (including without a login), telemetry disabled, and a broken bundle. - `TestSendMakesOneAttempt`: one request each for success, `numProtoSuccess: 0`, HTTP 403, 500 and 503, and a TCP reset. `TestSendFailingBeforeConnectingIsNotSent`: a refused connection and an untrusted certificate count as not sent. `TestSendTimeoutBoundsTokenAcquisition`: a blocked token source stops at the deadline. - Unit tests cover input checks and quoting hints, approval with `NO_COLOR`, `TERM=dumb` and stderr redirected, escaping, and that a declined or ended prompt sends nothing. The full unit and acceptance suites, `./task lint` and `./task checks` pass locally. - One real submission to a Databricks staging workspace was stored with every field as sent. The acceptance test stays local, because a cloud run would file real feedback on every CI run. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Add
databricks feedback [--command "<command>"] MESSAGE, a single CLI-wide command for sending feedback to Databricks.--commandnames the command the feedback is about. Its top-level command group decides where the feedback is filed:--command "air ..."goes toModelTraining.AIRuntime.--command "bundle ..."goes toDeveloperEcosystem.CoreDabs.--command, goes toDeveloperEcosystem.DatabricksCLI.Adding a product means adding one entry to
productComponents.--commandmust name a real CLI command.Each submission is one
fast_feedback_logevent sent to/telemetry-extin the authenticated workspace. This is the event the in-product feedback widget records, and the change uses only the fields the schema has today:parent_ui--commandgroup (see above)originair_runorbundle_deploy;clifor general feedbackreference_event_idhttp_access_loguser_agentcmd/...,interactive/..., plusagent/<name>when an AI agent is detected andrun-id/<id>when--run-idis setui_versionscli:<version>can_be_contacted--can-be-contacted(default false)textConfirmation for AI agents. When an agent is detected (
useragent.AgentProvider), the command asks the user to confirm. When no terminal is attached, it refuses unless--yesis passed, and the error includes a note telling the agent to show the user the exact message first. Feedback from a human is sent without a prompt.Receipt. On success the command prints a reference ID, or
{"received": true, "feedback_id": ...}with-o json. It makes one upload attempt with a 30s timeout and no retries, because a retry after an ambiguous failure could file the same feedback twice.Supporting changes:
telemetry.Sendfor a single, synchronous upload.FrontendLogEntry.DatabricksCliLogis now taggedomitzero(it wasomitempty, which has no effect on struct values), so a feedback event carries onlyfast_feedback_log, matching the proto oneof.root.CommandStringsooriginuses the same command naming as telemetry.Why
Users, and agents acting for them, have no way to report CLI issues from the CLI itself. This builds on the goals of #7254 (AIR feedback with agent attribution, run context and a receipt), with three differences:
--commandroutes the feedback to the owning team.Tests
acceptance/cmd/feedbackcovers:--command(a product subcommand, a command group, a CLI-owned command) and no--command--yes(refused) and with--yes(agent/recorded)--commandand--run-idtelemetry.Send.This pull request and its description were written by Isaac.