Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/cli/feedback-agent-submissions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Allow AI agents to submit feedback without confirmation and clarify the guidance on customer content and PII; deprecate `--yes` / `-y` and reject explicit false values from agents, including in a terminal. ([#7795](https://github.com/databricks/cli/pull/7795))
127 changes: 110 additions & 17 deletions acceptance/cmd/feedback/output.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,45 @@

=== Feedback help explains content guidance and agent submissions

>>> [CLI] feedback --help
Send feedback about the Databricks CLI to Databricks.

Describe the command you ran, what happened, what you expected, and any
workaround. Use --command to name the command the feedback is about so it
reaches the team that owns it.

The message is sent with the workspace and user you are authenticated as and
the CLI user agent, which includes the CLI version, operating system and, if
detected, the AI agent running the CLI. No logs, files or transcripts are
collected. Do not include customer content, personally identifiable information
(PII), credentials or other sensitive data. Follow-up contact is opt-in with
--can-be-contacted. Feedback is sent even when DATABRICKS_CLI_DISABLE_TELEMETRY
is set, because this command explicitly submits feedback.

AI agents can submit feedback directly without asking the user for approval.
Use --can-be-contacted only if the user explicitly agrees to be contacted.
After a successful submission, tell the user that feedback was sent and include
the reference ID.

Usage:
databricks feedback MESSAGE [flags]

Examples:
databricks feedback --command "bundle deploy" "deploy should print which resources changed"
databricks feedback --command "air run" --run-id 123 "logs omitted the failure reason"

Flags:
--can-be-contacted[=true|false] Allow Databricks to contact you about this feedback
--command string CLI command the feedback is about, for example "bundle deploy"
-h, --help[=true|false] help for feedback
--run-id string ID of the job run the feedback is about

Global Flags:
--debug[=true|false] enable debug logging
-o, --output type output type: text or json (default text)
-p, --profile string ~/.databrickscfg profile
-t, --target string bundle target to use (if applicable)

=== Feedback about a product command is routed to its team

>>> [CLI] feedback --command air run --run-id 123 --can-be-contacted logs omitted the failure reason
Expand Down Expand Up @@ -106,7 +147,6 @@ Flags:
--command string CLI command the feedback is about, for example "bundle deploy"
-h, --help[=true|false] help for feedback
--run-id string ID of the job run the feedback is about
-y, --yes[=true|false] Confirm sending feedback written by an AI agent

Global Flags:
--debug[=true|false] enable debug logging
Expand Down Expand Up @@ -135,10 +175,12 @@ Error: --target selects a bundle target, which feedback does not use; use --prof
>>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN [CLI] feedback
Error: feedback message must contain 1 to 10000 characters, including a visible one

>>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent [CLI] feedback hello
Error: feedback from an AI agent must be confirmed by the user; run with --yes once the user approves the message
>>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent [CLI] feedback --run-id abc hello
Error: --run-id must be a positive job run ID

Note for AI agents (test-agent): show the user the exact feedback message and the flags you pass with it, such as --can-be-contacted, and pass --yes only after the user approves sending it
>>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent [CLI] feedback --yes=false hello
Flag --yes has been deprecated, omit this flag when sending feedback; skip the feedback command when declining
Error: feedback was not sent because --yes=false

=== General CLI feedback

Expand All @@ -161,24 +203,75 @@ Thank you! Your feedback was sent. Reference: [UUID][8]
}
}

=== An AI agent needs the user's confirmation
=== An AI agent can submit without confirmation

>>> AGENT=test-agent [CLI] feedback hello
Error: feedback from an AI agent must be confirmed by the user; run with --yes once the user approves the message
Thank you! Your feedback was sent. Reference: [UUID][10]

>>> print_requests.py //telemetry-ext
{
"frontend_log_event_id": "[UUID][10]",
"entry": {
"fast_feedback_log": {
"reference_event_id": "[UUID][11]",
"parent_ui": "DeveloperEcosystem.DatabricksCLI",
"origin": "cli",
"can_be_contacted": false,
"ui_versions": "cli:[CLI_VERSION]",
"user_agent": "cli/[CLI_VERSION] cmd/feedback agent/test-agent",
"text": "hello"
}
}
}

=== An AI agent can receive a JSON receipt

>>> AGENT=test-agent [CLI] feedback agent JSON feedback -o json
{
"received": true,
"feedback_id": "[UUID][12]"
}

>>> print_requests.py //telemetry-ext
{
"frontend_log_event_id": "[UUID][12]",
"entry": {
"fast_feedback_log": {
"reference_event_id": "[UUID][13]",
"parent_ui": "DeveloperEcosystem.DatabricksCLI",
"origin": "cli",
"can_be_contacted": false,
"ui_versions": "cli:[CLI_VERSION]",
"user_agent": "cli/[CLI_VERSION] cmd/feedback agent/test-agent",
"text": "agent JSON feedback"
}
}
}

Note for AI agents (test-agent): show the user the exact feedback message and the flags you pass with it, such as --can-be-contacted, and pass --yes only after the user approves sending it
=== An explicit rejection from an AI agent sends nothing

=== An AI agent with --yes
>>> AGENT=test-agent [CLI] feedback --yes=false hello
Flag --yes has been deprecated, omit this flag when sending feedback; skip the feedback command when declining
Error: feedback was not sent because --yes=false

>>> AGENT=test-agent [CLI] feedback -y=false hello
Flag --yes has been deprecated, omit this flag when sending feedback; skip the feedback command when declining
Error: feedback was not sent because --yes=false

>>> assert_not_exists.py [OUTPUT_DIR]/out.requests.txt

=== The deprecated --yes flag remains accepted

>>> AGENT=test-agent [CLI] feedback --yes hello
Thank you! Your feedback was sent. Reference: [UUID][10]
Flag --yes has been deprecated, omit this flag when sending feedback; skip the feedback command when declining
Thank you! Your feedback was sent. Reference: [UUID][14]

>>> print_requests.py //telemetry-ext
{
"frontend_log_event_id": "[UUID][10]",
"frontend_log_event_id": "[UUID][14]",
"entry": {
"fast_feedback_log": {
"reference_event_id": "[UUID][11]",
"reference_event_id": "[UUID][15]",
"parent_ui": "DeveloperEcosystem.DatabricksCLI",
"origin": "cli",
"can_be_contacted": false,
Expand All @@ -192,14 +285,14 @@ Thank you! Your feedback was sent. Reference: [UUID][10]
=== Feedback is sent even with telemetry disabled

>>> DATABRICKS_CLI_DISABLE_TELEMETRY=1 [CLI] feedback telemetry is disabled
Thank you! Your feedback was sent. Reference: [UUID][12]
Thank you! Your feedback was sent. Reference: [UUID][16]

>>> print_requests.py //telemetry-ext
{
"frontend_log_event_id": "[UUID][12]",
"frontend_log_event_id": "[UUID][16]",
"entry": {
"fast_feedback_log": {
"reference_event_id": "[UUID][13]",
"reference_event_id": "[UUID][17]",
"parent_ui": "DeveloperEcosystem.DatabricksCLI",
"origin": "cli",
"can_be_contacted": false,
Expand All @@ -213,14 +306,14 @@ Thank you! Your feedback was sent. Reference: [UUID][12]
=== A broken bundle does not block feedback

>>> withdir broken-bundle [CLI] feedback bundle config does not block feedback
Thank you! Your feedback was sent. Reference: [UUID][14]
Thank you! Your feedback was sent. Reference: [UUID][18]

>>> print_requests.py //telemetry-ext
{
"frontend_log_event_id": "[UUID][14]",
"frontend_log_event_id": "[UUID][18]",
"entry": {
"fast_feedback_log": {
"reference_event_id": "[UUID][15]",
"reference_event_id": "[UUID][19]",
"parent_ui": "DeveloperEcosystem.DatabricksCLI",
"origin": "cli",
"can_be_contacted": false,
Expand Down
22 changes: 18 additions & 4 deletions acceptance/cmd/feedback/script
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ print_feedback() {
| .entry.fast_feedback_log.user_agent |= (split(" ") | map(select(test("^(cli|cmd|run-id|agent)/"))) | join(" "))'
}

title "Feedback help explains content guidance and agent submissions"
trace $CLI feedback --help

title "Feedback about a product command is routed to its team"
trace $CLI feedback --command "air run" --run-id 123 --can-be-contacted " logs omitted the failure reason "
print_feedback
Expand Down Expand Up @@ -32,16 +35,27 @@ musterr trace $CLI feedback --target dev "hello"

title "Input errors are reported before authenticating"
musterr trace env -u DATABRICKS_HOST -u DATABRICKS_TOKEN $CLI feedback " "
musterr trace env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent $CLI feedback "hello"
musterr trace env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent $CLI feedback --run-id abc "hello"
musterr trace env -u DATABRICKS_HOST -u DATABRICKS_TOKEN AGENT=test-agent $CLI feedback --yes=false "hello"

title "General CLI feedback"
trace $CLI feedback "the CLI should support ..."
print_feedback

title "An AI agent needs the user's confirmation"
musterr trace AGENT=test-agent $CLI feedback "hello"
title "An AI agent can submit without confirmation"
trace AGENT=test-agent $CLI feedback "hello" < /dev/null
print_feedback

title "An AI agent can receive a JSON receipt"
trace AGENT=test-agent $CLI feedback "agent JSON feedback" -o json < /dev/null
print_feedback

title "An explicit rejection from an AI agent sends nothing"
musterr trace AGENT=test-agent $CLI feedback --yes=false "hello"
musterr trace AGENT=test-agent $CLI feedback -y=false "hello"
trace assert_not_exists.py "$OUT_REQUESTS"

title "An AI agent with --yes"
title "The deprecated --yes flag remains accepted"
trace AGENT=test-agent $CLI feedback --yes "hello"
print_feedback

Expand Down
Loading
Loading