Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
81dcdcf
Add PLAN.md for bundle bitmap
denik Jul 29, 2026
12ede9a
Add bundle bitmap for per-field telemetry
denik Jul 29, 2026
9c7e0b0
Remove PLAN.md
denik Jul 29, 2026
cf70528
Add bool value bits, decode round-trip, and template recording
denik Jul 29, 2026
22b5c49
Move boolean telemetry into a typed Telemetry struct
denik Jul 30, 2026
aa53ef2
Regenerate goldens after rebase; ignore .databricks in bitmap test
denik Jul 30, 2026
c88261d
Fix bitmap test: executable decode script and deterministic cache
denik Jul 30, 2026
393522c
Strip CR when parsing embedded bitmap schema
denik Jul 30, 2026
13b1fc7
Regenerate bitmap schema from scratch; refresh goldens in generate task
denik Jul 30, 2026
b45f27e
Make Telemetry additive: mirror bool metrics into a struct on Bundle
denik Jul 31, 2026
d1737dc
Record DMS undeclared-writer flags as True/False pairs in bitmap
denik Jul 31, 2026
565107a
Revert DMS flags to single bits: false is not lost here
denik Aug 3, 2026
7454ff5
Fix bitmap encoding every telemetry bit as set
denik Aug 3, 2026
bdea82d
Regenerate bitmap schema and drop removed Local test option
denik Aug 11, 2026
375fcde
Regenerate bitmap schema and goldens after rebase
denik Aug 17, 2026
701747a
Port bitmap telemetry mirror to current main
denik Nov 3, 2026
513a976
Regenerate bitmap schema after rebase (data_quality_monitors)
denik Nov 3, 2026
dac236a
Bitmap: record user-authored config; fix presence detection; simplify…
denik Nov 4, 2026
61396ce
Add a dedicated bitmap test that records every schema line
denik Nov 4, 2026
da3b321
Make Telemetry the source of bool_values; encode bool values; add decode
denik Nov 4, 2026
d9d88a5
Add a self-contained bitmap decoder to databricks-bundles
denik Nov 4, 2026
df916ca
Regenerate bitmap schema after rebase (variables.*.sensitive)
denik Nov 4, 2026
9497cac
Regenerate bitmap schema after rebase (materialization_sets)
denik Nov 4, 2026
501221f
Keep the bitmap decoder out of pydabs codegen cleanup and docs
denik Nov 4, 2026
801d6e4
no_drift: log bitmap stderr instead of diffing it
denik Nov 4, 2026
400637e
Add a local-only invariant test for validate and the bitmap round trip
denik Nov 4, 2026
878aad3
Update selftest/selection for the new validate_bitmap invariant test
denik Nov 4, 2026
ce1e34c
bitmap: one bitmap command with -o compressed|text|json
denik Nov 5, 2026
e699e27
bitmap/full: split the schema golden per resource type
denik Nov 5, 2026
b34c554
bitmap/presence: lifecycle.started true on one cluster and false on a…
denik Nov 5, 2026
98e3baa
Report each lifecycle.started value seen instead of a random resource's
denik Nov 5, 2026
18e2acb
Regenerate bitmap schema after rebase (postgres_data_apis)
denik Nov 5, 2026
de04958
Telemetry: BoolSet.SetBool instead of appending to []bool
denik Nov 6, 2026
6b0e1ff
bitmap: compute bits early and fill in variable references later
denik Nov 9, 2026
cca2048
Adapt to optional config bools being *bool
denik Nov 9, 2026
b40cf8a
Regenerate bitmap schema after rebase (feature_specs, materialization…
denik Nov 11, 2026
ec6b8c7
Telemetry: record ai_runtime_task_scheduled/multitask per job
denik Nov 17, 2026
c6914cb
databricks-bundles bitmap decoder: add -o json
denik Nov 17, 2026
6680bd0
Regenerate bitmap schema and goldens after rebase
denik Nov 17, 2026
9446b73
bitmap: record the config at load, before command-line flags set fields
denik Nov 17, 2026
046d0a6
bitmap: WalkSchema returns each path once; test a list-element variable
denik Nov 17, 2026
77c41db
bitmap/full: put the bit first, right-aligned
denik Nov 18, 2026
2930180
bitmap JSON: key a list's elements as tasks[*] instead of tasks.[*]
denik Nov 18, 2026
1167e99
bitmap text: one "PTF <field>" line per field
denik Nov 18, 2026
4bdcde3
python mutator test: expect the PTF text lines
denik Nov 18, 2026
9a7cbc5
Keep the deploy-ai-runtime-task fixture as on main
denik Nov 18, 2026
f83f1df
telemetry/deploy-app-lifecycle-started: print the bitmap
denik Nov 18, 2026
ab12af5
bitmap: record the fields a Python mutator sets on existing resources
denik Nov 20, 2026
ff796db
bitmap/presence: keep the encoded bitmap as out.bitmap.b64; trace the…
denik Nov 20, 2026
779e651
bitmap: decode a newer CLI's bitmap, naming its extra fields "future #N"
denik Nov 20, 2026
809f0e0
telemetry tests: named fields, one per line
denik Nov 20, 2026
d94bc38
Revert the bitmap step in telemetry/deploy-app-lifecycle-started
denik Nov 20, 2026
30aea9a
bitmap tests: named fields, one per line
denik Nov 20, 2026
06659cc
bitmap: store the schema once, compressed (schema.deflate)
denik Nov 20, 2026
14f8670
Send the bundle bitmap with deploy telemetry (bundle_deploy_event.bun…
denik Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/rules/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log

Other things worth knowing:

- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- A one-off boolean needs no proto change: add a field to `bundle.Telemetry` (`bundle/telemetry.go`) with a snake_case `metric:"..."` tag and set it. Pick the type by what is measured: `bool` for a flag reported only when true, `*bool` for a measured value (nil when not measured), `BoolSet` for a measurement taken several times (e.g. once per artifact; record each with `SetBool`, and each value taken is reported once). It lands in `experimental.bool_values` and in the bundle bitmap (`bundle_deploy_event.bundle_bitmap`); run `./task generate-bitmap-schema` to append it to the bitmap schema.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Per-resource state sizes come from the direct engine's state in `resources.json`.
Expand Down
33 changes: 33 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,7 @@ tasks:
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand All @@ -842,6 +843,7 @@ tasks:
cmds:
- task: generate-cligen
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand Down Expand Up @@ -1018,6 +1020,37 @@ tasks:
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

# Appends any newly added bundle config fields to the bitmap schema (see
# cmd/bundle/bitmap). The schema is append-only, so update-schema reads the
# currently embedded schema and only appends. genschema compresses it into
# bundle/bitmap/schema.deflate, which the CLI embeds, and an identical copy in
# the databricks-bundles decoder (python/databricks/bundles/_bitmap). The
# compressed files are not reviewable; the listing in
# acceptance/bundle/bitmap/full shows every schema line with its bit.
generate-bitmap-schema:
desc: Regenerate the bitmap schema and the acceptance goldens that depend on it
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
- acceptance/bundle/bitmap/full/script
- acceptance/bundle/bitmap/full/split.py
- acceptance/bundle/bitmap/presence/script
- acceptance/bundle/bitmap/presence/databricks.yml
- go.mod
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- bundle/bitmap/schema.deflate
- acceptance/bundle/bitmap/full/output/*.txt
- acceptance/bundle/bitmap/presence/out.bitmap.b64
- python/databricks/bundles/_bitmap/schema.deflate
cmds:
- go run . bundle bitmap update-schema > bundle/bitmap/schema.txt.tmp
- go run ./bundle/bitmap/genschema bundle/bitmap/schema.txt.tmp bundle/bitmap/schema.deflate python/databricks/bundles/_bitmap/schema.deflate
- rm bundle/bitmap/schema.txt.tmp
# The encoded bitmap in presence carries the schema size.
- go test ./acceptance -run 'TestAccept/bundle/bitmap/(full|presence)' -update

# Upstream field documentation comes from the checked-in .codegen/cli.json;
# bundle/internal/schema/annotations.yml carries the CLI-owned docs and
# overrides and is rewritten in place (synced with the config structure).
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/bitmap/full/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = []
Empty file.
75 changes: 75 additions & 0 deletions acceptance/bundle/bitmap/full/output/alerts.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
4293 resources.alerts
4294 resources.alerts.*
4295 resources.alerts.*.url
4296 resources.alerts.*.modified_status
4297 resources.alerts.*.lifecycle
4298 resources.alerts.*.lifecycle.prevent_destroy
4299 resources.alerts.*.lifecycle.prevent_destroy=true
4300 resources.alerts.*.lifecycle.prevent_destroy=false
4301 resources.alerts.*.lifecycle.depends_on
4302 resources.alerts.*.lifecycle.depends_on[*]
4303 resources.alerts.*.lifecycle.depends_on[*].resource
4304 resources.alerts.*.create_time
4305 resources.alerts.*.custom_description
4306 resources.alerts.*.custom_summary
4307 resources.alerts.*.custom_template_format
4308 resources.alerts.*.display_name
4309 resources.alerts.*.effective_run_as
4310 resources.alerts.*.effective_run_as.service_principal_name
4311 resources.alerts.*.effective_run_as.user_name
4312 resources.alerts.*.evaluation
4313 resources.alerts.*.evaluation.comparison_operator
4314 resources.alerts.*.evaluation.empty_result_state
4315 resources.alerts.*.evaluation.last_evaluated_at
4316 resources.alerts.*.evaluation.notification
4317 resources.alerts.*.evaluation.notification.notify_on_ok
4318 resources.alerts.*.evaluation.notification.notify_on_ok=true
4319 resources.alerts.*.evaluation.notification.notify_on_ok=false
4320 resources.alerts.*.evaluation.notification.retrigger_seconds
4321 resources.alerts.*.evaluation.notification.subscriptions
4322 resources.alerts.*.evaluation.notification.subscriptions[*]
4323 resources.alerts.*.evaluation.notification.subscriptions[*].destination_id
4324 resources.alerts.*.evaluation.notification.subscriptions[*].user_email
4325 resources.alerts.*.evaluation.source
4326 resources.alerts.*.evaluation.source.aggregation
4327 resources.alerts.*.evaluation.source.display
4328 resources.alerts.*.evaluation.source.name
4329 resources.alerts.*.evaluation.state
4330 resources.alerts.*.evaluation.threshold
4331 resources.alerts.*.evaluation.threshold.column
4332 resources.alerts.*.evaluation.threshold.column.aggregation
4333 resources.alerts.*.evaluation.threshold.column.display
4334 resources.alerts.*.evaluation.threshold.column.name
4335 resources.alerts.*.evaluation.threshold.value
4336 resources.alerts.*.evaluation.threshold.value.bool_value
4337 resources.alerts.*.evaluation.threshold.value.bool_value=true
4338 resources.alerts.*.evaluation.threshold.value.bool_value=false
4339 resources.alerts.*.evaluation.threshold.value.double_value
4340 resources.alerts.*.evaluation.threshold.value.string_value
4341 resources.alerts.*.id
4342 resources.alerts.*.lifecycle_state
4343 resources.alerts.*.owner_user_name
4344 resources.alerts.*.parameters
4345 resources.alerts.*.parameters[*]
4346 resources.alerts.*.parameters[*].name
4347 resources.alerts.*.parameters[*].type
4348 resources.alerts.*.parameters[*].value
4349 resources.alerts.*.parent_path
4350 resources.alerts.*.query_text
4351 resources.alerts.*.run_as
4352 resources.alerts.*.run_as.service_principal_name
4353 resources.alerts.*.run_as.user_name
4354 resources.alerts.*.run_as_user_name
4355 resources.alerts.*.schedule
4356 resources.alerts.*.schedule.pause_status
4357 resources.alerts.*.schedule.quartz_cron_schedule
4358 resources.alerts.*.schedule.timezone_id
4359 resources.alerts.*.update_time
4360 resources.alerts.*.warehouse_id
4361 resources.alerts.*.permissions
4362 resources.alerts.*.permissions[*]
4363 resources.alerts.*.permissions[*].level
4364 resources.alerts.*.permissions[*].user_name
4365 resources.alerts.*.permissions[*].service_principal_name
4366 resources.alerts.*.permissions[*].group_name
4367 resources.alerts.*.file_path
Loading