Repository navigation
Conversation
denik
force-pushed
the
denik/bundle-bitmap
branch
from
July 30, 2026 10:11
221d417 to
39faace
Compare
Collaborator
✅ Integration test reportCommit: f83f1df
Top 39 slowest tests (at least 2 minutes):
|
denik
force-pushed
the
denik/bundle-bitmap
branch
2 times, most recently
from
August 3, 2026 10:01
eb6b1f0 to
29c5b44
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
2 times, most recently
from
August 17, 2026 11:52
45b8bfe to
9ad0bbb
Compare
Contributor
|
This PR has not received an update in a while. If you want to keep this PR open, please leave a comment below or push a new commit and auto-close will be canceled. |
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 3, 2026 16:34
9ad0bbb to
96a6df5
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 3, 2026 16:42
96a6df5 to
f972c44
Compare
Contributor
|
Commit: Regression test reportSelected 8 changed tests (limit=20, 0 not selected) 76 tests: regression=75 unreleased=0 coverage=1 failing=0
TestAccept/bundle/bitmap/full: branch ✅ | main (5c58219) ❌ | latest (v1.26.0) ❌main (5c58219): latest (v1.26.0): TestAccept/bundle/bitmap/presence/DMS=: branch ✅ | main (5c58219) ❌ | latest (v1.26.0) ❌main (5c58219): latest (v1.26.0): TestAccept/bundle/bitmap/presence/DMS=true: branch ✅ | main (5c58219) ❌ | latest (v1.26.0) ❌main (5c58219): latest (v1.26.0): 72 more failure details omitted, see the full report in the run. |
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 4, 2026 10:08
f972c44 to
1cc9186
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
3 times, most recently
from
November 5, 2026 13:52
5e60941 to
445b523
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 9, 2026 15:40
108e307 to
6d41413
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
2 times, most recently
from
November 9, 2026 22:59
e070c71 to
9739855
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 11, 2026 12:35
9739855 to
2f3d071
Compare
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 11, 2026
## Why Optional bools in CLI-defined config structs were plain `bool` with `omitempty`. The typed-to-dynamic conversion that runs after every mutator drops zero values, so an explicit `false` was lost unless the struct declared `ForceSendFields`: - it never showed in `bundle validate -o json`; - a target could not override `true` with `false`. On main this is broken for `artifacts.*.dynamic_version`, `bundle.deployment.lock.force`, `workspace.azure_use_msi` (auth then fails with "more than one authorization method configured: azure and pat") and `replace_existing` on postgres endpoints, databases and roles; - the bundle bitmap (databricks#6104) could not record it. ## Changes These fields become `*bool`, like `lock.enabled` already is; readers use `config.IsExplicitlyEnabled` (or dereference inline in the direct engine): - `variables.*.sensitive`, `artifacts.*.dynamic_version`, `bundle.deployment.lock.force`, `workspace.azure_use_msi`, `workspace.experimental_is_unified_host`, `targets.*.default`, `experimental.{immutable_folder,python_wheel_wrapper,use_legacy_run_as,skip_artifact_cleanup,skip_name_prefix_for_schema}`, `experimental.pydabs.enabled`; - `replace_existing` on postgres branches, endpoints, databases and roles, and `purge_on_delete` on postgres projects and branches; - `bundle.deployment.fail_on_active_runs`. The last two groups already kept an explicit `false` through `ForceSendFields` declared for them (postgres projects and branches, `Deployment` from databricks#7117); with `*bool` those `ForceSendFields` go away. The postgres configs keep their `MarshalJSON`/`UnmarshalJSON`, which exist because they embed an SDK spec. Still plain `bool`: `bundle.force` (readonly, set from `--force`), and three fields whose struct's `ForceSendFields` covers more than the bool (`lifecycle.prevent_destroy` with `depends_on`, `presets.artifacts_dynamic_version` with the other presets, `dashboards.*.embed_credentials`, which the direct engine passes to the dashboards API). `TestConfigBoolsArePointers` fails for any other `bool` in a CLI-defined config struct. ## Tests `acceptance/bundle/validate/explicit_false`: explicit `false` values are kept, and a target overrides `true` with `false` for every field above that a target can set (on main the broken ones stay `true`). This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 11, 2026 15:52
2f3d071 to
9c777e4
Compare
Co-authored-by: Isaac
Encode presence of every bundle config field as one bit. The schema is derived by reflecting over config.Root, recorded in bundle/bitmap/schema.txt, and embedded into the binary. Append-only so bit positions stay stable. Adds hidden `databricks bundle bitmap` group: schema, update-schema [--validate], bitmap-text, bitmap. Wires generate-bitmap-schema into the Taskfile and runs the encoder over all no_drift invariant configs. Co-authored-by: Isaac
Like acceptance/bundle/refschema: resources.<type>.* goes to output/<type>.txt and everything else to output/root.txt. Each line carries its bit, so a file still shows where its fields sit in the schema. Co-authored-by: Isaac <no-reply@databricks.com>
…nother Co-authored-by: Isaac <no-reply@databricks.com>
app_lifecycle_started, cluster_lifecycle_started and sql_warehouse_lifecycle_started took the value of whichever resource came first in map order, so with one app started and another stopped the reported value changed from run to run. They now report true if any resource of the type sets started: true and false if any sets false, each value once; a bundle whose resources agree still sends a single entry. The fields become []bool, which changes their bitmap entries; the schema is not released yet, so it is regenerated from scratch. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The per-artifact and per-resource flags were []bool fields grown with append. They are now BoolSet, which records the values a measurement took: call sites read b.Telemetry.ArtifactFilesIsSet.SetBool(x), bool_values gets each value taken once (true before false), and the bitmap gives the field a presence bit plus its =true/=false bits, the same entries as any bool. The lifecycle.started loops call SetBool per resource, which replaces the startedValues helper. Telemetry bits are now set directly instead of through a dyn conversion. The schema is not released yet, so it is regenerated. Co-authored-by: Isaac <no-reply@databricks.com>
Initialize now marks the loaded config into a bitmap.Builder kept on the
bundle (replacing the UserConfig snapshot), and the Python mutator marks the
resources it adds. Fields holding a pure ${var.*} reference are kept in a side
set; at the end of Initialize each reference is evaluated against the
variables' final values and its value is marked in place. So a bool set
through a variable records its value (strings from --var or BUNDLE_VAR_* are
coerced like the loader does), and a complex variable records the fields
inside it. Bits are only ever added: an unresolved reference keeps its
presence bit.
bundle.target and bundle.environment are pruned: target selection sets them
on every bundle. The schema is unreleased, so it is regenerated.
The bitmap tests that need bundle.Telemetry move to an external test package,
since bundle now imports bitmap.
Co-authored-by: Isaac <no-reply@databricks.com>
use_legacy_run_as is now a *bool, so read it with config.IsExplicitlyEnabled. An explicit false on such a field now survives loading, so the presence fixture records F experimental.use_legacy_run_as; the comment about the gap is updated. Co-authored-by: Isaac <no-reply@databricks.com>
…_sets triggers) Co-authored-by: Isaac <no-reply@databricks.com>
Each job with an ai_runtime_task now records its value, so a deploy with a scheduled and an unscheduled job reports both true and false, like *_lifecycle_started. Co-authored-by: Isaac <no-reply@databricks.com>
decode_json/to_json return the same object as `databricks bundle bitmap decode -o json`, and `python -m databricks.bundles._bitmap -o json` prints it byte for byte like the CLI. presence and invariant/validate_bitmap diff it against the CLI's JSON. Co-authored-by: Isaac <no-reply@databricks.com>
New fields from main (apps effective_resources and scan_*, pipelines testing, features iceberg_table_source, catalogs isolation_mode, environments image_uri), the [=true|false] bool flag help, and the blank line after a title. Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 17, 2026 13:36
9c777e4 to
6680bd0
Compare
Initialize ran after --var, --profile and the InitFuncs (--force, --force-lock, --fail-on-active-runs, --cluster-id), so passing a variable with --var set variables.*.value while BUNDLE_VAR_* did not. The bitmap now starts when the target is selected. Found by an adversarial review. Co-authored-by: Isaac <no-reply@databricks.com>
The type walk visits a flattened embed at its parent's path and a field together with an embedded field it shadows, so 132 of 5438 paths were repeats. update-schema dropped them in Merge; the schema does not change. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
A list field has no child but its elements, so the elements' subtree is keyed by the field name plus [*] and replaces the field's leaf. Co-authored-by: Isaac <no-reply@databricks.com>
P marks a present field, T and F the values a bool field held, and a space each that is not set, so a bool seen both true and false is one line instead of two. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
TestAiRuntimeTaskMetrics covers jobs that record both values. Co-authored-by: Isaac <no-reply@databricks.com>
Shows lifecycle.started recorded with both values. Co-authored-by: Isaac <no-reply@databricks.com>
Only the resources Python added were recorded. MarkChanges compares the config Python received with the result and marks what is new or different, so a field a mutator sets on a YAML resource counts, and the defaults that resource already had do not. Co-authored-by: Isaac <no-reply@databricks.com>
… diffs The encoded form carries the schema size, so generate-bitmap-schema now refreshes presence along with full. Co-authored-by: Isaac <no-reply@databricks.com>
Both decoders rejected a bitmap with more bits than their schema. They now render the fields they know and a placeholder for each set bit past the schema. bitmap/presence re-encodes its bitmap with two more fields and checks that the CLI and Python decoders agree. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
bitmap/presence already covers a bool with both values under a wildcard. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
schema.txt (357 KB) and the base64 copy in _schema.py (41 KB) become one raw DEFLATE file (29 KB), generated by genschema: the CLI embeds it and the databricks-bundles package ships the same bytes as package data. The listing in acceptance/bundle/bitmap/full is where a schema change is reviewed. Co-authored-by: Isaac <no-reply@databricks.com>
…dle_bitmap) Encoded at the end of the deploy, so it includes the telemetry flags only a deploy measures. Omitted when the config did not load: the bitmap starts once the target is selected. telemetry/deploy decodes the sent bitmap; telemetry/deploy-load-error covers the deploy that fails before the bitmap exists. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today each tracked DABs feature needs custom code (a key constant plus a SetBoolValue call), so only hand-picked features are tracked. The bundle bitmap encodes the presence of every bundle config field as one bit, derived by reflecting over config.Root, so all current and future fields are covered for free (http://go/dabs-telemetry).
Builds on #7412 (merged), which keeps an explicit
falseon optional config bools so the bitmap can record it.Changes
databricks bundle bitmapgroup:schema,bitmap -o compressed|text|json(compressed is DEFLATE + base64, the default),decode(stdin, same-o, defaults to text) andupdate-schema [--validate]. The schema is append-only so bit positions stay stable;task generateappends new fields. It is stored once, compressed (bundle/bitmap/schema.deflate, 29 KB, embedded into the binary, and the same bytes in databricks-bundles); acceptance/bundle/bitmap/full lists it with each field's bit for review. Bool fields also record their value.${var.*}(so a bool set through a variable records its value and a complex variable records the fields inside it).bundle.target/bundle.environmentare left out: target selection sets them on every bundle.bundle.Telemetrystruct (metrictags): one assignment per call site instead of a key constant plus SetBoolValue. Flags measured per artifact or per resource are aBoolSet(SetBool(v)) and report each value taken once:*_lifecycle_startedis true if any resource is started and false if any is stopped (it used to take a random resource's value),ai_runtime_task_scheduled/_multitaskare recorded for each job with an ai_runtime_task, andartifact_*_is_setno longer repeats a value once per artifact. Otherwisebool_valueskeeps the same entries, now in field order.bundle_deploy_event.bundle_bitmap(base64 string), encoded at the end of the deploy so it includes the flags only a deploy measures; omitted when the config did not load. The universe proto field is a separate change.databricks.bundles._bitmapin databricks-bundles: a self-contained decoder with the schema embedded (python -m databricks.bundles._bitmap [-o text|json]).Tests
Go and Python unit tests for presence, bool variants and the wire format. acceptance/bundle/bitmap/presence pins the set bits of a fixture and checks that the CLI and Python decoders (text, JSON) reproduce it; bitmap/full records the whole schema with each field's bit, split per resource type; python/telemetry covers a Python-defined resource; telemetry/deploy decodes the bitmap a deploy sends, and telemetry/deploy-load-error a deploy that fails before the bitmap exists; invariant/validate_bitmap (local-only) runs validate on every invariant config and checks that both decoders reproduce the bits from the encoded bitmap.
This pull request and its description were written by Isaac.