bundle: Add genie_space resource (direct engine only) by janniklasrose · Pull Request #5282 · databricks/cli

janniklasrose · 2026-05-20T13:12:25Z

Summary

Introduces a new genie_space bundle resource (direct-engine only) that mirrors the existing dashboard pattern.

Test plan

CI
⚠️ No cloud tests enabled in this iteration
Manual live workspace verification

This pull request and its description were written by Isaac.

longchass · 2026-05-26T01:59:45Z

Hi just proposing a fix on the permission URL, thank you.

janniklasrose · 2026-05-21T09:59:16Z

 			break
 		}

-		time.Sleep(1 * time.Second)


Note to reviewers: aligning dashboard with other resources. including here since genie spaces follow dashboards closely

Adds first-class support for Genie spaces as a bundle resource, complete with CRUD via direct-mode deploy, `bundle generate genie-space` to import an existing space, permissions handling, and acceptance tests. The resource configuration follows the dashboards pattern: a `file_path` field points to a local `.genie.json` file whose contents are inlined into `serialized_space` during deployment. The parent_path defaults to `${workspace.resource_path}` and is normalized to the `/Workspace` prefix, matching the API's expected form. Co-authored-by: Isaac

…ng-parent-path errors Genie surfaces a missing parent folder inconsistently across environments: some workspaces return a standard 404 missing-resource error, while others return 400 INVALID_PARAMETER_VALUE with a NOT_FOUND "Tree node ... does not exist" message embedded in the text. Treat both forms as "create the parent directory and retry once". Co-authored-by: Isaac

…sitor VisitGenieSpacePaths existed but was never called by VisitPaths, so NormalizePaths did not rewrite genie_space file_path values from "relative to YAML location" to "relative to bundle root" before applyGenieSpaceTranslations resolved them. The result was that generator output like "../src/<name>.genie.json" failed on deploy with "path ... is not contained in sync root path". Co-authored-by: Isaac

Inline YAML serialized_space stayed as a structured value (map[string]any with int leaves) in the config struct, while state held the JSON string that was sent to the API. structdiff compared an `any` field with reflect.DeepEqual, which reports map != string, so every plan after deploy showed a false update for the genie_space. Marshal inline serialized_space to its JSON string in ConfigureGenieSpaceSerializedSpace, mirroring the file_path code path, so config-side and state-side carry the same type. The genie_space_complex validate test is updated to reflect that serialized_space is now a string regardless of input form, and a new acceptance test under resources/genie_spaces/inline asserts that a deploy + plan cycle is drift-free for inline serialized_space. Co-authored-by: Isaac

Databricks workspaces do not expose a permissions endpoint for Genie Spaces (PUT /permissions/genie/spaces/<id> returns 404 ENDPOINT_NOT_FOUND). Without an upfront check the deploy creates the space first and then errors when applying permissions, leaving partial state behind. Add ValidateGenieSpacePermissions to the PreDeployChecks pipeline so both per-resource permissions and bundle-level permissions propagated by ApplyBundlePermissions surface a clear validation error before any API call is made. Co-authored-by: Isaac

Two minor follow-ups to the genie_spaces work: - ConfigureGenieSpaceSerializedSpace silently let file_path win when a user also set serialized_space inline. Emit a warning that points at the inline block so the user knows their YAML is being dropped on the floor. - ValidateDirectOnlyResources only mentioned the DATABRICKS_BUNDLE_ENGINE env var as a way to opt into direct mode, even though 'bundle.engine: direct' in databricks.yml is the more common entry point. Mention both. Co-authored-by: Isaac

When the Genie API returns parent_path on GetSpace, propagate it through bundle generate genie-space so the produced YAML deploys back to the same workspace folder. The testserver is updated to mirror that response shape so the acceptance fixture exercises the new path. Filter ParentPath out of ForceSendFields in DoRead and responseToGenieSpaceConfig: we deliberately clear ParentPath in the returned GenieSpaceConfig because the GET API does not reliably include it, but the SDK still surfaces it in ForceSendFields when the field appeared on the wire. Without this filter, deploy state serialization force-emits parent_path: "" even though the field is logically unset, producing spurious output diffs. Co-authored-by: Isaac

- Replace switch-with-fallthrough on dyn.Kind with a guard clause to satisfy the exhaustive linter without listing every Kind variant. - Use http.StatusBadRequest in isMissingGenieParentPathError instead of a magic 400 (auto-fix from golangci-lint). Co-authored-by: Isaac

…cally serialized_space is in ignore_remote_changes because we cannot diff a structured local YAML body against a remote JSON string. That makes UI edits invisible at plan time, but the unconditional UpdateSpace request was still sending the local body, so any later update to title or description would silently overwrite UI changes. Use the plan entry to detect whether the user actually changed serialized_space locally; only include it in the update request when the change is an Update action (not a Skip from ignore_remote_changes). Co-authored-by: Isaac

The previous implementation polled w.Workspace.GetStatusByPath using resource.FilePath, which is the local relative path (e.g. "src/foo.genie.json"). Both lookups (with and without the "/Workspace" prefix) were invalid for the workspace API, so currentModified stayed at 0 and the file never updated past the first iteration. Genie has no remote modification timestamp on the response, so use content comparison instead: canonicalize the just-fetched serialized_space and compare against the on-disk body, re-saving only when they differ. The first iteration still always saves, preserving the prior unconditional initial sync. Co-authored-by: Isaac

The parent generate command exposes --key as a persistent flag, but the genie-space subcommand was always deriving the key from the remote title. Read the flag value and fall back to the title-derived key only when not provided. Co-authored-by: Isaac

Calling json.Unmarshal on an empty serialized_space surfaces a confusing "unexpected end of JSON input" error and writes nothing useful. Bail out early with a clear message that names the target file. Co-authored-by: Isaac

DoRead duplicated the field copy and the ParentPath-drop comment that already lives in responseToGenieSpaceConfig. Reuse the helper directly so the two stay in sync. Co-authored-by: Isaac

The user-facing fields (title, description, warehouse_id, parent_path, file_path, serialized_space) had PLACEHOLDER descriptions, leaving the generated reference and resources docs blank. Fill them in with short descriptions and regenerate the schema and docs output. Co-authored-by: Isaac

Lowercase the genie_space error message to satisfy ST1005 and let the linter convert an empty []string{} to a nil slice. Co-authored-by: Isaac

The simple acceptance test fixture was a v1 serialized_space sample that the Genie backend now rejects with 409 ABORTED ("The export format has changed since this export was taken"). Bumps version to 2 and replaces get_example_values / build_value_dictionary with the v2-equivalent enable_format_assistance / enable_entity_matching, matching the format that bundle generate genie-space now produces. Co-authored-by: Isaac

The state DB API gained context, withRecovery and withWrite arguments on origin/main; mirror the dashboard generate command and use the same arguments. Also regenerates the simple acceptance plan output to pick up the WAL-implementation serial increment. Co-authored-by: Isaac

Adapt genie_space to changes that landed on main while this branch was open: - ResourceGenieSpace.DoDelete now takes the third state parameter (_ *resources.GenieSpaceConfig) to match main's IResource.DoDelete signature. Without it the adapter failed to initialize at runtime ("param count mismatch: interface 3, concrete 2"). - Bump the structwalk config.Root field-count guard from 5800 to 6000 to account for the new genie_space fields (count is now 5814). - Regenerate bundle docs, JSON schema field reference, and the genie plan fixture against the rebased tree. Co-authored-by: Isaac

Apply the unresolved self-review comments on PR #5282: - Permissions use object type "genie" (endpoint /permissions/genie/{id}) rather than "genie/spaces" (permissions.go, all_test.go, and the testserver object-type map). - parent_path is a normal, updatable field now that the Genie GET API returns it (without the /Workspace prefix) and the update API accepts it: * DoRead re-adds the /Workspace prefix via the shared ensureWorkspacePrefix (mirrors ResourceDashboard) instead of clearing the field. * DoUpdate sends parent_path so the backend can move the space. * Removed parent_path from recreate_on_changes and ignore_remote_changes. * The testserver strips the /Workspace prefix on write to match the real API, and `generate genie-space` re-adds it so the generated config keeps the conventional form. * Renamed the parent_path_recreate acceptance test to parent_path_update: changing parent_path now plans an update, not a recreate. - serialized_space uses reason: etag_based (matching dashboards) since drift is detected via etag. Also switches isMissingGenieParentPathError to errors.AsType[T] (forbidigo rule from the SDK bump on main) and lets testifylint rewrite the genie assertions in state_load_test.go to assert.Empty. Co-authored-by: Isaac

The genie_spaces permissions acceptance test is direct-only, so analyze_requests.py lists its request files as DIRECT_ONLY. Regenerate bundle/resources/permissions/output.txt to include those entries (the aggregate test wasn't refreshed when the genie_spaces permissions case was added). Co-authored-by: Isaac

Record the Genie create requests and assert the serialized_space we send: - from a file_path .json file: sent verbatim (formatting preserved). - from inline YAML: rendered as valid JSON (parsed with fromjson, which fails the test if the payload is not valid JSON). Co-authored-by: Isaac

Per review: the Genie backend bumps a space's etag when it migrates serialized_space to a newer schema version. Sending the last-observed etag as an If-Match guard would then fail a legitimate bundle update with 409 after such a migration. Stop sending it (Etag left empty). Drift is still detected on read via OverrideChangeDesc, which compares the stored and remote etags — this diverges from dashboards, which do send the etag. Co-authored-by: Isaac

…pace A genie space body can be authored inline via serialized_space (written as YAML and marshalled to a JSON string) or loaded from a .geniespace.json file via file_path, and the resolved value is stored in state. Setting both is ambiguous, so reject it with an error. This intentionally departs from dashboards, which silently let file_path win. The mutator previously only warned on the both-set case and produced a confusing error when neither field was set; an absent serialized_space now passes through (the backend rejects an empty body). Add a table-driven unit test and a validate acceptance test for the both-set error. Co-authored-by: Isaac

Genie spaces are not exposed as workspace files, so the Workspace API (GetStatusByPath) can never resolve one by path. The --existing-path and --existing-genie-space-path flags, resolveFromPath, and their help example were forked from the dashboard command but cannot succeed for genie spaces. Remove them; resolveFromID is now the only resolver. Document --key in the examples; it is supported via the parent generate command's persistent flag but was undocumented here. Add acceptance tests to match dashboard coverage: - genie_space_existing_id_not_found exercises the genie-specific 403-as-missing branch. - genie_space_inplace covers the deploy -> update -> generate --resource --force round trip, pinned to the direct engine (the only engine that deploys genie spaces). Co-authored-by: Isaac

janniklasrose · 2026-06-05T07:51:39Z

+				if ss.IsValid() && ss.Kind() != dyn.KindNil {
+					diags = diags.Append(diag.Diagnostic{
+						Severity:  diag.Error,
+						Summary:   "both file_path and serialized_space are set; specify only one",


Note to reviewers: departure from dashboards, but I think erroring here makes sense to avoid unexpected behaviour.

Adds the two test artifacts the dresources README requires for a new resource: - acceptance/bundle/invariant/configs/genie_space.yml.tmpl (no-drift), wired into the invariant matrix and excluded from cloud since the serialized_space schema is workspace-version-sensitive. - acceptance/bundle/deployment/bind/genie_space/ exercising the bind -> deploy -> unbind -> destroy lifecycle on the direct engine. Also adds the NEXT_CHANGELOG.md entry for the genie_spaces resource. Co-authored-by: Isaac

janniklasrose · 2026-06-05T08:44:43Z

+// isResourceGone would treat a remotely-deleted space as a hard permission
+// error instead of recreating it (on read) or tolerating it (on delete).
+func genieSpaceGoneError(err error) error {
+	if errors.Is(err, apierr.ErrPermissionDenied) {


Note for reviewers: contentious, but that's how genie have implemented it

# Conflicts: # NEXT_CHANGELOG.md

The genie_space.yml.tmpl entry added to acceptance/bundle/invariant/test.toml propagates to the inherited config view dumped in the sibling invariant subdirectories. Regenerated via ./task test-update. Co-authored-by: Isaac

etag is internal drift-detection state persisted by the direct engine, never authored by the user, but it was exposed as a writable genie_space config field with no guard (unlike dashboards, which have ValidateDashboardEtags). A user-set etag was silently ignored, violating the repo rule to reject incompatible inputs with an actionable error. Add ValidateGenieSpaceEtags, mirroring ValidateDashboardEtags, wired into the Initialize phase, plus an acceptance test (pinned to the direct engine since genie spaces are direct-only). Co-authored-by: Isaac

…nt tests The genie_space invariant config (added when wiring up the resource) is incompatible with two invariant subdirs: migrate seeds the scenario with a terraform deploy (genie spaces are direct-only), and continue_293 deploys with v0.293.0 (which predates the resource). Exclude it from both, mirroring the existing direct-only (catalog, external_location) and not-in-old-version (vector_search) exclusions. no_drift still exercises it on the direct engine. Co-authored-by: Isaac

Direct bind only copied the remote etag into temp bind state for dashboards. Genie spaces use the same etag-based drift signal, so binding a genie space (or 'generate genie-space --bind') finalized state without the etag, producing a bogus etag-driven update on the very next plan. Extend the condition to genie_spaces, matching dstate.ExportStateFromData which already handles both. The bind acceptance test now runs 'bundle plan' immediately after bind and asserts it is clean (0 to change); its serialized_space is an inline body that matches the out-of-band-created space byte-for-byte, so the etag is the only possible drift signal. Verified the test fails (1 to change) without the fix. Co-authored-by: Isaac

github-actions · 2026-06-05T09:52:55Z

Approval status: pending

`/acceptance/bundle/` - needs approval

92 files changed
Suggested: @denik
Also eligible: @pietern, @andrewnester, @shreyas-goenka, @anton-107, @lennartkats-db

`/bundle/` - needs approval

39 files changed
Suggested: @denik
Also eligible: @pietern, @andrewnester, @shreyas-goenka, @anton-107, @lennartkats-db

`/cmd/bundle/` - needs approval

4 files changed
Suggested: @denik
Also eligible: @pietern, @andrewnester, @shreyas-goenka, @anton-107, @lennartkats-db

General files (require maintainer)

11 files changed
Based on git history:

@denik -- recent work in bundle/direct/dresources/, libs/testserver/, bundle/config/mutator/resourcemutator/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db) can approve all areas.

See OWNERS for ownership rules.}

janniklasrose · 2026-06-05T09:58:49Z

+# Uses a literal warehouse id and a version-sensitive serialized_space, so this
+# lifecycle test runs against the local mock server only (overrides the Cloud=true
+# inherited from acceptance/bundle/deployment/test.toml).
+Cloud = false


Note to reviewers: All cloud tests are currently disabled, currently doing a couple of runs to spot obvious issues in advance but can be follow-up

janniklasrose temporarily deployed to test-trigger-is May 20, 2026 13:12 — with GitHub Actions Inactive

janniklasrose temporarily deployed to test-trigger-is May 21, 2026 09:56 — with GitHub Actions Inactive

longchass mentioned this pull request May 26, 2026

bundle: fix genie_space permissions URL prefix (stacked on #5282) #5327

Closed

4 tasks

janniklasrose commented Jun 3, 2026

View reviewed changes

janniklasrose added 22 commits June 4, 2026 06:22

bundle: guard empty serialized_space in generate genie-space

0b27cc4

Calling json.Unmarshal on an empty serialized_space surfaces a confusing "unexpected end of JSON input" error and writes nothing useful. Bail out early with a clear message that names the target file. Co-authored-by: Isaac

bundle: collapse DoRead into responseToGenieSpaceConfig for genie_space

2ff9273

DoRead duplicated the field copy and the ParentPath-drop comment that already lives in responseToGenieSpaceConfig. Reuse the helper directly so the two stay in sync. Co-authored-by: Isaac

bundle: lint cleanups for genie_space review fixes

3303b8d

Lowercase the genie_space error message to satisfy ST1005 and let the linter convert an empty []string{} to a nil slice. Co-authored-by: Isaac

Regenerate test toml

b1f7595

Agent review

9891229

janniklasrose force-pushed the janniklasrose/genie branch from d633f4a to a595ca4 Compare June 4, 2026 06:45

janniklasrose added 4 commits June 4, 2026 11:47

Regenerate direct

cdf670c

janniklasrose temporarily deployed to test-trigger-is June 5, 2026 07:25 — with GitHub Actions Inactive

janniklasrose temporarily deployed to test-trigger-is June 5, 2026 07:54 — with GitHub Actions Inactive

janniklasrose commented Jun 5, 2026

View reviewed changes

janniklasrose added 3 commits June 5, 2026 10:36

Autogenerate validation

3911870

Add gitattribute to force geniespace.json to eol=lf

23b0f68

janniklasrose commented Jun 5, 2026

View reviewed changes

janniklasrose added 2 commits June 5, 2026 11:04

Merge remote-tracking branch 'origin/main' into genie

ea9e983

# Conflicts: # NEXT_CHANGELOG.md

janniklasrose temporarily deployed to test-trigger-is June 5, 2026 09:13 — with GitHub Actions Inactive

janniklasrose added 3 commits June 5, 2026 11:38

janniklasrose temporarily deployed to test-trigger-is June 5, 2026 09:46 — with GitHub Actions Inactive

janniklasrose changed the title ~~bundle: add genie_space resource for direct deploy~~ bundle: Add genie_space resource (direct engine only) Jun 5, 2026

janniklasrose requested review from andrewnester, denik and pietern June 5, 2026 09:51

janniklasrose marked this pull request as ready for review June 5, 2026 09:52

janniklasrose commented Jun 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bundle: Add genie_space resource (direct engine only)#5282

bundle: Add genie_space resource (direct engine only)#5282
janniklasrose wants to merge 43 commits into
mainfrom
janniklasrose/genie

janniklasrose commented May 20, 2026 •

edited

Loading

Uh oh!

longchass commented May 26, 2026

Uh oh!

janniklasrose May 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janniklasrose Jun 5, 2026

Uh oh!

janniklasrose Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

janniklasrose Jun 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

janniklasrose commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

longchass commented May 26, 2026

Uh oh!

janniklasrose May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janniklasrose Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

janniklasrose Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 5, 2026

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

/cmd/bundle/ - needs approval

General files (require maintainer)

Uh oh!

janniklasrose Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

janniklasrose commented May 20, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval

`/cmd/bundle/` - needs approval

janniklasrose Jun 5, 2026 •

edited

Loading