Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .agents/rules/auto-generated-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ globs:
- "internal/genkit/tagging.py"
- "internal/mocks/**/*.go"
- "bundle/direct/dresources/configs/*.generated.yml"
- "bundle/direct/dresources/testdata/field_copies.txt"
- "bundle/internal/validation/generated/*.go"
- "bundle/schema/jsonschema.json"
- "python/databricks/bundles/version.py"
Expand All @@ -33,6 +34,7 @@ paths:
- "internal/genkit/tagging.py"
- "internal/mocks/**/*.go"
- "bundle/direct/dresources/configs/*.generated.yml"
- "bundle/direct/dresources/testdata/field_copies.txt"
- "bundle/internal/validation/generated/*.go"
- "bundle/schema/jsonschema.json"
- "python/databricks/bundles/version.py"
Expand Down Expand Up @@ -63,8 +65,8 @@ Files matching this rule's glob pattern are most likely generated artifacts. Aut
- OpenAPI SDK/CLI command stubs and related generated artifacts:
- `./task generate-cligen` — regenerates `cmd/account/**`, `cmd/workspace/**` and `.gitattributes` from the checked-in `.codegen/cli.json`.
- `./task generate-clijson` — refreshes `.codegen/cli.json` from the OpenAPI spec via genkit (requires universe repo); also updates `internal/genkit/tagging.py`.
- Direct engine generated YAML:
- `./task generate-direct` (or `./task generate-direct-apitypes`, `./task generate-direct-resources`)
- Direct engine generated YAML and field copy listing:
- `./task generate-direct` (or `./task generate-direct-apitypes`, `./task generate-direct-resources`, `./task generate-direct-copies`)
- Bundle schemas:
- `./task generate-schema`
- Rewrites `bundle/internal/schema/annotations.yml` in place: upstream docs are sourced from `.codegen/cli.json` at generation time, and the file is synced with the config structure (placeholders added, stale entries dropped).
Expand Down
9 changes: 5 additions & 4 deletions .agents/skills/bump-sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion

The SDK version lives in `go.mod` (`github.com/databricks/databricks-sdk-go`) and the pinned spec SHA lives in `.codegen/_openapi_sha`.
These two move as a pair; everything else in this skill is regenerated from them or is fallout you fix by hand.
Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `bundle/direct/dresources/configs/*.generated.yml`, `bundle/direct/dresources/testdata/field_copies.txt`, `python/databricks/bundles/**`); regenerate them.

The Python tasks (`pydabs-*`, and the `pydabs-codegen` step inside `generate-check`) all run through `uv`. If one fails because `uv` is missing or because the host's `python3` is too old (e.g. 3.9), install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`) rather than touching the system Python: `uv run` provisions the interpreter each package pins (`>=3.10`, and `==3.13.*` under `python/codegen/`) and downloads it if needed. Do not chase the system Python version.
The acceptance goldens are the exception: they need CI's `uv` index, see step 4.
Expand Down Expand Up @@ -52,7 +52,7 @@ Decide with the user whether to keep the tagging-producer drift or restore those
The task ends with step 4 (`generate-clijson-post` runs `bump-sdk-post`) and ignores its error, so read the end of its output, or go on with step 4 directly.

**4. Regenerate everything downstream: `./task bump-sdk-post`.**
This is the whole mechanical part of the bump: `install-pythons` and `install-ruff`, then `generate-check` (command stubs, refschema, schema, validation, direct-engine YAML, pydabs), then `test-update` for the acceptance goldens.
This is the whole mechanical part of the bump: `install-pythons` and `install-ruff`, then `generate-check` (command stubs, refschema, schema, validation, direct-engine YAML and field copy listing, pydabs), then `test-update` for the acceptance goldens.
Everything after it needs judgment.
Run directly, it fails on errors: `generate-refschema` and `test-update` build the CLI, so it fails while the bump still breaks compilation; fix that (step 5) and rerun the task.

Expand All @@ -62,13 +62,14 @@ Run it in CI's environment, or the goldens pick up local drift:
- Ruff: `./task install-ruff` installs CI's version (0.9.1) via uv; `bump-sdk-post` adds uv's tool executable directory to `PATH` for `test-update`. The acceptance runner only checks that Ruff is available.
- Go: the `toolchain` in `go.mod`. A newer local `go` also changes goldens and generated `gofmt` output; use `GOTOOLCHAIN=<pinned>` if they differ.

Before going on, account for every changed file outside the generated set, including the bumper's (its environment is not CI's either): a bump without fallout changes `cmd/**`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `python/databricks/bundles/**`, `acceptance/bundle/refschema/output` and help-text goldens (`acceptance/help/output.txt`, `acceptance/cmd/account/account-help/output.txt`).
Before going on, account for every changed file outside the generated set, including the bumper's (its environment is not CI's either): a bump without fallout changes `cmd/**`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `bundle/direct/dresources/testdata/field_copies.txt`, `python/databricks/bundles/**`, `acceptance/bundle/refschema/output` and help-text goldens (`acceptance/help/output.txt`, `acceptance/cmd/account/account-help/output.txt`).
Revert any golden change you can't attribute to the new SDK or spec.

**5. Handle SDK breaking changes.**
Read the SDK's `CHANGELOG.md` at the target version (in the module cache) to enumerate breaking changes before chasing compile errors.
A removed struct field that the CLI used (e.g. `jobs.AiRuntimeTask.CodeSourcePath`) should have its usage temporarily disabled with a comment noting it returns in a later SDK bump, not deleted outright.
A new struct field triggers an `exhaustruct` lint failure in `bundle/direct/dresources/*`. Run `./task lint` and (in case of issues) wire the field through `PrepareState` and `RemapState` when it exists on both the input and remote types.
A new struct field usually needs no wiring in `bundle/direct/dresources/*`: the structs built with `newCopier` pick it up, and step 4 records that in `bundle/direct/dresources/testdata/field_copies.txt`. Read that file's diff as the review of the wiring: `copied` means a request or state now carries the field, `not in source` means it stays unset because the struct it is copied from has no such field, and `not in destination` means the copy drops it (see "Copying fields between SDK structs" in `bundle/direct/dresources/README.md`). A state field that an update request drops cannot be updated in place, so classify it in `<resource_type>.yml`. `TestFieldCopies` fails on a field whose JSON name or type differs between the two structs (types of the same basic kind are converted) and on a path or query parameter; skip it in the copier and set it by hand.
A struct literal that is still built by hand triggers an `exhaustruct` lint failure instead. Run `./task lint` and (in case of issues) wire the field through by hand.
Where a remote is rebuilt by hand from a GET response (e.g. `makeJobRunRemote` maps `jobs.Run` into the `jobs.RunNow`-shaped `JobRunRemote`), copy the new field from the response if the response has it, and list it with the request-only zero values otherwise.
`job_runs.yml` must list every `jobs.RunNow` field under `ignore_remote_changes`; `TestJobRunIgnoresEveryRequestField` fails on a new one.
A field the new spec now annotates as output-only may already be emitted into `<resource_type>.generated.yml`, making the manual entry in `<resource_type>.yml` redundant; `TestResourcesYMLNoRedundantRules` catches this, so remove the manual entry.
Expand Down
5 changes: 5 additions & 0 deletions .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,11 @@ linters:
- encoded-compare # Always replaces Equal() with JSONEq, but sometimes exact string match is wanted for JSON
exhaustive:
default-signifies-exhaustive: true
exhaustruct_v5:
# A struct that copies another one gets its ForceSendFields from
# structcopy, and any other literal lists it only to force a zero value.
optional-patterns:
- '.*#ForceSendFields$'
exclusions:
generated: lax
presets:
Expand Down
39 changes: 30 additions & 9 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -996,8 +996,9 @@ tasks:
# then refresh the acceptance goldens. generate-clijson-post runs it last;
# run it directly to redo it, e.g. after hand fixes. generate-refschema and
# test-update build the CLI, so it fails while the bump breaks compilation
# (code using a removed SDK field). Those fixes, exhaustruct wiring and the
# changelog PR link stay manual (.agents/skills/bump-sdk).
# (code using a removed SDK field). Those fixes, exhaustruct wiring of the
# struct literals that are built by hand and the changelog PR link stay
# manual (.agents/skills/bump-sdk).
bump-sdk-post:
desc: Regenerate everything downstream of an SDK bump and refresh acceptance goldens
cmds:
Expand Down Expand Up @@ -1066,13 +1067,14 @@ tasks:
- "sh -c 'go run ./bundle/internal/validation/. && gofmt -w -s ./bundle/internal/validation/generated'"

generate-direct:
desc: Generate direct engine config (apitypes + resources)
# Depend only on generate-direct-resources; it already pulls in
# generate-direct-apitypes transitively. Listing both as parallel deps
# races: resources reads apitypes.generated.yml while the apitypes step's
# `>` redirect is truncating it, silently dropping every resource whose
# type comes from that file. Keep this chain linear.
deps: ['generate-direct-resources']
desc: Generate direct engine config (apitypes + resources) and field copy listing
# Depend only on generate-direct-copies; it already pulls in
# generate-direct-resources and generate-direct-apitypes transitively.
# Listing them as parallel deps races: resources reads
# apitypes.generated.yml while the apitypes step's `>` redirect is
# truncating it, silently dropping every resource whose type comes from
# that file. Keep this chain linear.
deps: ['generate-direct-copies']

generate-direct-apitypes:
desc: Generate direct engine API types YAML
Expand All @@ -1099,6 +1101,25 @@ tasks:
cmds:
- "uv run --script bundle/direct/tools/generate_resources.py .codegen/cli.json bundle/direct/dresources/apitypes.generated.yml bundle/direct/dresources/apitypes.yml acceptance/bundle/refschema/output bundle/direct/dresources/configs"

# Refreshes the listing of what every structcopy.Copier of the direct engine
# does with each field of the two structs it copies between (see
# TestFieldCopies). An SDK bump (go.mod / go.sum) that adds a field to one of
# them changes it. Runs after generate-direct-resources, whose output the
# package embeds.
generate-direct-copies:
desc: Regenerate bundle/direct/dresources/testdata/field_copies.txt
deps: ['generate-direct-resources']
sources:
- bundle/direct/dresources/**/*.go
- bundle/direct/dresources/configs/*.yml
- libs/structs/structcopy/*.go
- go.mod
- go.sum
generates:
- bundle/direct/dresources/testdata/field_copies.txt
cmds:
- go test ./bundle/direct/dresources -run '^TestFieldCopies$' -update

# pydabs-* tasks are defined in python/Taskfile.yml (included above).

# --- Benchmarks ---
Expand Down
40 changes: 40 additions & 0 deletions bundle/direct/dresources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,46 @@ So when a field comes back under a different path than `StateType` uses:
This makes the field present in `InputType`, `StateType`, and `RemoteType`, so it participates
in normal drift detection and is no longer subject to the `missing_in_remote` suppression.

## Copying fields between SDK structs: newCopier

The SDK describes a resource with several structs (a response, a create request, an update
request) that share most of their fields. Do not list those fields in a struct literal. Declare a
copier instead, which copies every field the two structs have under the same name:

```go
var toSchemaState = newCopier[catalog.SchemaInfo, SchemaState]()

func (*ResourceSchema) RemapState(info *catalog.SchemaInfo) *SchemaState {
return toSchemaState.Copy(info)
}
```

A field the SDK adds to both structs is then copied without a code change. A `RemapState` written
this way is the dumb copy the section above asks for.

Tell the copier about the exceptions only, each with a comment that says why:

- `structcopy.Skip("Owner")` leaves a field unset although the source has it. Use it for a field the request must not carry and for one that is set by hand after the copy (the ID, a value that needs converting).
- `structcopy.ForceSend("Comment")` sends a field even when it is zero, so that removing it from the config clears it. Only list fields verified against a real workspace, see `schemaForceSend`.

Do not set `ForceSendFields` next to a copy. The copier derives it: the `ForceSend` fields plus
the copied fields that the source lists. The SDK rejects a name its struct does not have, and the
name of a field that was not copied would send that field's zero value.

A field with the same name but another JSON name or type in the two structs is an error until it
is skipped. Only a type of the same basic kind is converted, as with the enum types the SDK
declares once per request and response. A destination field outside the JSON body (`json:"-"`, a
path or query parameter) is an error until it is skipped as well: it addresses the resource, so
set it from the ID.

`TestFieldCopies` reports these errors, and records what every copier does with each field of
either struct in `testdata/field_copies.txt`. `./task generate-direct` regenerates that file. On
an SDK bump, read its diff the way you would review the wiring of a new field:

- `copied`: the request or state picks the field up.
- `not in source`: the field stays unset, because the struct it is copied from has no such field.
- `not in destination`: the copy drops the field. A state field that an update request drops cannot be updated in place, so classify it in `<resource_type>.yml`.

## OverrideChangeDesc

Use `OverrideChangeDesc` only as a last resort when the `<resource_type>.yml` settings cannot express the needed logic. Skipping an action with `change.Action = deployplan.Skip` in `OverrideChangeDesc` creates a silent no-op: the plan shows no change even if the user's config differs from remote. Document the skip reason clearly in both the comment and `change.Reason`.
Expand Down
5 changes: 2 additions & 3 deletions bundle/direct/dresources/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,8 @@ func (r *ResourceApp) DoCreate(ctx context.Context, config *AppState) (string, *
// For nil (omitted) or false, use no_compute=true (do not start compute).
noCompute := config.Lifecycle == nil || config.Lifecycle.Started == nil || !*config.Lifecycle.Started
request := apps.CreateAppRequest{
App: appRequestBody(config.App),
NoCompute: noCompute,
ForceSendFields: nil,
App: appRequestBody(config.App),
NoCompute: noCompute,
}

retrier := retries.New[apps.App](retries.WithTimeout(15*time.Minute), retries.WithRetryFunc(shouldRetry))
Expand Down
Loading
Loading