Skip to content

direct: copy SDK structs with structcopy instead of field by field - #7789

Draft
janniklasrose wants to merge 9 commits into
mainfrom
janniklasrose/struct-copier
Draft

janniklasrose wants to merge 9 commits into
mainfrom
janniklasrose/struct-copier

Conversation

@janniklasrose

@janniklasrose janniklasrose commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Changes

Adds libs/structs/structcopy and uses it for the field-by-field copies between SDK structs in bundle/direct/dresources. This picks up the idea of #7158 (most of these structs are copies of each other) with a different mechanism.

// Name addresses the catalog, so it is the id; the name in the config goes in NewName when it differs.
var toUpdateCatalog = newCopier[CatalogState, catalog.UpdateCatalog](
	structcopy.Skip("Name"),
	structcopy.ForceSend(catalogForceSend...),
)

updateRequest := toUpdateCatalog.Copy(config)
updateRequest.Name = id
  • A copier copies the fields both structs have under the same Go name and is told about the exceptions only: Skip leaves a field unset, ForceSend always force-sends it.
  • ForceSendFields is derived, not copied: the ForceSend fields plus the copied fields that the source lists. The direct engine no longer assembles it with utils.FilterFields.
  • 22 resources declare 47 copiers for RemapState and their create/edit/update requests. Resource code shrinks by about 640 lines.
  • testdata/field_copies.txt lists what every copier does with each field of either struct (copied, skipped, not in source, not in destination). ./task generate-direct regenerates it, so bump-sdk-post and the generated-files check cover it.
  • exhaustruct treats ForceSendFields as optional, and the ForceSendFields: nil lines are gone (except in job_run.go, where comments refer to them). That is its own commit, mostly gofmt realignment, and can be split off.

Start with libs/structs/structcopy/copy.go, then schema.go and catalog.go for the pattern, then testdata/field_copies.txt.

Why

A field the SDK adds to a request or response has to be wired by hand through RemapState and every request literal before exhaustruct passes, one of the hand fixes that keep the SDK bumper's PRs red. With a copier, a field that both structs have is copied without a code change and shows up as one line in field_copies.txt. A field only the destination has shows up as not in source, and a field only the source has as not in destination, e.g. a state field that the update request cannot carry, which nothing reported before. The review of the wiring moves from hand-written code to a generated diff.

Design decisions:

  • Matching. Fields match by Go name and must agree on JSON name and type; only types of the same basic kind are converted, for the enum types the SDK declares once per request and response. Anything else is an error until it is skipped, reported by TestFieldCopies, and so is a destination field outside the JSON body. A path parameter such as UpdateCatalog.Name (json:"-") therefore cannot be filled from the config's name by accident, and a nested response type has to be converted explicitly.
  • Reflection at run time, rather than generated Go or the source rewriting of Auto-complete reviewed SDK field copies during bumps #7158. It deletes the copies instead of keeping them up to date, and it needs no generator that must compile the package it writes into (which fails exactly when a bump breaks compilation). It also matches how the direct engine already works: reflection, invariants enforced by unit tests, generated listings for review. A copier inspects its types on first use, and TestFieldCopies validates all of them.
  • What exhaustruct still guards. Literals that rename or recompute most fields (job_run.go, PrepareState of secrets and secret scopes) and requests built by hand are unchanged, so a new field there still fails the lint.

One behavior change: a request no longer force-sends the zero value of a field it deliberately leaves unset. Before, utils.FilterFields kept such a field in ForceSendFields when the state listed it, e.g. when the config set it to an explicit zero value. Affected: effective_enable_file_events on an external location update, latest_monitor_failure_msg on a quality monitor update, the output-only fields of a dashboard (create_time, path, ...) on create and update, and etag on a Genie space update. No changelog entry, since no documented behavior changes.

Tests

  • Unit tests for structcopy; TestFieldCopies for the copiers of the direct engine.

  • Before/after comparison with a throwaway harness (not in the PR): for every resource type, 40 seeded random states and remote states went through RemapState, DoCreate, DoUpdate, DoUpdateWithID, DoResize and DoDelete on main and on this branch, recording the remapped state (JSON and ForceSendFields) and every non-GET HTTP request. The response conversions of pipelines, serving endpoints, models, dashboards and Genie spaces were compared the same way. Leaving aside output that already differs between two runs of main (random idempotency tokens, map iteration order, timeouts), the output is identical except for the behavior change above and two ForceSendFields entries without effect (a json:"-" field of a quality monitor, the any-typed SerializedDashboard). Not visible in that output: a remapped quality monitor no longer fills CreateMonitor.TableName, the json:"-" field that QualityMonitorState.TableName shadows and that nothing reads, and PrepareState of an experiment returns &input.CreateExperiment as the other resources do.

  • Simulated SDK bump: in a local copy of the SDK, one field added to sql.CreateWarehouseRequest, EditWarehouseRequest and GetWarehouseResponse, and one to EditWarehouseRequest only. On main, lint fails twice in sql_warehouse.go. On this branch lint passes, and the only change needed is the regenerated listing:

     dresources.SqlWarehouseRemote -> dresources.SqlWarehouseState
    +  SimulatedBumpField: copied
     dresources.SqlWarehouseState -> sql.EditWarehouseRequest
    +  SimulatedBumpField: copied
    +  SimulatedEditOnly: not in source
  • ./task fmt, ./task checks, ./task lint, ./task generate-schema and ./task generate-direct pass without changes.

  • The unit tests of libs/structs/... and bundle/direct/... and the bundle acceptance suite (go test ./acceptance -run '^TestAccept/bundle') pass. The other acceptance tests were not run locally. ./task test-unit fails in libs/auth, libs/genie, cmd/auth, cmd/root and cmd/air only, and those packages fail on main in the same environment as well.

This pull request and its description were written by Isaac.

janniklasrose and others added 7 commits November 19, 2026 01:05
The SDK describes one resource with several structs: a response, a create request, an update request. A field they share has the same Go name, JSON name and type in each, so a Copier matches fields by name and is told about the exceptions only (Skip, ForceSend).

ForceSendFields is derived rather than copied: it lists the copied fields that the source lists, plus the fields declared with ForceSend. The SDK marshaller rejects a name its struct lacks and would send the zero value of a field that was left unset, which is what utils.FilterFields is called for by hand today.

Describe lists what a Copier does with each destination field, for a golden file that shows the effect of an SDK bump.

Co-authored-by: Isaac <no-reply@databricks.com>
RemapState and the create, edit and update requests of 20 resources listed every field of the struct they built, so each field the SDK added had to be wired through by hand before exhaustruct passed. They now declare a Copier, which copies the fields both structs have and needs to be told only what to leave out and what to force-send.

testdata/field_copies.txt records what every Copier does with each field of its destination. A field the SDK adds changes that file instead of failing the lint, so a bump shows which requests and states pick the field up.

A request no longer force-sends the zero value of a field it deliberately leaves unset, which utils.FilterFields allowed when the state listed that field: effective_enable_file_events on an external location update and latest_monitor_failure_msg on a quality monitor update. Requests and remapped state are otherwise unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
The struct literals that the copiers replaced said why a field stayed zero (not supported by DABs, input-only, missing in remote). Carry those notes over to the copier declarations.

Co-authored-by: Isaac <no-reply@databricks.com>
generate-direct-copies refreshes testdata/field_copies.txt and runs last in the generate-direct chain, so bump-sdk-post records the fields an SDK bump adds and the generated-files check covers the listing.

Document newCopier in the resource guidelines, and reviewing the listing in the bump-sdk skill.

Co-authored-by: Isaac <no-reply@databricks.com>
exhaustruct made every struct literal list ForceSendFields, nearly always as nil. Mark the field optional for the linter and delete the lines: a struct that copies another one gets its ForceSendFields from structcopy, and any other literal lists it only to force a zero value.

The rest of the diff is gofmt realigning the literals. job_run.go keeps its explicit nils, which its comments refer to.

Co-authored-by: Isaac <no-reply@databricks.com>
Their content field is a string in the API and any in the config, so each copier skips it and the caller sets it. The force-send rules that were spliced into ForceSendFields by hand become ForceSend options.

DoRead of a dashboard built the same state as responseToState, so it calls it.

As with the other resources, a request no longer force-sends the zero value of a field it leaves unset when the state lists that field: the output-only fields of a dashboard (create_time, path, ...) and the etag of a Genie space update. A dashboard state no longer lists SerializedDashboard in ForceSendFields, which has no effect on a field of type any.

Co-authored-by: Isaac <no-reply@databricks.com>
The ForceSend fields come first, then the copied fields the source lists. Say so in the Copy doc comment and in the resource guidelines, which also now mention that enum types are converted. Drop zero values from three struct literals.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 19, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 592ca0d

Run: 44445728738

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 4:06
💚​ aws-windows-latest-4core-16gb 1 274 17 3:08
💚​ azure linux-2core-8gb 1 271 19 3:36
💚​ azure-windows-latest-4core-16gb 1 273 17 3:31
💚​ gcp linux-2core-8gb 1 272 19 3:52
💚​ gcp-windows-latest-4core-16gb 1 274 17 4:31
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:50 gcp-windows-latest-4core-16gb TestAccept
3:42 gcp-windows-latest-4core-16gb TestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_uc-volumes
3:27 azure-windows-latest-4core-16gb TestAccept
3:04 aws-windows-latest-4core-16gb TestAccept

janniklasrose and others added 2 commits November 19, 2026 02:36
Describe now lists every field of either struct, so the listing also shows a source field the destination lacks ("not in destination"). For a request built from the state that is a field the request cannot carry, which nothing reported before.

A destination field outside the JSON body (json:"-", a path or query parameter) is an error until it is skipped, also when the source field has the same tag. It addresses the resource, so it is never the source's field of the same name.

isBasic mirrors the SDK's list of basic kinds, and a test covers ForceSend on a field promoted from an embedded struct.

Co-authored-by: Isaac <no-reply@databricks.com>
…ttings

The nolint comments said why the deprecated continuous and development fields of a pipeline are still mapped, and makeCreateJob noted that nothing checks whether every job setting is used. Carry both over to the copiers. Name the dashboard's force-sent field once.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants