Repository navigation
direct: copy SDK structs with structcopy instead of field by field - #7789
Draft
janniklasrose wants to merge 9 commits into
Draft
janniklasrose wants to merge 9 commits into
janniklasrose wants to merge 9 commits into
Conversation
The SDK describes one resource with several structs: a response, a create request, an update request. A field they share has the same Go name, JSON name and type in each, so a Copier matches fields by name and is told about the exceptions only (Skip, ForceSend). ForceSendFields is derived rather than copied: it lists the copied fields that the source lists, plus the fields declared with ForceSend. The SDK marshaller rejects a name its struct lacks and would send the zero value of a field that was left unset, which is what utils.FilterFields is called for by hand today. Describe lists what a Copier does with each destination field, for a golden file that shows the effect of an SDK bump. Co-authored-by: Isaac <no-reply@databricks.com>
RemapState and the create, edit and update requests of 20 resources listed every field of the struct they built, so each field the SDK added had to be wired through by hand before exhaustruct passed. They now declare a Copier, which copies the fields both structs have and needs to be told only what to leave out and what to force-send. testdata/field_copies.txt records what every Copier does with each field of its destination. A field the SDK adds changes that file instead of failing the lint, so a bump shows which requests and states pick the field up. A request no longer force-sends the zero value of a field it deliberately leaves unset, which utils.FilterFields allowed when the state listed that field: effective_enable_file_events on an external location update and latest_monitor_failure_msg on a quality monitor update. Requests and remapped state are otherwise unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
The struct literals that the copiers replaced said why a field stayed zero (not supported by DABs, input-only, missing in remote). Carry those notes over to the copier declarations. Co-authored-by: Isaac <no-reply@databricks.com>
generate-direct-copies refreshes testdata/field_copies.txt and runs last in the generate-direct chain, so bump-sdk-post records the fields an SDK bump adds and the generated-files check covers the listing. Document newCopier in the resource guidelines, and reviewing the listing in the bump-sdk skill. Co-authored-by: Isaac <no-reply@databricks.com>
exhaustruct made every struct literal list ForceSendFields, nearly always as nil. Mark the field optional for the linter and delete the lines: a struct that copies another one gets its ForceSendFields from structcopy, and any other literal lists it only to force a zero value. The rest of the diff is gofmt realigning the literals. job_run.go keeps its explicit nils, which its comments refer to. Co-authored-by: Isaac <no-reply@databricks.com>
Their content field is a string in the API and any in the config, so each copier skips it and the caller sets it. The force-send rules that were spliced into ForceSendFields by hand become ForceSend options. DoRead of a dashboard built the same state as responseToState, so it calls it. As with the other resources, a request no longer force-sends the zero value of a field it leaves unset when the state lists that field: the output-only fields of a dashboard (create_time, path, ...) and the etag of a Genie space update. A dashboard state no longer lists SerializedDashboard in ForceSendFields, which has no effect on a field of type any. Co-authored-by: Isaac <no-reply@databricks.com>
The ForceSend fields come first, then the copied fields the source lists. Say so in the Copy doc comment and in the resource guidelines, which also now mention that enum types are converted. Drop zero values from three struct literals. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 592ca0d
Top 4 slowest tests (at least 2 minutes):
|
Describe now lists every field of either struct, so the listing also shows a source field the destination lacks ("not in destination"). For a request built from the state that is a field the request cannot carry, which nothing reported before.
A destination field outside the JSON body (json:"-", a path or query parameter) is an error until it is skipped, also when the source field has the same tag. It addresses the resource, so it is never the source's field of the same name.
isBasic mirrors the SDK's list of basic kinds, and a test covers ForceSend on a field promoted from an embedded struct.
Co-authored-by: Isaac <no-reply@databricks.com>
…ttings The nolint comments said why the deprecated continuous and development fields of a pipeline are still mapped, and makeCreateJob noted that nothing checks whether every job setting is used. Carry both over to the copiers. Name the dashboard's force-sent field once. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds
libs/structs/structcopyand uses it for the field-by-field copies between SDK structs inbundle/direct/dresources. This picks up the idea of #7158 (most of these structs are copies of each other) with a different mechanism.Skipleaves a field unset,ForceSendalways force-sends it.ForceSendFieldsis derived, not copied: theForceSendfields plus the copied fields that the source lists. The direct engine no longer assembles it withutils.FilterFields.RemapStateand their create/edit/update requests. Resource code shrinks by about 640 lines.testdata/field_copies.txtlists what every copier does with each field of either struct (copied,skipped,not in source,not in destination)../task generate-directregenerates it, sobump-sdk-postand the generated-files check cover it.exhaustructtreatsForceSendFieldsas optional, and theForceSendFields: nillines are gone (except injob_run.go, where comments refer to them). That is its own commit, mostly gofmt realignment, and can be split off.Start with
libs/structs/structcopy/copy.go, thenschema.goandcatalog.gofor the pattern, thentestdata/field_copies.txt.Why
A field the SDK adds to a request or response has to be wired by hand through
RemapStateand every request literal beforeexhaustructpasses, one of the hand fixes that keep the SDK bumper's PRs red. With a copier, a field that both structs have is copied without a code change and shows up as one line infield_copies.txt. A field only the destination has shows up asnot in source, and a field only the source has asnot in destination, e.g. a state field that the update request cannot carry, which nothing reported before. The review of the wiring moves from hand-written code to a generated diff.Design decisions:
TestFieldCopies, and so is a destination field outside the JSON body. A path parameter such asUpdateCatalog.Name(json:"-") therefore cannot be filled from the config'snameby accident, and a nested response type has to be converted explicitly.TestFieldCopiesvalidates all of them.exhaustructstill guards. Literals that rename or recompute most fields (job_run.go,PrepareStateof secrets and secret scopes) and requests built by hand are unchanged, so a new field there still fails the lint.One behavior change: a request no longer force-sends the zero value of a field it deliberately leaves unset. Before,
utils.FilterFieldskept such a field inForceSendFieldswhen the state listed it, e.g. when the config set it to an explicit zero value. Affected:effective_enable_file_eventson an external location update,latest_monitor_failure_msgon a quality monitor update, the output-only fields of a dashboard (create_time,path, ...) on create and update, andetagon a Genie space update. No changelog entry, since no documented behavior changes.Tests
Unit tests for
structcopy;TestFieldCopiesfor the copiers of the direct engine.Before/after comparison with a throwaway harness (not in the PR): for every resource type, 40 seeded random states and remote states went through
RemapState,DoCreate,DoUpdate,DoUpdateWithID,DoResizeandDoDeleteonmainand on this branch, recording the remapped state (JSON andForceSendFields) and every non-GET HTTP request. The response conversions of pipelines, serving endpoints, models, dashboards and Genie spaces were compared the same way. Leaving aside output that already differs between two runs ofmain(random idempotency tokens, map iteration order, timeouts), the output is identical except for the behavior change above and twoForceSendFieldsentries without effect (ajson:"-"field of a quality monitor, theany-typedSerializedDashboard). Not visible in that output: a remapped quality monitor no longer fillsCreateMonitor.TableName, thejson:"-"field thatQualityMonitorState.TableNameshadows and that nothing reads, andPrepareStateof an experiment returns&input.CreateExperimentas the other resources do.Simulated SDK bump: in a local copy of the SDK, one field added to
sql.CreateWarehouseRequest,EditWarehouseRequestandGetWarehouseResponse, and one toEditWarehouseRequestonly. Onmain, lint fails twice insql_warehouse.go. On this branch lint passes, and the only change needed is the regenerated listing:./task fmt,./task checks,./task lint,./task generate-schemaand./task generate-directpass without changes.The unit tests of
libs/structs/...andbundle/direct/...and the bundle acceptance suite (go test ./acceptance -run '^TestAccept/bundle') pass. The other acceptance tests were not run locally../task test-unitfails inlibs/auth,libs/genie,cmd/auth,cmd/rootandcmd/aironly, and those packages fail onmainin the same environment as well.This pull request and its description were written by Isaac.