Skip to content

Fix apps import not inlining app.yml into databricks.yml - #7675

Draft
denik wants to merge 11 commits into
mainfrom
denik/apps-import-inline-app-yml
Draft

denik wants to merge 11 commits into
mainfrom
denik/apps-import-inline-app-yml

Conversation

@denik

@denik denik commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Not ready for merge. Making inlining run loses data: valueFrom entries from app.yml are copied verbatim, but bundle config expects value_from, so the variable is deployed with no value (only a warning) while app.yml is deleted; unquoted values like 3.10 are re-typed and deployed as 3.1. See the app-yml golden diff and its Badness. Stacked on #7689 (test-only), which records today's behaviour.

Changes

Download the app's source files before inlining app.yml, and build the inlined config from ordered pairs.

Why

apps import looked for app.yml in the output directory before the downloader had written anything, so app.yml was never inlined into databricks.yml and stayed on disk — since the command was added in #4293. Once inlining ran, command/env order varied between runs because they were stored in a Go map.

Tests

Uses acceptance/apps/import/app-yml from #7689; the golden diff shows config appearing in databricks.yml, app.yml being removed, and the data loss described above.

Found by analyzing test coverage gaps: the code path was not covered by any test.

This pull request and its description were written by Isaac.

@denik
denik requested review from a team as code owners November 16, 2026 10:28
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Commit: d7fdb6b08 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (250c0ef) latest (v1.26.0)
TestAccept/apps/import/app-yml ✅ ❌ ❌
TestAccept/apps/import/app-yml: branch ✅ | main (250c0ef) ❌ | latest (v1.26.0) ❌

main (250c0ef):

acceptance_test.go:1380: Diff:
        --- apps/import/app-yml/output.txt
        +++ /tmp/TestAcceptappsimportapp-yml467867411/002/output.txt
        @@ -3,27 +3,13 @@
         Loading app 'myapp' configuration
         Creating bundle configuration
         File successfully saved to app.yml
        -Inlined and removed app.yml
         Bundle configuration created at databricks.yml
         Binding bundle to existing app
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
         Successfully bound to app 'myapp'
         Deploying bundle
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/myapp/default/files...
        -Updated apps.app
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Files: 2 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
         Bundle deployed successfully
         Starting app
         ✓ Getting the status of the app myapp
        @@ -38,6 +19,7 @@
         You can now deploy changes with: databricks bundle deploy
         
         >>> ls out
        +app.yml
         databricks.yml
         
         >>> cat out/databricks.yml
        @@ -50,46 +32,23 @@
         resources:
           apps:
             app:
        -      config:
        -        command:
        -          - python
        -          - app.py
        -        env:
        -          - name: GREETING
        -            value: hello
        -          - name: WAREHOUSE_ID
        -            valueFrom: sql-warehouse
        -          - name: PYTHON_VERSION
        -            value: 3.1
               name: myapp
               description: ""
               source_code_path: .
         
        +>>> cat out/app.yml
        +command:
        +  - python
        +  - app.py
        +env:
        +  - name: GREETING
        +    value: hello
        +  - name: WAREHOUSE_ID
        +    valueFrom: sql-warehouse
        +  - name: PYTHON_VERSION
        +    value: 3.10
        +
         === App config the bundle deploys (app.yml, if kept, is uploaded with the source)
         
         >>> [CLI] bundle validate -o json
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
        -{
        -  "command": [
        -    "python",
        -    "app.py"
        -  ],
        -  "env": [
        -    {
        -      "name": "GREETING",
        -      "value": "hello"
        -    },
        -    {
        -      "name": "WAREHOUSE_ID"
        -    },
        -    {
        -      "name": "PYTHON_VERSION",
        -      "value": "3.1"
        -    }
        -  ]
        -}
        +null
        
    acceptance_test.go:1158: 
        LOG.setup: Importing files from src
        LOG.setup: app.yml -> /Workspace/Users/tester@databricks.com/myapp/app.yml
        LOG.setup: Import complete
        LOG.setup: {
        LOG.setup:   "active_deployment": {
        LOG.setup:     "deployment_id": "deploy-8794919112821874234",
        LOG.setup:     "source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:     "status": {
        LOG.setup:       "message": "Deployment succeeded",
        LOG.setup:       "state": "SUCCEEDED"
        LOG.setup:     }
        LOG.setup:   },
        LOG.setup:   "app_status": {
        LOG.setup:     "message": "Application is running.",
        LOG.setup:     "state": "RUNNING"
        LOG.setup:   },
        LOG.setup:   "compute_size": "MEDIUM",
        LOG.setup:   "compute_status": {
        LOG.setup:     "message": "App compute is active.",
        LOG.setup:     "state": "ACTIVE"
        LOG.setup:   },
        LOG.setup:   "default_source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:   "forward_user_access_token": true,
        LOG.setup:   "id": "1000",
        LOG.setup:   "name": "myapp",
        LOG.setup:   "service_principal_client_id": "7a0dd455-ec0a-bcdb-7a0d-d455ec0abd21",
        LOG.setup:   "service_principal_id": 8794919112821882846,
        LOG.setup:   "service_principal_name": "app-myapp",
        LOG.setup:   "url": "myapp-123.cloud.databricksapps.com"
        LOG.setup: }
        LOG.setup: {
        LOG.setup:   "deployment_id": "deploy-8794919112837697563",
        LOG.setup:   "source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:   "status": {
        LOG.setup:     "message": "Deployment succeeded",
        LOG.setup:     "state": "SUCCEEDED"
        LOG.setup:   }
        LOG.setup: }
FAIL

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- apps/import/app-yml/output.txt
        +++ /tmp/TestAcceptappsimportapp-yml2528709551/002/output.txt
        @@ -3,27 +3,13 @@
         Loading app 'myapp' configuration
         Creating bundle configuration
         File successfully saved to app.yml
        -Inlined and removed app.yml
         Bundle configuration created at databricks.yml
         Binding bundle to existing app
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
         Successfully bound to app 'myapp'
         Deploying bundle
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/myapp/default/files...
        -Updated apps.app
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Files: 2 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
         Bundle deployed successfully
         Starting app
         ✓ Getting the status of the app myapp
        @@ -38,6 +19,7 @@
         You can now deploy changes with: databricks bundle deploy
         
         >>> ls out
        +app.yml
         databricks.yml
         
         >>> cat out/databricks.yml
        @@ -50,46 +32,23 @@
         resources:
           apps:
             app:
        -      config:
        -        command:
        -          - python
        -          - app.py
        -        env:
        -          - name: GREETING
        -            value: hello
        -          - name: WAREHOUSE_ID
        -            valueFrom: sql-warehouse
        -          - name: PYTHON_VERSION
        -            value: 3.1
               name: myapp
               description: ""
               source_code_path: .
         
        +>>> cat out/app.yml
        +command:
        +  - python
        +  - app.py
        +env:
        +  - name: GREETING
        +    value: hello
        +  - name: WAREHOUSE_ID
        +    valueFrom: sql-warehouse
        +  - name: PYTHON_VERSION
        +    value: 3.10
        +
         === App config the bundle deploys (app.yml, if kept, is uploaded with the source)
         
         >>> [CLI] bundle validate -o json
        -Warning: Use 'value_from' instead of 'valueFrom'
        -  at resources.apps.app.config.env[1]
        -  in databricks.yml:18:13
        -
        -The field 'valueFrom' should be 'value_from' (snake_case). The 'valueFrom' field will be ignored.
        -
        -{
        -  "command": [
        -    "python",
        -    "app.py"
        -  ],
        -  "env": [
        -    {
        -      "name": "GREETING",
        -      "value": "hello"
        -    },
        -    {
        -      "name": "WAREHOUSE_ID"
        -    },
        -    {
        -      "name": "PYTHON_VERSION",
        -      "value": "3.1"
        -    }
        -  ]
        -}
        +null
        
    acceptance_test.go:1158: 
        LOG.setup: Importing files from src
        LOG.setup: app.yml -> /Workspace/Users/tester@databricks.com/myapp/app.yml
        LOG.setup: Import complete
        LOG.setup: {
        LOG.setup:   "active_deployment": {
        LOG.setup:     "deployment_id": "deploy-8794919115579448673",
        LOG.setup:     "source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:     "status": {
        LOG.setup:       "message": "Deployment succeeded",
        LOG.setup:       "state": "SUCCEEDED"
        LOG.setup:     }
        LOG.setup:   },
        LOG.setup:   "app_status": {
        LOG.setup:     "message": "Application is running.",
        LOG.setup:     "state": "RUNNING"
        LOG.setup:   },
        LOG.setup:   "compute_size": "MEDIUM",
        LOG.setup:   "compute_status": {
        LOG.setup:     "message": "App compute is active.",
        LOG.setup:     "state": "ACTIVE"
        LOG.setup:   },
        LOG.setup:   "default_source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:   "forward_user_access_token": true,
        LOG.setup:   "id": "1000",
        LOG.setup:   "name": "myapp",
        LOG.setup:   "service_principal_client_id": "7a0dd456-9067-fd7f-7a0d-d4569067fdbb",
        LOG.setup:   "service_principal_id": 8794919115579457846,
        LOG.setup:   "service_principal_name": "app-myapp",
        LOG.setup:   "url": "myapp-123.cloud.databricksapps.com"
        LOG.setup: }
        LOG.setup: {
        LOG.setup:   "deployment_id": "deploy-8794919115607419146",
        LOG.setup:   "source_code_path": "/Workspace/Users/tester@databricks.com/myapp",
        LOG.setup:   "status": {
        LOG.setup:     "message": "Deployment succeeded",
        LOG.setup:     "state": "SUCCEEDED"
        LOG.setup:   }
        LOG.setup: }
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: d7fdb6b

Run: 44084242956

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 11:50
💚​ aws-windows-latest-4core-16gb 1 272 14 10:47
💚​ azure linux-2core-8gb 1 269 16 10:21
💚​ azure-windows-latest-4core-16gb 1 271 14 8:42
🔄​ gcp linux-2core-8gb 1 1 269 16 14:50
💚​ gcp-windows-latest-4core-16gb 1 272 14 13:04
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalSyncPythonNotebookDelete ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
Top 9 slowest tests (at least 2 minutes):
duration env testname
4:51 gcp-windows-latest-4core-16gb TestAccept
3:49 aws-windows-latest-4core-16gb TestAccept
3:35 azure-windows-latest-4core-16gb TestAccept
3:24 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:29 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/dbfs
2:26 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:12 gcp-windows-latest-4core-16gb TestFsCpFileToFileWithOverwriteFlag/dbfs_to_uc-volumes
2:10 aws-windows-latest-4core-16gb TestExportDir
2:05 aws linux-2core-8gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_python

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmd/apps/import.go:604: Entries inside the env list still come from a Go map, so key order is random and the new golden fails on macOS and Windows. Can the entries be converted with a stable key order?

@denik

denik commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in 8673a71: app.yml is now loaded with yamlloader into a dyn.Value instead of map[string]any, so env entries (and resources) keep the key order from the file — the YAML saver orders keys by source line. The new acceptance test passes 20/20 locally.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the remaining changes before approving.

@denik
denik marked this pull request as draft November 16, 2026 13:46
The app's app.yml has a plain env value, a valueFrom reference and an
unquoted value that YAML reads as a number. Today databricks.yml gets no
config and app.yml is kept as is, although the command means to inline it.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Not ready for merge — keeping this as a draft. An adversarial review found that once inlining actually runs it loses data, and the rebased test now shows it (acceptance/apps/import/app-yml, see its Badness):

  • valueFrom from app.yml is copied verbatim; bundle config only knows value_from, so WAREHOUSE_ID is deployed with no value and only a warning, while app.yml (which had the working reference) is deleted.
  • Unquoted values are re-typed by the YAML loader: value: 3.10 is deployed as 3.1.

Options: (A) translate valueFrom → value_from and keep scalar values as written, or (B) drop the dead inlining code and keep app.yml on disk. The current behaviour is recorded by the test-only #7689, which this PR is stacked on.

…test

On Windows, Git Bash turns a leading-/ argument into a Windows path, so
import-dir and apps deploy got C:/Program Files/Git/Workspace/... and
failed. Disable the conversion for those two commands.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/apps-import-inline-app-yml branch from a154835 to c701158 Compare November 16, 2026 16:14
Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/apps-import-inline-app-yml branch from c701158 to fd29ede Compare November 16, 2026 16:37
Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/apps-import-inline-app-yml branch from fd29ede to 7c29561 Compare November 17, 2026 12:23
denik and others added 7 commits November 17, 2026 12:31
macOS runners have bash 3.2, which has no &>>.

Co-authored-by: Isaac <no-reply@databricks.com>
Records current behavior: app.yml is not inlined into databricks.yml.

Co-authored-by: Isaac <no-reply@databricks.com>
generateAppBundle looked for app.yml/app.yaml before the downloader had
written the app source to disk, so the config was never inlined into
databricks.yml and app.yml was left in place. Download first, and build
the inlined config as an ordered mapping so the output is deterministic.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
app.yml was parsed into map[string]any, so the keys inside env entries
(and resources) came out in random order. Load it as a dyn.Value instead;
the YAML saver orders keys by source line.

Co-authored-by: Isaac <no-reply@databricks.com>
Replace the app-yml-inlined test with the app-yml test from the test-only
PR. Its golden diff shows inlining now runs, and its Badness records that
valueFrom entries lose their value and unquoted values are re-typed.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/apps-import-inline-app-yml branch from 7c29561 to d7fdb6b Compare November 17, 2026 12:31
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
## Changes
Add acceptance test `acceptance/apps/import/app-yml`: `apps import` of
an app whose `app.yml` has a plain env value, a `valueFrom` reference
and an unquoted value YAML reads as a number (`3.10`). It records the
generated `databricks.yml`, what is left on disk, and the app `config`
the imported bundle deploys.

## Why
`apps import` means to inline `app.yml` into `databricks.yml` as the
app's `config` and remove it, but it looks for `app.yml` before the
source files are downloaded, so it never finds it: `databricks.yml` gets
no `config` and `app.yml` stays. The test pins that (with a `Badness`
note) so a change to the inlining shows up as a golden diff.
databricks#7675 is the attempt to make
inlining run, and this test shows what it changes.

## Tests
Test-only. Passes 20/20 runs locally. PR CI skips Windows for
`acceptance/apps/**`; the script disables Git Bash path conversion for
its two `/Workspace` arguments, and
databricks#7675 (stacked on this, touches
`cmd/apps`) runs the same script on Windows.

_Found by analyzing test coverage gaps: the code path was not covered by
any test._

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants