Refactor deployment layout and colocate broker registration errands

AGENTS.md

@@ -1,5 +1,9 @@
 # Development loop
 
+When changing documented commands, paths, or development constraints, update this
+file in the same change. Verify claims against code; link to detailed documentation
+rather than duplicating it.
+
 This is the `juicefs` BOSH release. `src/juicefs` is a pinned upstream Git submodule;
 `packages/juicefs` builds it with the vendored `golang-1.25-linux` BOSH package.
 `nginx_webdav` is vendored from CAPI release and powers our own singleton
@@ -21,8 +25,8 @@ run `scripts/generate-fakes` to regenerate fakes using the pinned vendored tool.
 errand. Local checks include a TLS CF API fixture; the live suite requires
 `JUICEFS_ACCEPTANCE_CONFIG`. CI orchestration lives in `scripts/acceptance/run`;
 see `docs/acceptance-testing.md`. The workflow currently uses the standard
-GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together in
-`scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts.
+GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together
+in `scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts.
 Do not check generated vendor trees for upstream submodules into Git.
 `packages/juicefs` contains only the upstream client. Release-owned tools have
 separate packages and Go modules under matching `src/` directories; these compile
@@ -47,23 +51,18 @@ after adding blobs. The repository home is
 director credentials. Do not commit it or print its contents. Refresh it from
 the instant-bosh repo when the director credentials change.
 
-From the release root:
-
-```sh
-devbox run deploy
-```
+From the release root, run `devbox run deploy` for the instant-bosh validation
+environment. `scripts/deploy` defines its deployment sequence. See
+`README.md#deploy-and-register` and `docs/acceptance-testing.md` for details.
 
-This sources `bosh.env`, runs `bosh create-release --force`, uploads that release,
-updates the named `juicefs` runtime-config with `bosh update-config` using the
-newly built release's explicit version, deploys the JuiceFS release, then runs
-`bosh deploy -n -d cf <(bosh manifest -d cf)` so pending runtime-config changes
-are applied. The JuiceFS manifest contains
-`juicefs-backends` and `juicefs-broker` instance groups and uses `version: latest` for the
-`juicefs` release. The director must have an
-`ubuntu-noble` stemcell and cloud-config with `z1`, `default` network and
-`default` VM type. Postgres and BPM are pinned external releases; BPM is used by
-the blobstore job. The JuiceFS FUSE client runs directly on the VM under Monit,
-not in BPM, so it can access the host `/dev/fuse` and publish its mount normally.
+The base `deployment/juicefs.yml` accepts `juicefs_metadata` and
+`juicefs_storage` objects. `deployment/operations/non-ha-validation.yml` adds
+singleton Postgres/WebDAV, their DNS aliases, and the 50 GiB backend disk.
+Instant-bosh DNS and credential adjustments live in
+`deployment/operations/misc/`; other directors should use their normal DNS and
+credential configuration. The director needs an `ubuntu-noble` stemcell and
+cloud-config with `z1`, the `default` network, and `default` VM type. The JuiceFS
+FUSE client runs directly on the VM under Monit so it can access host `/dev/fuse`.
 The singleton
 WebDAV blobstore keeps objects on its persistent disk at
 `/var/vcap/store/juicefs-blobstore/`. The job provides a `webdav` BOSH link;
@@ -129,38 +128,15 @@ records make failed quota application retryable; completed provision retries do
 not change quotas. Existing pre-quota directories retain the unlimited shared
 plan. The broker's instance metadata lives on JuiceFS, not on its ephemeral disk.
 
-`operations/enable-juicefs-cf-volume-service.yml` enables CAPI volume services
-and defines registration/deregistration errands. The CF deployment is separate
-from the JuiceFS deployment. The JuiceFS deployment marks Postgres, WebDAV, the
-full-root mountpoint, and broker as shared link providers. The CF runtime-config
-consumes the broker mounter's shared `juicefs_mount` link from deployment
-`juicefs`. That mounter is authoritative for `juicefs.metadata` (postgres or
-mysql), `juicefs.storage` (webdav, s3 or wasb/Azure Blob), and the filesystem name. The Diego
-driver is link-only, with no independent backend overrides. Quota
-commands reuse the mounter's rendered configuration. The colocated broker
-consumes its mountpoint link. `operations/use-mysql-s3.yml` shows external
-backend configuration; `operations/use-azure-blob.yml` configures Azure Blob.
-Changing properties does not migrate existing data. The CF control group has
-no full-root mount; only the broker and cell driver mount JuiceFS.
-
-The runtime-config is `runtime-config/juicefs-volume-driver.yml`. It targets the
-live `compute` group for the isolated
-volume driver. Broker registration errands consume the shared broker link. The
-broker provisions
-directories beneath `/cf-volumes/<service-instance-guid>`; each cell mount uses
-JuiceFS `--subdir` for only that directory. Bindings are synchronous and stateless,
-with `bindings_retrievable: false`; binding retrieval is not implemented.
-The optional binding parameter `mount` selects a clean absolute container path
-(for example, `cf bind-service cf-env foobar -c '{"mount":"/data"}'`). Without it,
-the path is `/var/vcap/data/<service-instance-guid>`. Cloud Controller stores the
-returned mount instructions. This view restriction is not strong
-isolation against hostile users who can access shared backend credentials.
-The broker-registration errands currently use the configured CF admin
-credential; replace that with a dedicated account authorized to manage service
-brokers and service access before using this POC beyond a test environment.
+See `README.md#cloud-foundry-volume-service-poc` and
+`docs/acceptance-testing.md` for the volume-service deployment and validation
+details. `scripts/deploy` is the source of truth for the instant-bosh deployment
+sequence. The CF runtime-config consumes the broker mounter's shared
+`juicefs_mount` link; the mounter defines the backend configuration and
+filesystem name. The driver mounts only the service-instance subdirectory.
 
-Build and validate from this release root with `bosh create-release --force`,
-then interpolate the CF manifest with the ops file and the runtime-config with
-the required backend variables. After applying both to the director,
-run `register-juicefs-broker`, then verify CAPI provision/bind and app-container
-read/write/unmount behavior before relying on deprovision cleanup.
+Service bindings are synchronous and stateless; binding retrieval is not
+implemented. The optional `mount` binding parameter selects an absolute container
+path, defaulting to `/var/vcap/data/<service-instance-guid>`. The broker currently
+uses the configured CF administrator credential; replace it with a dedicated
+broker-management account before using this POC beyond a test environment.

README.md

@@ -11,7 +11,9 @@
 A proof-of-concept [JuiceFS](https://juicefs.com/) volume service for Cloud Foundry.
 The release includes a service broker, a Diego volume driver, and a shared
 filesystem with PostgreSQL or MySQL metadata and WebDAV, S3, or Azure Blob object storage.
-The supplied deployment uses PostgreSQL and persistent WebDAV storage.
+The base deployment accepts external metadata and object-storage configuration.
+The `deployment/operations/non-ha-validation.yml` ops file adds singleton
+PostgreSQL and persistent WebDAV backends for validation.
 
 Applications can provision a directory, bind it at a chosen container path, and
 keep files across container restarts. BOSH-configurable service plans apply
@@ -39,7 +41,7 @@ Diego cell → JuiceFS volume driver → subdirectory FUSE mount → app contain
                       └─ WebDAV / S3 / Azure Blob: persistent data blocks
 ```
 
-The `juicefs` deployment has two singleton instance groups:
+With the non-HA validation ops file, `juicefs` has two singleton instance groups:
 
 | Group | Jobs | Storage |
 | --- | --- | --- |
@@ -56,7 +58,7 @@ The CF runtime-config adds the volume driver to `compute`. It consumes the
 broker VM's shared `juicefs_mount` BOSH link for
 the filesystem name and all backend settings, credentials, and trust roots.
 The broker's volume mounter is the authoritative configuration source;
-registration errands run in CF.
+registration errands are colocated on `juicefs-broker` in the JuiceFS deployment.
 
 FUSE clients run directly under Monit so they can use the host `/dev/fuse` and
 mount namespace. In the default WebDAV setup, clients connect directly to
@@ -142,22 +144,42 @@ devbox run deploy
 This command:
 
 1. Sources `bosh.env` and creates and uploads a development release.
-2. Updates the named `juicefs` runtime-config with the built release's explicit
-   version.
-3. Deploys `manifests/juicefs.yml`.
-4. Redeploys CF from its director manifest, applying
-   `operations/enable-juicefs-cf-volume-service.yml`.
-
-The ops file enables CAPI volume services and adds registration/deregistration
-errands. Registration is a separate step, also needed after catalog changes:
+2. Reads CF's system domain and deploys `deployment/juicefs.yml` with
+   `deployment/operations/non-ha-validation.yml` and the instant-bosh DNS
+   workaround `deployment/operations/misc/ibosh-bosh-dns-fix.yml`. The
+   `misc/ibosh-cf-credentials.yml` ops reuses CF credentials in instant-bosh's
+   config server; `scripts/deploy` supplies the system domain from CF's manifest.
+3. Updates the named `juicefs` runtime-config with the built release's explicit
+   version and redeploys CF, applying
+   `deployment/operations/cf/non-ha-validation.yml`,
+   `deployment/operations/cf/enable-volume-services.yml`.
+
+The main JuiceFS manifest colocates `register-broker` and `deregister-broker`
+as errands on the broker VM. Registration is a separate step, also needed after catalog changes:
 
 ```sh
 source bosh.env
-bosh -n -d cf run-errand register-juicefs-broker
+bosh -n -d juicefs run-errand register-broker --instance=juicefs-broker/0
 cf service-brokers
 cf marketplace -e juicefs
 ```
 
+The base `deployment/juicefs.yml` requires `juicefs_metadata` and
+`juicefs_storage` objects in a vars file, or an external-backend ops file.
+It also requires `juicefs_cf_password`, `juicefs_cf_system_domain`, and
+`juicefs_cf_ca_certificates` for registration, unless using the instant-bosh
+credential ops. Registration calls the public `https://api.<system_domain>`
+route without a cross-deployment Cloud Controller link. Use
+`deployment/operations/misc/provide-cf-api-ca.yml` to configure the trusted API CA.
+Supply `juicefs_cf_api_ca_certificate` from a vars file or `-v` when applying it.
+For singleton validation, apply `deployment/operations/non-ha-validation.yml`
+instead; it supplies both objects, Postgres/WebDAV jobs, credentials, certificates,
+and backend DNS aliases. Apply its CF companion
+`deployment/operations/cf/non-ha-validation.yml` to CF.
+For instant-bosh, the misc DNS workaround adds a JuiceFS addon that reuses
+CF's global certificates. For other directors, use the upstream DNS
+runtime-config; see [DNS setup](deployment/runtime-configs/README.md).
+
 The registration errand creates or updates the broker and enables access to its
 plans. It trusts both Cloud Controller's linked CA and the configured router CA
 used by the public authentication endpoint. The supplied ops file uses CF's
@@ -296,19 +318,19 @@ link. CF's volume driver consumes that link directly from deployment
 `juicefs`; they have no independent backend overrides. Broker quota operations
 reuse the mounter's rendered configuration, including its metadata engine.
 
-`operations/use-mysql-s3.yml` provides an example deployment ops file with
-external backend variables. PostgreSQL/WebDAV remains the default. Provision
+`deployment/operations/use-mysql-s3.yml` provides an example deployment ops file with
+external backend variables. PostgreSQL/WebDAV is selected by the non-HA validation ops file. Provision
 the external database, object bucket, and access credentials before initializing
-a new filesystem. The bundled backend jobs are not removed by that example ops
-file. Changing properties is not a data migration: `format --no-update` does
+a new filesystem. External backend ops can be applied directly to the base
+manifest without deploying singleton backend jobs. Changing properties is not a data migration: `format --no-update` does
 not rewrite an existing filesystem's storage settings or copy metadata/objects.
 
 The linked PostgreSQL/WebDAV configuration is validated on the live POC;
 MySQL/S3/Azure Blob support has configuration tests but has not been integration-tested.
 
 ### Azure Blob Storage
 
-Use `operations/use-azure-blob.yml`, or configure the mounter's storage object:
+Use `deployment/operations/use-azure-blob.yml`, or configure the mounter's storage object:
 
 ```yaml
 juicefs:
@@ -357,8 +379,8 @@ under `/var/vcap/sys/log/juicefs-volume/`; broker logs are under
 PostgreSQL metadata and WebDAV blocks share the backend persistent disk (50 GiB
 in the example manifest). Monitor its free space: a full object store can also
 prevent PostgreSQL checkpoints, causing metadata EIO errors across all mounts.
-`operations/expand-backend-disk.yml` expands the example backend disk to 50 GiB
-when applied to the deployed manifest; increase that value for larger workloads.
+`deployment/operations/non-ha-validation.yml` sets the backend persistent disk
+to 50 GiB; increase its `persistent_disk` value for larger workloads.
 Preserve the disk and recover the database before reconnecting affected clients.
 If an app retains a disconnected FUSE mount, stop it and wait for its volume
 unmount to finish before starting it again; an immediate restart can race the

deployment/juicefs.yml

@@ -0,0 +1,87 @@
+---
+name: juicefs
+
+releases:
+- name: juicefs
+  version: latest
+
+stemcells:
+- alias: default
+  os: ubuntu-noble
+  version: latest
+
+update:
+  canaries: 1
+  max_in_flight: 1
+  canary_watch_time: 30000-60000
+  update_watch_time: 30000-60000
+
+instance_groups:
+- name: juicefs-broker
+  migrated_from:
+  - name: juicefs-mount-broker
+    az: z1
+  azs: [z1]
+  instances: 1
+  vm_type: default
+  stemcell: default
+  networks:
+  - name: default
+  jobs:
+  - name: juicefs-volume
+    release: juicefs
+    properties:
+      juicefs:
+        volume:
+          initialize: true
+        metadata: ((juicefs_metadata))
+        storage: ((juicefs_storage))
+    provides:
+      juicefs_mount: {shared: true}
+    consumes:
+      juicefs_volume: nil
+  - name: juicefs-broker
+    release: juicefs
+    provides:
+      juicefs_broker: {shared: true}
+    properties:
+      juicefs:
+        broker:
+          password: ((juicefs-broker-password))
+          plans:
+          - id: 40bf3e21-885a-4c2a-99b4-d4e8186c467e
+            name: shared
+            description: Shared JuiceFS directory without a quota
+            free: true
+            capacity_gib: 0
+            inodes: 0
+          - id: 5d3c1f42-eaa0-4d1b-88ce-b26cf6545cb1
+            name: small
+            description: Shared JuiceFS directory with 1 GiB and 1000 inodes
+            free: true
+            capacity_gib: 1
+            inodes: 1000
+    consumes:
+      juicefs_mount: {from: juicefs_mount}
+  - name: register-broker
+    release: juicefs
+    lifecycle: errand
+    consumes:
+      juicefs_broker: {from: juicefs_broker, ip_addresses: true}
+  - name: deregister-broker
+    release: juicefs
+    lifecycle: errand
+    consumes:
+      juicefs_broker: {from: juicefs_broker, ip_addresses: true}
+  properties:
+    juicefs:
+      cf:
+        username: admin
+        system_domain: ((juicefs_cf_system_domain))
+        password: ((juicefs_cf_password))
+        additional_ca_certificates: ((juicefs_cf_ca_certificates))
+        skip_ssl_validation: false
+
+variables:
+- name: juicefs-broker-password
+  type: password

operations/enable-juicefs-acceptance-tests.yml → deployment/operations/cf/enable-juicefs-acceptance-tests.yml

@@ -1,3 +1,4 @@
+# Apply to the CF deployment to install the acceptance-test errand.
 - type: replace
   path: /instance_groups/name=juicefs-acceptance-tests?
   value:
@@ -11,17 +12,18 @@
     jobs:
     - name: juicefs-acceptance-tests
       release: juicefs
-      consumes:
-        cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint}
       properties:
         juicefs:
           acceptance:
             apps_domain: ((juicefs_acceptance_apps_domain))
           cf:
             username: admin
             password: ((cf_admin_password))
+            system_domain: ((juicefs_acceptance_apps_domain))
             additional_ca_certificates: [((router_ssl.ca))]
             skip_ssl_validation: false
 - type: replace
-  path: /releases/name=juicefs/version
-  value: ((juicefs_release_version))
+  path: /releases/name=juicefs?
+  value:
+    name: juicefs
+    version: ((juicefs_release_version))

deployment/operations/cf/enable-volume-services.yml

@@ -0,0 +1,9 @@
+- type: replace
+  path: /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/cc/volume_services_enabled?
+  value: true
+- type: replace
+  path: /instance_groups/name=control/jobs/name=cloud_controller_worker/properties/cc/volume_services_enabled?
+  value: true
+- type: replace
+  path: /instance_groups/name=control/jobs/name=cloud_controller_clock/properties/cc/volume_services_enabled?
+  value: true

deployment/operations/cf/non-ha-validation.yml

@@ -0,0 +1,13 @@
+# Apply to CF when using the singleton backends in ../non-ha-validation.yml.
+- type: replace
+  path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=blobstore.service.juicefs.internal?
+  value:
+    domain: blobstore.service.juicefs.internal
+    targets:
+    - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh}
+- type: replace
+  path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=postgres.service.juicefs.internal?
+  value:
+    domain: postgres.service.juicefs.internal
+    targets:
+    - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh}

operations/instant-bosh-acceptance.yml → deployment/operations/misc/ibosh-acceptance.yml

@@ -1,13 +1,12 @@
-# Reuse the stemcell and DNS releases installed by ibosh cf deploy.
+# Apply to JuiceFS after ../non-ha-validation.yml. Reuse the CF stemcell and alias release.
 - type: replace
   path: /stemcells/alias=default/version
   value: ((acceptance_stemcell_version))
 - type: replace
-  path: /releases/name=bosh-dns/version
-  value: ((acceptance_dns_version))
-- type: replace
-  path: /releases/name=bosh-dns-aliases/version
-  value: ((acceptance_dns_aliases_version))
+  path: /releases/name=bosh-dns-aliases
+  value:
+    name: bosh-dns-aliases
+    version: ((acceptance_dns_aliases_version))
 - type: replace
   path: /releases/name=juicefs/version
   value: ((juicefs_release_version))