From 45c73d3982a64a3cceff51f7767a6da8bb5af31a Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 15:58:10 +0100 Subject: [PATCH 01/11] Refactor deployment manifests and colocate broker registration errands --- AGENTS.md | 29 ++- README.md | 57 ++++-- deployment/juicefs.yml | 88 +++++++++ .../operations/cf/enable-volume-services.yml | 9 + .../operations/cf/non-ha-validation.yml | 13 ++ .../operations/cf/share-cloud-controller.yml | 4 + .../enable-juicefs-acceptance-tests.yml | 6 +- .../operations}/expand-backend-disk.yml | 0 .../operations}/instant-bosh-acceptance.yml | 11 +- .../operations/misc/ibosh-bosh-dns-fix.yml | 28 +++ .../operations/misc/ibosh-cf-credentials.yml | 7 + deployment/operations/non-ha-validation.yml | 120 ++++++++++++ .../use-azure-blob-managed-identity.yml | 0 .../operations}/use-azure-blob.yml | 0 .../use-azure-managed-identities.yml | 9 +- .../operations}/use-mysql-s3.yml | 0 deployment/runtime-configs/README.md | 15 ++ .../azure-managed-identities.yml | 2 +- .../juicefs-volume-driver.yml | 0 devbox.json | 2 +- docs/acceptance-testing.md | 22 ++- docs/azure-managed-identities.md | 20 +- manifests/juicefs.yml | 174 ------------------ .../enable-juicefs-cf-volume-service.yml | 76 -------- scripts/acceptance/run | 21 ++- 25 files changed, 406 insertions(+), 307 deletions(-) create mode 100644 deployment/juicefs.yml create mode 100644 deployment/operations/cf/enable-volume-services.yml create mode 100644 deployment/operations/cf/non-ha-validation.yml create mode 100644 deployment/operations/cf/share-cloud-controller.yml rename {operations => deployment/operations}/enable-juicefs-acceptance-tests.yml (87%) rename {operations => deployment/operations}/expand-backend-disk.yml (100%) rename {operations => deployment/operations}/instant-bosh-acceptance.yml (54%) create mode 100644 deployment/operations/misc/ibosh-bosh-dns-fix.yml create mode 100644 deployment/operations/misc/ibosh-cf-credentials.yml create mode 100644 deployment/operations/non-ha-validation.yml rename {operations => deployment/operations}/use-azure-blob-managed-identity.yml (100%) rename {operations => deployment/operations}/use-azure-blob.yml (100%) rename {operations => deployment/operations}/use-azure-managed-identities.yml (82%) rename {operations => deployment/operations}/use-mysql-s3.yml (100%) create mode 100644 deployment/runtime-configs/README.md rename operations/azure-managed-identities-runtime-config.yml => deployment/runtime-configs/azure-managed-identities.yml (86%) rename {runtime-config => deployment/runtime-configs}/juicefs-volume-driver.yml (100%) delete mode 100644 manifests/juicefs.yml delete mode 100644 operations/enable-juicefs-cf-volume-service.yml diff --git a/AGENTS.md b/AGENTS.md index 92d6391..68ae962 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,8 +57,16 @@ This sources `bosh.env`, runs `bosh create-release --force`, uploads that releas updates the named `juicefs` runtime-config with `bosh update-config` using the newly built release's explicit version, deploys the JuiceFS release, then runs `bosh deploy -n -d cf <(bosh manifest -d cf)` so pending runtime-config changes -are applied. The JuiceFS manifest contains -`juicefs-backends` and `juicefs-broker` instance groups and uses `version: latest` for the +are applied. `deployment/juicefs.yml` contains the `juicefs-broker` group and accepts +`juicefs_metadata` and `juicefs_storage` objects for external backends. +`deployment/operations/non-ha-validation.yml` adds singleton Postgres/WebDAV, +their variables and DNS aliases, and overrides those backend objects. Apply it +before `instant-bosh-acceptance.yml` or disk-expansion ops. The CF aliases are in +`deployment/operations/cf/non-ha-validation.yml`. Instant-bosh embeds DNS only +in CF; `deployment/operations/misc/ibosh-bosh-dns-fix.yml` adds it to JuiceFS, +reusing global `/dns_*` certificates without declaring variables. Other directors +should use the upstream DNS runtime-config. See `deployment/runtime-configs/README.md`. +The manifest uses `version: latest` for the `juicefs` release. The director must have an `ubuntu-noble` stemcell and cloud-config with `z1`, `default` network and `default` VM type. Postgres and BPM are pinned external releases; BPM is used by @@ -129,8 +137,13 @@ records make failed quota application retryable; completed provision retries do not change quotas. Existing pre-quota directories retain the unlimited shared plan. The broker's instance metadata lives on JuiceFS, not on its ephemeral disk. -`operations/enable-juicefs-cf-volume-service.yml` enables CAPI volume services -and defines registration/deregistration errands. The CF deployment is separate +`deployment/operations/cf/enable-volume-services.yml` enables CAPI volume services. +The main JuiceFS manifest colocates `register-broker` and `deregister-broker` +as errands on `juicefs-broker`. They consume the local broker link and CF's shared +HTTPS endpoint; apply `deployment/operations/cf/share-cloud-controller.yml` +before deploying JuiceFS. `misc/ibosh-cf-credentials.yml` references existing CF +credentials in instant-bosh; other environments supply `juicefs_cf_password` and +`juicefs_cf_ca_certificates`. The CF deployment is separate from the JuiceFS deployment. The JuiceFS deployment marks Postgres, WebDAV, the full-root mountpoint, and broker as shared link providers. The CF runtime-config consumes the broker mounter's shared `juicefs_mount` link from deployment @@ -138,12 +151,12 @@ consumes the broker mounter's shared `juicefs_mount` link from deployment mysql), `juicefs.storage` (webdav, s3 or wasb/Azure Blob), and the filesystem name. The Diego driver is link-only, with no independent backend overrides. Quota commands reuse the mounter's rendered configuration. The colocated broker -consumes its mountpoint link. `operations/use-mysql-s3.yml` shows external -backend configuration; `operations/use-azure-blob.yml` configures Azure Blob. +consumes its mountpoint link. `deployment/operations/use-mysql-s3.yml` shows external +backend configuration; `deployment/operations/use-azure-blob.yml` configures Azure Blob. Changing properties does not migrate existing data. The CF control group has no full-root mount; only the broker and cell driver mount JuiceFS. -The runtime-config is `runtime-config/juicefs-volume-driver.yml`. It targets the +The runtime-config is `deployment/runtime-configs/juicefs-volume-driver.yml`. It targets the live `compute` group for the isolated volume driver. Broker registration errands consume the shared broker link. The broker provisions @@ -162,5 +175,5 @@ brokers and service access before using this POC beyond a test environment. Build and validate from this release root with `bosh create-release --force`, then interpolate the CF manifest with the ops file and the runtime-config with the required backend variables. After applying both to the director, -run `register-juicefs-broker`, then verify CAPI provision/bind and app-container +run `bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0`, then verify CAPI provision/bind and app-container read/write/unmount behavior before relying on deprovision cleanup. diff --git a/README.md b/README.md index d9aaa13..6d9e5e5 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,9 @@ A proof-of-concept [JuiceFS](https://juicefs.com/) volume service for Cloud Foundry. The release includes a service broker, a Diego volume driver, and a shared filesystem with PostgreSQL or MySQL metadata and WebDAV, S3, or Azure Blob object storage. -The supplied deployment uses PostgreSQL and persistent WebDAV storage. +The base deployment accepts external metadata and object-storage configuration. +The `deployment/operations/non-ha-validation.yml` ops file adds singleton +PostgreSQL and persistent WebDAV backends for validation. Applications can provision a directory, bind it at a chosen container path, and keep files across container restarts. BOSH-configurable service plans apply @@ -39,7 +41,7 @@ Diego cell → JuiceFS volume driver → subdirectory FUSE mount → app contain └─ WebDAV / S3 / Azure Blob: persistent data blocks ``` -The `juicefs` deployment has two singleton instance groups: +With the non-HA validation ops file, `juicefs` has two singleton instance groups: | Group | Jobs | Storage | | --- | --- | --- | @@ -56,7 +58,7 @@ The CF runtime-config adds the volume driver to `compute`. It consumes the broker VM's shared `juicefs_mount` BOSH link for the filesystem name and all backend settings, credentials, and trust roots. The broker's volume mounter is the authoritative configuration source; -registration errands run in CF. +registration errands are colocated on `juicefs-broker` in the JuiceFS deployment. FUSE clients run directly under Monit so they can use the host `/dev/fuse` and mount namespace. In the default WebDAV setup, clients connect directly to @@ -142,22 +144,41 @@ devbox run deploy This command: 1. Sources `bosh.env` and creates and uploads a development release. -2. Updates the named `juicefs` runtime-config with the built release's explicit - version. -3. Deploys `manifests/juicefs.yml`. -4. Redeploys CF from its director manifest, applying - `operations/enable-juicefs-cf-volume-service.yml`. - -The ops file enables CAPI volume services and adds registration/deregistration -errands. Registration is a separate step, also needed after catalog changes: +2. Shares CF's HTTPS endpoint link using + `deployment/operations/cf/share-cloud-controller.yml`. +3. Deploys `deployment/juicefs.yml` with + `deployment/operations/non-ha-validation.yml` and the instant-bosh DNS + workaround `deployment/operations/misc/ibosh-bosh-dns-fix.yml`. The + `misc/ibosh-cf-credentials.yml` ops reuses CF credentials in instant-bosh's + config server. +4. Updates the named `juicefs` runtime-config with the built release's explicit + version and redeploys CF, applying + `deployment/operations/cf/non-ha-validation.yml`, + `deployment/operations/cf/enable-volume-services.yml`. + +The main JuiceFS manifest colocates `register-broker` and `deregister-broker` +as errands on the broker VM. Registration is a separate step, also needed after catalog changes: ```sh source bosh.env -bosh -n -d cf run-errand register-juicefs-broker +bosh -n -d juicefs run-errand register-broker --instance=juicefs-broker/0 cf service-brokers cf marketplace -e juicefs ``` +The base `deployment/juicefs.yml` requires `juicefs_metadata` and +`juicefs_storage` objects in a vars file, or an external-backend ops file. +It also requires `juicefs_cf_password` and `juicefs_cf_ca_certificates` for +registration, unless using the instant-bosh credential ops. CF must publish its +shared HTTPS endpoint link before the JuiceFS deployment. +For singleton validation, apply `deployment/operations/non-ha-validation.yml` +instead; it supplies both objects, Postgres/WebDAV jobs, credentials, certificates, +and backend DNS aliases. Apply its CF companion +`deployment/operations/cf/non-ha-validation.yml` to CF. +For instant-bosh, the misc DNS workaround adds a JuiceFS addon that reuses +CF's global certificates. For other directors, use the upstream DNS +runtime-config; see [DNS setup](deployment/runtime-configs/README.md). + The registration errand creates or updates the broker and enables access to its plans. It trusts both Cloud Controller's linked CA and the configured router CA used by the public authentication endpoint. The supplied ops file uses CF's @@ -296,11 +317,11 @@ link. CF's volume driver consumes that link directly from deployment `juicefs`; they have no independent backend overrides. Broker quota operations reuse the mounter's rendered configuration, including its metadata engine. -`operations/use-mysql-s3.yml` provides an example deployment ops file with -external backend variables. PostgreSQL/WebDAV remains the default. Provision +`deployment/operations/use-mysql-s3.yml` provides an example deployment ops file with +external backend variables. PostgreSQL/WebDAV is selected by the non-HA validation ops file. Provision the external database, object bucket, and access credentials before initializing -a new filesystem. The bundled backend jobs are not removed by that example ops -file. Changing properties is not a data migration: `format --no-update` does +a new filesystem. External backend ops can be applied directly to the base +manifest without deploying singleton backend jobs. Changing properties is not a data migration: `format --no-update` does not rewrite an existing filesystem's storage settings or copy metadata/objects. The linked PostgreSQL/WebDAV configuration is validated on the live POC; @@ -308,7 +329,7 @@ MySQL/S3/Azure Blob support has configuration tests but has not been integration ### Azure Blob Storage -Use `operations/use-azure-blob.yml`, or configure the mounter's storage object: +Use `deployment/operations/use-azure-blob.yml`, or configure the mounter's storage object: ```yaml juicefs: @@ -357,7 +378,7 @@ under `/var/vcap/sys/log/juicefs-volume/`; broker logs are under PostgreSQL metadata and WebDAV blocks share the backend persistent disk (50 GiB in the example manifest). Monitor its free space: a full object store can also prevent PostgreSQL checkpoints, causing metadata EIO errors across all mounts. -`operations/expand-backend-disk.yml` expands the example backend disk to 50 GiB +`deployment/operations/expand-backend-disk.yml` expands the example backend disk to 50 GiB when applied to the deployed manifest; increase that value for larger workloads. Preserve the disk and recover the database before reconnecting affected clients. If an app retains a disconnected FUSE mount, stop it and wait for its volume diff --git a/deployment/juicefs.yml b/deployment/juicefs.yml new file mode 100644 index 0000000..1cfa743 --- /dev/null +++ b/deployment/juicefs.yml @@ -0,0 +1,88 @@ +--- +name: juicefs + +releases: +- name: juicefs + version: latest + +stemcells: +- alias: default + os: ubuntu-noble + version: latest + +update: + canaries: 1 + max_in_flight: 1 + canary_watch_time: 30000-60000 + update_watch_time: 30000-60000 + +instance_groups: +- name: juicefs-broker + migrated_from: + - name: juicefs-mount-broker + az: z1 + azs: [z1] + instances: 1 + vm_type: default + stemcell: default + networks: + - name: default + jobs: + - name: juicefs-volume + release: juicefs + properties: + juicefs: + volume: + initialize: true + metadata: ((juicefs_metadata)) + storage: ((juicefs_storage)) + provides: + juicefs_mount: {shared: true} + consumes: + juicefs_volume: nil + - name: juicefs-broker + release: juicefs + provides: + juicefs_broker: {shared: true} + properties: + juicefs: + broker: + password: ((juicefs-broker-password)) + plans: + - id: 40bf3e21-885a-4c2a-99b4-d4e8186c467e + name: shared + description: Shared JuiceFS directory without a quota + free: true + capacity_gib: 0 + inodes: 0 + - id: 5d3c1f42-eaa0-4d1b-88ce-b26cf6545cb1 + name: small + description: Shared JuiceFS directory with 1 GiB and 1000 inodes + free: true + capacity_gib: 1 + inodes: 1000 + consumes: + juicefs_mount: {from: juicefs_mount} + - name: register-broker + release: juicefs + lifecycle: errand + consumes: + juicefs_broker: {from: juicefs_broker, ip_addresses: true} + cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint, deployment: cf} + - name: deregister-broker + release: juicefs + lifecycle: errand + consumes: + juicefs_broker: {from: juicefs_broker, ip_addresses: true} + cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint, deployment: cf} + properties: + juicefs: + cf: + username: admin + password: ((juicefs_cf_password)) + additional_ca_certificates: ((juicefs_cf_ca_certificates)) + skip_ssl_validation: false + +variables: +- name: juicefs-broker-password + type: password diff --git a/deployment/operations/cf/enable-volume-services.yml b/deployment/operations/cf/enable-volume-services.yml new file mode 100644 index 0000000..7241010 --- /dev/null +++ b/deployment/operations/cf/enable-volume-services.yml @@ -0,0 +1,9 @@ +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/cc/volume_services_enabled? + value: true +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_worker/properties/cc/volume_services_enabled? + value: true +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_clock/properties/cc/volume_services_enabled? + value: true diff --git a/deployment/operations/cf/non-ha-validation.yml b/deployment/operations/cf/non-ha-validation.yml new file mode 100644 index 0000000..a957724 --- /dev/null +++ b/deployment/operations/cf/non-ha-validation.yml @@ -0,0 +1,13 @@ +# Apply to CF when using the singleton backends in ../non-ha-validation.yml. +- type: replace + path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=blobstore.service.juicefs.internal? + value: + domain: blobstore.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} +- type: replace + path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=postgres.service.juicefs.internal? + value: + domain: postgres.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} diff --git a/deployment/operations/cf/share-cloud-controller.yml b/deployment/operations/cf/share-cloud-controller.yml new file mode 100644 index 0000000..41d3cdb --- /dev/null +++ b/deployment/operations/cf/share-cloud-controller.yml @@ -0,0 +1,4 @@ +# Registration errands in the JuiceFS deployment consume CF's HTTPS endpoint. +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_ng/provides/cloud_controller_https_endpoint? + value: {as: cloud_controller_https_endpoint, shared: true} diff --git a/operations/enable-juicefs-acceptance-tests.yml b/deployment/operations/enable-juicefs-acceptance-tests.yml similarity index 87% rename from operations/enable-juicefs-acceptance-tests.yml rename to deployment/operations/enable-juicefs-acceptance-tests.yml index b41ebe8..adbc394 100644 --- a/operations/enable-juicefs-acceptance-tests.yml +++ b/deployment/operations/enable-juicefs-acceptance-tests.yml @@ -23,5 +23,7 @@ additional_ca_certificates: [((router_ssl.ca))] skip_ssl_validation: false - type: replace - path: /releases/name=juicefs/version - value: ((juicefs_release_version)) + path: /releases/name=juicefs? + value: + name: juicefs + version: ((juicefs_release_version)) diff --git a/operations/expand-backend-disk.yml b/deployment/operations/expand-backend-disk.yml similarity index 100% rename from operations/expand-backend-disk.yml rename to deployment/operations/expand-backend-disk.yml diff --git a/operations/instant-bosh-acceptance.yml b/deployment/operations/instant-bosh-acceptance.yml similarity index 54% rename from operations/instant-bosh-acceptance.yml rename to deployment/operations/instant-bosh-acceptance.yml index a5d6599..2ab5514 100644 --- a/operations/instant-bosh-acceptance.yml +++ b/deployment/operations/instant-bosh-acceptance.yml @@ -1,13 +1,12 @@ -# Reuse the stemcell and DNS releases installed by ibosh cf deploy. +# Apply after non-ha-validation.yml. Reuse the CF stemcell and alias release. - type: replace path: /stemcells/alias=default/version value: ((acceptance_stemcell_version)) - type: replace - path: /releases/name=bosh-dns/version - value: ((acceptance_dns_version)) -- type: replace - path: /releases/name=bosh-dns-aliases/version - value: ((acceptance_dns_aliases_version)) + path: /releases/name=bosh-dns-aliases + value: + name: bosh-dns-aliases + version: ((acceptance_dns_aliases_version)) - type: replace path: /releases/name=juicefs/version value: ((juicefs_release_version)) diff --git a/deployment/operations/misc/ibosh-bosh-dns-fix.yml b/deployment/operations/misc/ibosh-bosh-dns-fix.yml new file mode 100644 index 0000000..e3d1792 --- /dev/null +++ b/deployment/operations/misc/ibosh-bosh-dns-fix.yml @@ -0,0 +1,28 @@ +# instant-bosh embeds DNS in CF instead of installing a director runtime-config. +# Reuse the global certificates already generated by the CF deployment. +- type: replace + path: /releases/name=bosh-dns? + value: + name: bosh-dns + version: 1.39.34 + url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.39.34 + sha1: 7561733fb62595b0ccf6b73d771902da4bc519be +- type: replace + path: /addons?/name=ibosh-bosh-dns? + value: + name: ibosh-bosh-dns + jobs: + - name: bosh-dns + release: bosh-dns + properties: + configure_systemd_resolved: true + override_nameserver: false + disable_recursors: true + cache: {enabled: true} + api: + client: {tls: ((/dns_api_client_tls))} + server: {tls: ((/dns_api_server_tls))} + health: + enabled: true + client: {tls: ((/dns_healthcheck_client_tls))} + server: {tls: ((/dns_healthcheck_server_tls))} diff --git a/deployment/operations/misc/ibosh-cf-credentials.yml b/deployment/operations/misc/ibosh-cf-credentials.yml new file mode 100644 index 0000000..dc145b5 --- /dev/null +++ b/deployment/operations/misc/ibosh-cf-credentials.yml @@ -0,0 +1,7 @@ +# Reuse CF credentials from instant-bosh's config-server namespace. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/password + value: ((/instant-bosh/cf/cf_admin_password)) +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/additional_ca_certificates + value: [((/instant-bosh/cf/router_ssl.ca))] diff --git a/deployment/operations/non-ha-validation.yml b/deployment/operations/non-ha-validation.yml new file mode 100644 index 0000000..6f7059c --- /dev/null +++ b/deployment/operations/non-ha-validation.yml @@ -0,0 +1,120 @@ +# Disposable validation backends. Each singleton owns a separate persistent disk; +# increasing instances does not create HA or shared object storage. +- type: replace + path: /releases/- + value: + name: postgres + version: '57.0.0' + url: https://bosh.io/d/github.com/cloudfoundry/postgres-release?v=57.0.0 + sha1: sha256:fc5f73f57a6b9e7208e4293a7fb7faf11e4fb8fc8bb79db80d0d97e41ab1ee2a +- type: replace + path: /releases/name=bpm? + value: + name: bpm + version: '1.4.41' + url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 + sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 +- type: replace + path: /releases/- + value: + name: bosh-dns-aliases + version: 0.0.5 + url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-aliases-release?v=0.0.5 + sha1: sha256:a140a3e0aef083c1b356970ddd27fe6a84d7d9272a4dded7c3cb4e55d0e4247e +- type: replace + path: /addons?/name=juicefs-validation-aliases? + value: + name: juicefs-validation-aliases + jobs: + - name: bosh-dns-aliases + release: bosh-dns-aliases + properties: + aliases: + - domain: postgres.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} + - domain: blobstore.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} +- type: replace + path: /instance_groups/0:before + value: + name: juicefs-backends + azs: [z1] + instances: 1 + vm_type: default + stemcell: default + persistent_disk: 51200 + networks: + - name: default + jobs: + - name: postgres + release: postgres + provides: + postgres: {as: postgres, shared: true} + properties: + databases: + databases: + - name: juicefs + roles: + - name: juicefs + password: ((juicefs_postgres_password)) + - name: bpm + release: bpm + - name: blobstore + release: juicefs + provides: + blobstore: {shared: true} + properties: + blobstore: + admin_users: + - username: juicefs + password: ((juicefs_blobstore_password)) + tls: + cert: ((juicefs_blobstore_server_tls.certificate)) + private_key: ((juicefs_blobstore_server_tls.private_key)) + ca: ((juicefs_blobstore_ca.certificate)) + hostname: blobstore.service.juicefs.internal +- type: replace + path: /instance_groups/name=juicefs-broker/jobs/name=juicefs-volume/properties/juicefs/metadata + value: + type: postgres + host: postgres.service.juicefs.internal + port: 5432 + database: juicefs + user: juicefs + password: ((juicefs_postgres_password)) + options: sslmode=disable +- type: replace + path: /instance_groups/name=juicefs-broker/jobs/name=juicefs-volume/properties/juicefs/storage + value: + type: webdav + bucket: https://blobstore.service.juicefs.internal:4443/admin/ + access_key: juicefs + secret_key: ((juicefs_blobstore_password)) + ca_cert: ((juicefs_blobstore_ca.certificate)) +- type: replace + path: /variables/- + value: {name: juicefs_postgres_password, type: password} +- type: replace + path: /variables/- + value: {name: juicefs_blobstore_password, type: password} +- type: replace + path: /variables/- + value: + name: juicefs_blobstore_ca + type: certificate + options: + is_ca: true + common_name: juicefs-blobstore-ca +- type: replace + path: /variables/- + value: + name: juicefs_blobstore_server_tls + type: certificate + update_mode: converge + options: + ca: juicefs_blobstore_ca + common_name: blobstore.service.juicefs.internal + alternative_names: [blobstore.service.juicefs.internal] + extended_key_usage: [server_auth] diff --git a/operations/use-azure-blob-managed-identity.yml b/deployment/operations/use-azure-blob-managed-identity.yml similarity index 100% rename from operations/use-azure-blob-managed-identity.yml rename to deployment/operations/use-azure-blob-managed-identity.yml diff --git a/operations/use-azure-blob.yml b/deployment/operations/use-azure-blob.yml similarity index 100% rename from operations/use-azure-blob.yml rename to deployment/operations/use-azure-blob.yml diff --git a/operations/use-azure-managed-identities.yml b/deployment/operations/use-azure-managed-identities.yml similarity index 82% rename from operations/use-azure-managed-identities.yml rename to deployment/operations/use-azure-managed-identities.yml index 5c00523..7c6c2aa 100644 --- a/operations/use-azure-managed-identities.yml +++ b/deployment/operations/use-azure-managed-identities.yml @@ -1,4 +1,11 @@ -# Apply to manifests/juicefs.yml. Apply the matching runtime-config ops to CF. +# Apply to deployment/juicefs.yml. Apply the matching runtime-config ops to CF. +- type: replace + path: /releases/name=bpm? + value: + name: bpm + version: '1.4.41' + url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 + sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 - type: replace path: /instance_groups/name=juicefs-broker/jobs/- value: diff --git a/operations/use-mysql-s3.yml b/deployment/operations/use-mysql-s3.yml similarity index 100% rename from operations/use-mysql-s3.yml rename to deployment/operations/use-mysql-s3.yml diff --git a/deployment/runtime-configs/README.md b/deployment/runtime-configs/README.md new file mode 100644 index 0000000..36ac6fd --- /dev/null +++ b/deployment/runtime-configs/README.md @@ -0,0 +1,15 @@ +# Runtime configs + +Normally install BOSH DNS using the upstream +[`bosh-deployment/runtime-configs/dns.yml`](https://github.com/cloudfoundry/bosh-deployment/blob/master/runtime-configs/dns.yml). + +The pinned instant-bosh CLI embeds DNS into CF instead of installing it +director-wide. For that environment, apply +`deployment/operations/misc/ibosh-bosh-dns-fix.yml` to JuiceFS. It injects +the DNS addon and references the global `/dns_*` certificates already defined +by the CF deployment. It does not declare or regenerate those variables. +CI and the local Devbox deployment command apply this workaround; CF's own +DNS configuration is left in place. Omit this ops file on directors already +providing BOSH DNS through a runtime-config. + +`juicefs-volume-driver.yml` is the separate named `juicefs` runtime-config. diff --git a/operations/azure-managed-identities-runtime-config.yml b/deployment/runtime-configs/azure-managed-identities.yml similarity index 86% rename from operations/azure-managed-identities-runtime-config.yml rename to deployment/runtime-configs/azure-managed-identities.yml index 538cbf3..0cef7a9 100644 --- a/operations/azure-managed-identities-runtime-config.yml +++ b/deployment/runtime-configs/azure-managed-identities.yml @@ -1,4 +1,4 @@ -# Apply to runtime-config/juicefs-volume-driver.yml; requires BPM on compute. +# Apply to deployment/runtime-configs/juicefs-volume-driver.yml; requires BPM on compute. - type: replace path: /addons/name=juicefs-volume-driver/jobs/- value: diff --git a/runtime-config/juicefs-volume-driver.yml b/deployment/runtime-configs/juicefs-volume-driver.yml similarity index 100% rename from runtime-config/juicefs-volume-driver.yml rename to deployment/runtime-configs/juicefs-volume-driver.yml diff --git a/devbox.json b/devbox.json index 1777925..d1376a6 100644 --- a/devbox.json +++ b/devbox.json @@ -7,7 +7,7 @@ ], "scripts": { "deploy": [ - "bash -euo pipefail -c 'set -a; source ./bosh.env; set +a; bosh create-release --force && bosh upload-release && release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) && bosh update-config --type=runtime --name=juicefs -n runtime-config/juicefs-volume-driver.yml -v juicefs_release_version=\"$(bosh interpolate \"$release_manifest\" --path /version)\" && bosh -n -d juicefs deploy manifests/juicefs.yml && bosh deploy -n -d cf <(bosh manifest -d cf) -o operations/enable-juicefs-cf-volume-service.yml'" + "bash -euo pipefail -c 'set -a; source ./bosh.env; set +a; bosh create-release --force && bosh upload-release && release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) && bosh deploy -n -d cf <(bosh manifest -d cf) -o deployment/operations/cf/share-cloud-controller.yml && bosh -n -d juicefs deploy deployment/juicefs.yml -o deployment/operations/non-ha-validation.yml -o deployment/operations/misc/ibosh-bosh-dns-fix.yml -o deployment/operations/misc/ibosh-cf-credentials.yml && bosh update-config --type=runtime --name=juicefs -n deployment/runtime-configs/juicefs-volume-driver.yml -v juicefs_release_version=\"$(bosh interpolate \"$release_manifest\" --path /version)\" && bosh deploy -n -d cf <(bosh manifest -d cf) -o deployment/operations/cf/non-ha-validation.yml -o deployment/operations/cf/enable-volume-services.yml'" ] } } diff --git a/docs/acceptance-testing.md b/docs/acceptance-testing.md index 593b72b..f08638e 100644 --- a/docs/acceptance-testing.md +++ b/docs/acceptance-testing.md @@ -34,8 +34,16 @@ Tool versions are in `scripts/acceptance/versions.env`. The instant-bosh CLI is built from a pinned commit because published `v0.3` predates `ibosh cf deploy`. The director image is pinned by digest. Update these together and verify a full run. CF release and compiled-stemcell versions come from that CLI's embedded -cf-deployment; the JuiceFS deployment reuses its installed DNS releases and -stemcell. CI uses a 10 GiB backend disk. +cf-deployment; the JuiceFS deployment reuses its installed alias release and +stemcell. CI uses a 10 GiB backend disk. Singleton Postgres/WebDAV and their +DNS aliases are enabled with `deployment/operations/non-ha-validation.yml` +and the CF companion `deployment/operations/cf/non-ha-validation.yml`. + +The pinned instant-bosh CLI embeds DNS only in CF. CI applies +`deployment/operations/misc/ibosh-bosh-dns-fix.yml` to JuiceFS to add a DNS +addon using the global certificates already generated by CF. No DNS variable +definitions or changes to CF's DNS configuration are needed. On other directors, +use the upstream DNS runtime-config instead (see `deployment/runtime-configs/README.md`). Director startup skips the default `latest` stemcell upload, whose registry version-tag discovery can be slow. `ibosh cf deploy` uploads the exact stemcell @@ -61,12 +69,18 @@ The mount directory must already exist; the app never creates a fallback `/data` TLS verification uses the Cloud Controller link CA and the CF router CA. The errand can be enabled independently on the consolidated CF topology: apply -`operations/enable-juicefs-cf-volume-service.yml` followed by -`operations/enable-juicefs-acceptance-tests.yml`, supplying +`deployment/operations/cf/enable-volume-services.yml`, +`deployment/operations/enable-juicefs-acceptance-tests.yml`, supplying `juicefs_release_version` and `juicefs_acceptance_apps_domain`, then run `bosh -d cf run-errand juicefs-acceptance-tests` after registering the broker. Its job properties allow choosing another broker name or plan. +Registration runs in JuiceFS with +`bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0`. +Before deploying JuiceFS, share CF's endpoint with +`deployment/operations/cf/share-cloud-controller.yml`; the main manifest consumes +that shared link and colocates both registration errands on the broker VM. + ## Local execution Install Go 1.25, BOSH CLI, jq, and the pinned instant-bosh CLI; initialize the diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index 74d668b..b8e8079 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -105,9 +105,9 @@ The JuiceFS `wasb` bucket hostname starts with the **container**, not the accoun domains can be configured, but the PostgreSQL proxy's token audience is currently hardcoded for public Azure; sovereign-cloud PostgreSQL requires upstream changes. -`operations/use-azure-managed-identities.yml` adds BPM and the proxy to the +`deployment/operations/use-azure-managed-identities.yml` adds BPM and the proxy to the broker group and replaces the mounter's backend settings. The matching -`operations/azure-managed-identities-runtime-config.yml` adds a proxy to each +`deployment/runtime-configs/azure-managed-identities.yml` adds a proxy to each targeted CF compute VM. **BPM must already be colocated on compute** (as in cf-deployment); add it in your CF manifest if absent. The proxy shares the host network, as required for loopback clients, DNS, and IMDS. @@ -133,7 +133,7 @@ requires `wasb` and an HTTPS bucket. Custom CA bundles must include all roots needed by that process. Public trust is packaged from pinned certifi sources, not loaded from stemcell certificate files. -For Blob only, apply `operations/use-azure-blob-managed-identity.yml` instead; +For Blob only, apply `deployment/operations/use-azure-blob-managed-identity.yml` instead; no proxy or runtime-config modification is needed. For PostgreSQL only, colocate the proxy/BPM and consume its link while retaining your existing storage object. The proxy's properties live under `azure_postgres_auth_proxy`: `host`, @@ -156,13 +156,17 @@ Set `release_version` to the explicit version reported by `create-release`, then ```sh bosh update-config --type=runtime --name=juicefs -n \ - runtime-config/juicefs-volume-driver.yml \ - -o operations/azure-managed-identities-runtime-config.yml \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -o deployment/runtime-configs/azure-managed-identities.yml \ --vars-file azure-vars.yml -v juicefs_release_version="$release_version" -bosh -n -d juicefs deploy manifests/juicefs.yml \ - -o operations/use-azure-managed-identities.yml --vars-file azure-vars.yml +bosh -n -d cf deploy <(bosh -d cf manifest) \ + -o deployment/operations/cf/share-cloud-controller.yml +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -o deployment/operations/use-azure-managed-identities.yml --vars-file azure-vars.yml bosh deploy -n -d cf <(bosh manifest -d cf) \ - -o operations/enable-juicefs-cf-volume-service.yml + -o deployment/operations/cf/enable-volume-services.yml ``` Use these Azure-specific commands on subsequent deployments too. The default diff --git a/manifests/juicefs.yml b/manifests/juicefs.yml deleted file mode 100644 index 0a1fd04..0000000 --- a/manifests/juicefs.yml +++ /dev/null @@ -1,174 +0,0 @@ ---- -name: juicefs - -releases: -- name: juicefs - version: latest -- name: postgres - version: '57.0.0' - url: https://bosh.io/d/github.com/cloudfoundry/postgres-release?v=57.0.0 - sha1: sha256:fc5f73f57a6b9e7208e4293a7fb7faf11e4fb8fc8bb79db80d0d97e41ab1ee2a -- name: bpm - version: '1.4.41' - url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 - sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 -- name: bosh-dns - version: 1.39.30 -- name: bosh-dns-aliases - version: 0.0.4 - -addons: -- name: juicefs-dns - jobs: - - name: bosh-dns - release: bosh-dns - properties: - configure_systemd_resolved: true - override_nameserver: false - disable_recursors: true - cache: {enabled: true} - api: - client: {tls: ((/dns_api_client_tls))} - server: {tls: ((/dns_api_server_tls))} - health: - enabled: true - client: {tls: ((/dns_healthcheck_client_tls))} - server: {tls: ((/dns_healthcheck_server_tls))} - - name: bosh-dns-aliases - release: bosh-dns-aliases - properties: - aliases: - - domain: postgres.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} - - domain: blobstore.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} - -stemcells: -- alias: default - os: ubuntu-noble - version: latest - -update: - canaries: 1 - max_in_flight: 1 - canary_watch_time: 30000-60000 - update_watch_time: 30000-60000 - -instance_groups: -- name: juicefs-backends - azs: [z1] - instances: 1 - vm_type: default - stemcell: default - persistent_disk: 51200 - networks: - - name: default - jobs: - - name: postgres - release: postgres - provides: - postgres: {as: postgres, shared: true} - properties: - databases: - databases: - - name: juicefs - roles: - - name: juicefs - password: ((juicefs_postgres_password)) - - name: bpm - release: bpm - - name: blobstore - release: juicefs - provides: - blobstore: {shared: true} - properties: - blobstore: - admin_users: - - username: juicefs - password: ((juicefs_blobstore_password)) - tls: - cert: ((juicefs_blobstore_server_tls.certificate)) - private_key: ((juicefs_blobstore_server_tls.private_key)) - ca: ((juicefs_blobstore_ca.certificate)) - hostname: blobstore.service.juicefs.internal -- name: juicefs-broker - migrated_from: - - name: juicefs-mount-broker - az: z1 - azs: [z1] - instances: 1 - vm_type: default - stemcell: default - networks: - - name: default - jobs: - - name: juicefs-volume - release: juicefs - properties: - juicefs: - volume: - initialize: true - metadata: - type: postgres - host: postgres.service.juicefs.internal - port: 5432 - database: juicefs - user: juicefs - password: ((juicefs_postgres_password)) - options: sslmode=disable - storage: - type: webdav - bucket: https://blobstore.service.juicefs.internal:4443/admin/ - access_key: juicefs - secret_key: ((juicefs_blobstore_password)) - ca_cert: ((juicefs_blobstore_ca.certificate)) - provides: - juicefs_mount: {shared: true} - consumes: - juicefs_volume: nil - - name: juicefs-broker - release: juicefs - provides: - juicefs_broker: {shared: true} - properties: - juicefs: - broker: - password: ((juicefs-broker-password)) - plans: - - id: 40bf3e21-885a-4c2a-99b4-d4e8186c467e - name: shared - description: Shared JuiceFS directory without a quota - free: true - capacity_gib: 0 - inodes: 0 - - id: 5d3c1f42-eaa0-4d1b-88ce-b26cf6545cb1 - name: small - description: Shared JuiceFS directory with 1 GiB and 1000 inodes - free: true - capacity_gib: 1 - inodes: 1000 - consumes: - juicefs_mount: {from: juicefs_mount} -variables: -- name: juicefs-broker-password - type: password -- name: juicefs_postgres_password - type: password -- name: juicefs_blobstore_password - type: password -- name: juicefs_blobstore_ca - type: certificate - options: - is_ca: true - common_name: juicefs-blobstore-ca -- name: juicefs_blobstore_server_tls - type: certificate - update_mode: converge - options: - ca: juicefs_blobstore_ca - common_name: blobstore.service.juicefs.internal - alternative_names: - - blobstore.service.juicefs.internal - extended_key_usage: [server_auth] diff --git a/operations/enable-juicefs-cf-volume-service.yml b/operations/enable-juicefs-cf-volume-service.yml deleted file mode 100644 index 4f7beed..0000000 --- a/operations/enable-juicefs-cf-volume-service.yml +++ /dev/null @@ -1,76 +0,0 @@ -- type: replace - path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=blobstore.service.juicefs.internal? - value: - domain: blobstore.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=postgres.service.juicefs.internal? - value: - domain: postgres.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_worker/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_clock/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /instance_groups/name=register-juicefs-broker? - value: - azs: [z1] - instances: 1 - jobs: - - name: register-broker - release: juicefs - consumes: - juicefs_broker: {from: juicefs_broker, deployment: juicefs, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} - lifecycle: errand - name: register-juicefs-broker - networks: [{name: default}] - stemcell: default - vm_type: default - properties: - juicefs: - broker: - name: juicefs - cf: - username: admin - password: ((cf_admin_password)) - additional_ca_certificates: [((router_ssl.ca))] - skip_ssl_validation: false -- type: replace - path: /instance_groups/name=deregister-juicefs-broker? - value: - azs: [z1] - instances: 1 - jobs: - - name: deregister-broker - release: juicefs - consumes: - juicefs_broker: {from: juicefs_broker, deployment: juicefs, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} - lifecycle: errand - name: deregister-juicefs-broker - networks: [{name: default}] - stemcell: default - vm_type: default - properties: - juicefs: - broker: - name: juicefs - cf: - username: admin - password: ((cf_admin_password)) - additional_ca_certificates: [((router_ssl.ca))] - skip_ssl_validation: false -- type: replace - path: /releases/name=juicefs? - value: - name: juicefs - version: latest diff --git a/scripts/acceptance/run b/scripts/acceptance/run index aa3bcdc..372111e 100755 --- a/scripts/acceptance/run +++ b/scripts/acceptance/run @@ -59,22 +59,27 @@ case "${1:-}" in bosh -d cf manifest > "$work/cf.yml" domain=$(bosh interpolate "$work/cf.yml" --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) stemcell=$(bosh interpolate "$work/cf.yml" --path /stemcells/alias=default/version) - dns=$(bosh interpolate "$work/cf.yml" --path /releases/name=bosh-dns/version) aliases=$(bosh interpolate "$work/cf.yml" --path /releases/name=bosh-dns-aliases/version) - bosh -d juicefs deploy manifests/juicefs.yml \ - -o operations/instant-bosh-acceptance.yml \ + # Publish CF's endpoint before JuiceFS resolves its cross-deployment link. + bosh -d cf deploy "$work/cf.yml" -o deployment/operations/cf/share-cloud-controller.yml + bosh -d cf manifest > "$work/cf.yml" + bosh -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/non-ha-validation.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -o deployment/operations/instant-bosh-acceptance.yml \ -v juicefs_release_version="$version" \ -v acceptance_stemcell_version="$stemcell" \ - -v acceptance_dns_version="$dns" \ -v acceptance_dns_aliases_version="$aliases" - bosh update-config --type=runtime --name=juicefs runtime-config/juicefs-volume-driver.yml \ + bosh update-config --type=runtime --name=juicefs deployment/runtime-configs/juicefs-volume-driver.yml \ -v juicefs_release_version="$version" bosh -d cf deploy "$work/cf.yml" \ - -o operations/enable-juicefs-cf-volume-service.yml \ - -o operations/enable-juicefs-acceptance-tests.yml \ + -o deployment/operations/cf/non-ha-validation.yml \ + -o deployment/operations/cf/enable-volume-services.yml \ + -o deployment/operations/enable-juicefs-acceptance-tests.yml \ -v juicefs_release_version="$version" \ -v juicefs_acceptance_apps_domain="$domain" - bosh -d cf run-errand register-juicefs-broker --download-logs --logs-dir="$artifacts" + bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0 --download-logs --logs-dir="$artifacts" ;; test) load_env From 8c54c9bb4fa980a87aab221b30f6cc7b2c80c254 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 16:03:01 +0100 Subject: [PATCH 02/11] Organize Azure runtime config ops under operations --- .../use-azure-managed-identities.yml} | 0 docs/azure-managed-identities.md | 4 ++-- 2 files changed, 2 insertions(+), 2 deletions(-) rename deployment/runtime-configs/{azure-managed-identities.yml => operations/use-azure-managed-identities.yml} (100%) diff --git a/deployment/runtime-configs/azure-managed-identities.yml b/deployment/runtime-configs/operations/use-azure-managed-identities.yml similarity index 100% rename from deployment/runtime-configs/azure-managed-identities.yml rename to deployment/runtime-configs/operations/use-azure-managed-identities.yml diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index b8e8079..874c57c 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -107,7 +107,7 @@ hardcoded for public Azure; sovereign-cloud PostgreSQL requires upstream changes `deployment/operations/use-azure-managed-identities.yml` adds BPM and the proxy to the broker group and replaces the mounter's backend settings. The matching -`deployment/runtime-configs/azure-managed-identities.yml` adds a proxy to each +`deployment/runtime-configs/operations/use-azure-managed-identities.yml` adds a proxy to each targeted CF compute VM. **BPM must already be colocated on compute** (as in cf-deployment); add it in your CF manifest if absent. The proxy shares the host network, as required for loopback clients, DNS, and IMDS. @@ -157,7 +157,7 @@ Set `release_version` to the explicit version reported by `create-release`, then ```sh bosh update-config --type=runtime --name=juicefs -n \ deployment/runtime-configs/juicefs-volume-driver.yml \ - -o deployment/runtime-configs/azure-managed-identities.yml \ + -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ --vars-file azure-vars.yml -v juicefs_release_version="$release_version" bosh -n -d cf deploy <(bosh -d cf manifest) \ -o deployment/operations/cf/share-cloud-controller.yml From 1c56b5bd10db446ed7e82d0012ef7798e068f228 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 16:04:49 +0100 Subject: [PATCH 03/11] Document volume driver runtime config before development DNS workaround --- deployment/runtime-configs/README.md | 52 ++++++++++++++++++++++------ 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/deployment/runtime-configs/README.md b/deployment/runtime-configs/README.md index 36ac6fd..f7db8b3 100644 --- a/deployment/runtime-configs/README.md +++ b/deployment/runtime-configs/README.md @@ -1,15 +1,45 @@ # Runtime configs -Normally install BOSH DNS using the upstream -[`bosh-deployment/runtime-configs/dns.yml`](https://github.com/cloudfoundry/bosh-deployment/blob/master/runtime-configs/dns.yml). +## JuiceFS volume driver + +[`juicefs-volume-driver.yml`](juicefs-volume-driver.yml) installs the JuiceFS +volume driver on Cloud Foundry Diego cells. It adds the `juicefs-volume-driver` +job to the `compute` instance group in deployment `cf`, enabling cells to mount +the service-instance directories provisioned by the JuiceFS broker. Adjust the +addon's `include` filters if your CF deployment or cell group has another name. + +The driver consumes the shared `juicefs_mount` link from deployment `juicefs`. +That link supplies the filesystem name, backend configuration, credentials, and +trust roots; the cell driver does not need independent backend settings. Deploy +JuiceFS first so this link is available. + +Apply the runtime-config with the explicit uploaded JuiceFS release version: + +```sh +bosh update-config --type=runtime --name=juicefs -n \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -v juicefs_release_version="$release_version" +bosh deploy -n -d cf <(bosh manifest -d cf) +``` -The pinned instant-bosh CLI embeds DNS into CF instead of installing it -director-wide. For that environment, apply -`deployment/operations/misc/ibosh-bosh-dns-fix.yml` to JuiceFS. It injects -the DNS addon and references the global `/dns_*` certificates already defined -by the CF deployment. It does not declare or regenerate those variables. -CI and the local Devbox deployment command apply this workaround; CF's own -DNS configuration is left in place. Omit this ops file on directors already -providing BOSH DNS through a runtime-config. +Updating the runtime-config records the configuration on the director; +redeploying CF installs or updates the addon jobs on the cells. + +### Azure managed identities + +Apply [`operations/use-azure-managed-identities.yml`](operations/use-azure-managed-identities.yml) +to the runtime-config to colocate the Azure PostgreSQL authentication proxy on +each targeted cell and connect the driver's local proxy link. BPM must already +be present on those cells. See [Azure managed identities](../../docs/azure-managed-identities.md) +for backend configuration, required variables, and deployment commands. + +## BOSH DNS prerequisite + +Install BOSH DNS through the upstream +[`bosh-deployment/runtime-configs/dns.yml`](https://github.com/cloudfoundry/bosh-deployment/blob/master/runtime-configs/dns.yml). +This is separate from the JuiceFS volume-driver runtime-config. -`juicefs-volume-driver.yml` is the separate named `juicefs` runtime-config. +For instant-bosh development environments, DNS is currently embedded only in CF. +Apply [`../operations/misc/ibosh-bosh-dns-fix.yml`](../operations/misc/ibosh-bosh-dns-fix.yml) +to JuiceFS to reuse CF's global DNS certificates. Omit this workaround when the +director already supplies DNS through a runtime-config. From fa6abdb8dd48fcfb9855fdb0daabc6b072426c82 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 16:09:30 +0100 Subject: [PATCH 04/11] Keep validation backend disk size in non-HA ops --- AGENTS.md | 3 ++- README.md | 4 ++-- deployment/operations/expand-backend-disk.yml | 3 --- 3 files changed, 4 insertions(+), 6 deletions(-) delete mode 100644 deployment/operations/expand-backend-disk.yml diff --git a/AGENTS.md b/AGENTS.md index 68ae962..aae00af 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,8 @@ are applied. `deployment/juicefs.yml` contains the `juicefs-broker` group and ac `juicefs_metadata` and `juicefs_storage` objects for external backends. `deployment/operations/non-ha-validation.yml` adds singleton Postgres/WebDAV, their variables and DNS aliases, and overrides those backend objects. Apply it -before `instant-bosh-acceptance.yml` or disk-expansion ops. The CF aliases are in +before `instant-bosh-acceptance.yml`. Its backend persistent disk is 50 GiB. +The CF aliases are in `deployment/operations/cf/non-ha-validation.yml`. Instant-bosh embeds DNS only in CF; `deployment/operations/misc/ibosh-bosh-dns-fix.yml` adds it to JuiceFS, reusing global `/dns_*` certificates without declaring variables. Other directors diff --git a/README.md b/README.md index 6d9e5e5..42b8da7 100644 --- a/README.md +++ b/README.md @@ -378,8 +378,8 @@ under `/var/vcap/sys/log/juicefs-volume/`; broker logs are under PostgreSQL metadata and WebDAV blocks share the backend persistent disk (50 GiB in the example manifest). Monitor its free space: a full object store can also prevent PostgreSQL checkpoints, causing metadata EIO errors across all mounts. -`deployment/operations/expand-backend-disk.yml` expands the example backend disk to 50 GiB -when applied to the deployed manifest; increase that value for larger workloads. +`deployment/operations/non-ha-validation.yml` sets the backend persistent disk +to 50 GiB; increase its `persistent_disk` value for larger workloads. Preserve the disk and recover the database before reconnecting affected clients. If an app retains a disconnected FUSE mount, stop it and wait for its volume unmount to finish before starting it again; an immediate restart can race the diff --git a/deployment/operations/expand-backend-disk.yml b/deployment/operations/expand-backend-disk.yml deleted file mode 100644 index f913aca..0000000 --- a/deployment/operations/expand-backend-disk.yml +++ /dev/null @@ -1,3 +0,0 @@ -- type: replace - path: /instance_groups/name=juicefs-backends/persistent_disk - value: 51200 From 18d1f6f8ac1078807c36d693d5547218203c4e40 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 16:12:10 +0100 Subject: [PATCH 05/11] Group acceptance ops by target deployment --- AGENTS.md | 2 +- .../operations/{ => cf}/enable-juicefs-acceptance-tests.yml | 1 + .../ibosh-acceptance.yml} | 2 +- docs/acceptance-testing.md | 2 +- scripts/acceptance/run | 4 ++-- 5 files changed, 6 insertions(+), 5 deletions(-) rename deployment/operations/{ => cf}/enable-juicefs-acceptance-tests.yml (92%) rename deployment/operations/{instant-bosh-acceptance.yml => misc/ibosh-acceptance.yml} (82%) diff --git a/AGENTS.md b/AGENTS.md index aae00af..8bbde67 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,7 @@ are applied. `deployment/juicefs.yml` contains the `juicefs-broker` group and ac `juicefs_metadata` and `juicefs_storage` objects for external backends. `deployment/operations/non-ha-validation.yml` adds singleton Postgres/WebDAV, their variables and DNS aliases, and overrides those backend objects. Apply it -before `instant-bosh-acceptance.yml`. Its backend persistent disk is 50 GiB. +before `misc/ibosh-acceptance.yml`. Its backend persistent disk is 50 GiB. The CF aliases are in `deployment/operations/cf/non-ha-validation.yml`. Instant-bosh embeds DNS only in CF; `deployment/operations/misc/ibosh-bosh-dns-fix.yml` adds it to JuiceFS, diff --git a/deployment/operations/enable-juicefs-acceptance-tests.yml b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml similarity index 92% rename from deployment/operations/enable-juicefs-acceptance-tests.yml rename to deployment/operations/cf/enable-juicefs-acceptance-tests.yml index adbc394..036e8dc 100644 --- a/deployment/operations/enable-juicefs-acceptance-tests.yml +++ b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml @@ -1,3 +1,4 @@ +# Apply to the CF deployment to install the acceptance-test errand. - type: replace path: /instance_groups/name=juicefs-acceptance-tests? value: diff --git a/deployment/operations/instant-bosh-acceptance.yml b/deployment/operations/misc/ibosh-acceptance.yml similarity index 82% rename from deployment/operations/instant-bosh-acceptance.yml rename to deployment/operations/misc/ibosh-acceptance.yml index 2ab5514..df75672 100644 --- a/deployment/operations/instant-bosh-acceptance.yml +++ b/deployment/operations/misc/ibosh-acceptance.yml @@ -1,4 +1,4 @@ -# Apply after non-ha-validation.yml. Reuse the CF stemcell and alias release. +# Apply to JuiceFS after ../non-ha-validation.yml. Reuse the CF stemcell and alias release. - type: replace path: /stemcells/alias=default/version value: ((acceptance_stemcell_version)) diff --git a/docs/acceptance-testing.md b/docs/acceptance-testing.md index f08638e..def0673 100644 --- a/docs/acceptance-testing.md +++ b/docs/acceptance-testing.md @@ -70,7 +70,7 @@ TLS verification uses the Cloud Controller link CA and the CF router CA. The errand can be enabled independently on the consolidated CF topology: apply `deployment/operations/cf/enable-volume-services.yml`, -`deployment/operations/enable-juicefs-acceptance-tests.yml`, supplying +`deployment/operations/cf/enable-juicefs-acceptance-tests.yml`, supplying `juicefs_release_version` and `juicefs_acceptance_apps_domain`, then run `bosh -d cf run-errand juicefs-acceptance-tests` after registering the broker. Its job properties allow choosing another broker name or plan. diff --git a/scripts/acceptance/run b/scripts/acceptance/run index 372111e..5df7e78 100755 --- a/scripts/acceptance/run +++ b/scripts/acceptance/run @@ -67,7 +67,7 @@ case "${1:-}" in -o deployment/operations/non-ha-validation.yml \ -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ -o deployment/operations/misc/ibosh-cf-credentials.yml \ - -o deployment/operations/instant-bosh-acceptance.yml \ + -o deployment/operations/misc/ibosh-acceptance.yml \ -v juicefs_release_version="$version" \ -v acceptance_stemcell_version="$stemcell" \ -v acceptance_dns_aliases_version="$aliases" @@ -76,7 +76,7 @@ case "${1:-}" in bosh -d cf deploy "$work/cf.yml" \ -o deployment/operations/cf/non-ha-validation.yml \ -o deployment/operations/cf/enable-volume-services.yml \ - -o deployment/operations/enable-juicefs-acceptance-tests.yml \ + -o deployment/operations/cf/enable-juicefs-acceptance-tests.yml \ -v juicefs_release_version="$version" \ -v juicefs_acceptance_apps_domain="$domain" bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0 --download-logs --logs-dir="$artifacts" From 0188f6bcec06b536f45b229fdb9b323fde966692 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 17:05:46 +0100 Subject: [PATCH 06/11] Use CF public API route for colocated registration errands --- AGENTS.md | 11 ++++--- README.md | 14 ++++---- deployment/juicefs.yml | 3 +- .../cf/enable-juicefs-acceptance-tests.yml | 3 +- .../operations/cf/share-cloud-controller.yml | 4 --- .../operations/misc/ibosh-cf-credentials.yml | 3 ++ .../operations/misc/provide-cf-api-ca.yml | 10 ++++++ .../misc/skip-cf-api-tls-validation.yml | 4 +++ devbox.json | 2 +- docs/acceptance-testing.md | 9 ++++-- docs/azure-managed-identities.md | 6 ++-- jobs/deregister-broker/spec | 4 +-- .../templates/config.json.erb | 5 ++- jobs/juicefs-acceptance-tests/spec | 4 +-- .../templates/config.json.erb | 5 ++- jobs/register-broker/spec | 4 +-- .../register-broker/templates/config.json.erb | 5 ++- scripts/acceptance/run | 5 ++- scripts/deploy | 32 +++++++++++++++++++ tests/render_templates.rb | 32 +++++++++++-------- 20 files changed, 107 insertions(+), 58 deletions(-) delete mode 100644 deployment/operations/cf/share-cloud-controller.yml create mode 100644 deployment/operations/misc/provide-cf-api-ca.yml create mode 100644 deployment/operations/misc/skip-cf-api-tls-validation.yml create mode 100755 scripts/deploy diff --git a/AGENTS.md b/AGENTS.md index 8bbde67..3293891 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -141,10 +141,13 @@ plan. The broker's instance metadata lives on JuiceFS, not on its ephemeral disk `deployment/operations/cf/enable-volume-services.yml` enables CAPI volume services. The main JuiceFS manifest colocates `register-broker` and `deregister-broker` as errands on `juicefs-broker`. They consume the local broker link and CF's shared -HTTPS endpoint; apply `deployment/operations/cf/share-cloud-controller.yml` -before deploying JuiceFS. `misc/ibosh-cf-credentials.yml` references existing CF -credentials in instant-bosh; other environments supply `juicefs_cf_password` and -`juicefs_cf_ca_certificates`. The CF deployment is separate +API system domain, constructing `https://api.` without a +cross-deployment endpoint link. `misc/ibosh-cf-credentials.yml` references the +existing CF domain and credentials in instant-bosh; other environments supply +`juicefs_cf_password`, `juicefs_cf_system_domain`, and +`juicefs_cf_ca_certificates`. `misc/provide-cf-api-ca.yml` configures the trusted +API CA; TLS verification can be skipped for disposable tests with +`misc/skip-cf-api-tls-validation.yml`. The CF deployment is separate from the JuiceFS deployment. The JuiceFS deployment marks Postgres, WebDAV, the full-root mountpoint, and broker as shared link providers. The CF runtime-config consumes the broker mounter's shared `juicefs_mount` link from deployment diff --git a/README.md b/README.md index 42b8da7..c300583 100644 --- a/README.md +++ b/README.md @@ -144,14 +144,12 @@ devbox run deploy This command: 1. Sources `bosh.env` and creates and uploads a development release. -2. Shares CF's HTTPS endpoint link using - `deployment/operations/cf/share-cloud-controller.yml`. -3. Deploys `deployment/juicefs.yml` with +2. Reads CF's system domain and deploys `deployment/juicefs.yml` with `deployment/operations/non-ha-validation.yml` and the instant-bosh DNS workaround `deployment/operations/misc/ibosh-bosh-dns-fix.yml`. The `misc/ibosh-cf-credentials.yml` ops reuses CF credentials in instant-bosh's config server. -4. Updates the named `juicefs` runtime-config with the built release's explicit +3. Updates the named `juicefs` runtime-config with the built release's explicit version and redeploys CF, applying `deployment/operations/cf/non-ha-validation.yml`, `deployment/operations/cf/enable-volume-services.yml`. @@ -168,9 +166,11 @@ cf marketplace -e juicefs The base `deployment/juicefs.yml` requires `juicefs_metadata` and `juicefs_storage` objects in a vars file, or an external-backend ops file. -It also requires `juicefs_cf_password` and `juicefs_cf_ca_certificates` for -registration, unless using the instant-bosh credential ops. CF must publish its -shared HTTPS endpoint link before the JuiceFS deployment. +It also requires `juicefs_cf_password`, `juicefs_cf_system_domain`, and +`juicefs_cf_ca_certificates` for registration, unless using the instant-bosh +credential ops. Registration calls the public `https://api.` +route without a cross-deployment Cloud Controller link. Use +`deployment/operations/misc/provide-cf-api-ca.yml` to configure the trusted API CA. For singleton validation, apply `deployment/operations/non-ha-validation.yml` instead; it supplies both objects, Postgres/WebDAV jobs, credentials, certificates, and backend DNS aliases. Apply its CF companion diff --git a/deployment/juicefs.yml b/deployment/juicefs.yml index 1cfa743..a480ea4 100644 --- a/deployment/juicefs.yml +++ b/deployment/juicefs.yml @@ -68,17 +68,16 @@ instance_groups: lifecycle: errand consumes: juicefs_broker: {from: juicefs_broker, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint, deployment: cf} - name: deregister-broker release: juicefs lifecycle: errand consumes: juicefs_broker: {from: juicefs_broker, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint, deployment: cf} properties: juicefs: cf: username: admin + system_domain: ((juicefs_cf_system_domain)) password: ((juicefs_cf_password)) additional_ca_certificates: ((juicefs_cf_ca_certificates)) skip_ssl_validation: false diff --git a/deployment/operations/cf/enable-juicefs-acceptance-tests.yml b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml index 036e8dc..14d7be4 100644 --- a/deployment/operations/cf/enable-juicefs-acceptance-tests.yml +++ b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml @@ -12,8 +12,6 @@ jobs: - name: juicefs-acceptance-tests release: juicefs - consumes: - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} properties: juicefs: acceptance: @@ -21,6 +19,7 @@ cf: username: admin password: ((cf_admin_password)) + system_domain: ((juicefs_acceptance_apps_domain)) additional_ca_certificates: [((router_ssl.ca))] skip_ssl_validation: false - type: replace diff --git a/deployment/operations/cf/share-cloud-controller.yml b/deployment/operations/cf/share-cloud-controller.yml deleted file mode 100644 index 41d3cdb..0000000 --- a/deployment/operations/cf/share-cloud-controller.yml +++ /dev/null @@ -1,4 +0,0 @@ -# Registration errands in the JuiceFS deployment consume CF's HTTPS endpoint. -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_ng/provides/cloud_controller_https_endpoint? - value: {as: cloud_controller_https_endpoint, shared: true} diff --git a/deployment/operations/misc/ibosh-cf-credentials.yml b/deployment/operations/misc/ibosh-cf-credentials.yml index dc145b5..87695eb 100644 --- a/deployment/operations/misc/ibosh-cf-credentials.yml +++ b/deployment/operations/misc/ibosh-cf-credentials.yml @@ -1,4 +1,7 @@ # Reuse CF credentials from instant-bosh's config-server namespace. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/system_domain + value: ((/instant-bosh/cf/system_domain)) - type: replace path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/password value: ((/instant-bosh/cf/cf_admin_password)) diff --git a/deployment/operations/misc/provide-cf-api-ca.yml b/deployment/operations/misc/provide-cf-api-ca.yml new file mode 100644 index 0000000..d1b59e0 --- /dev/null +++ b/deployment/operations/misc/provide-cf-api-ca.yml @@ -0,0 +1,10 @@ +# Supply the CA certificate that signs CF's public API route certificate. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/additional_ca_certificates + value: + - ((juicefs_cf_api_ca_certificate)) +- type: replace + path: /variables/- + value: + name: juicefs_cf_api_ca_certificate + type: value diff --git a/deployment/operations/misc/skip-cf-api-tls-validation.yml b/deployment/operations/misc/skip-cf-api-tls-validation.yml new file mode 100644 index 0000000..d91e3c2 --- /dev/null +++ b/deployment/operations/misc/skip-cf-api-tls-validation.yml @@ -0,0 +1,4 @@ +# For disposable test environments only. Prefer a trusted CA bundle. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/skip_ssl_validation + value: true diff --git a/devbox.json b/devbox.json index d1376a6..cc1fc9c 100644 --- a/devbox.json +++ b/devbox.json @@ -7,7 +7,7 @@ ], "scripts": { "deploy": [ - "bash -euo pipefail -c 'set -a; source ./bosh.env; set +a; bosh create-release --force && bosh upload-release && release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) && bosh deploy -n -d cf <(bosh manifest -d cf) -o deployment/operations/cf/share-cloud-controller.yml && bosh -n -d juicefs deploy deployment/juicefs.yml -o deployment/operations/non-ha-validation.yml -o deployment/operations/misc/ibosh-bosh-dns-fix.yml -o deployment/operations/misc/ibosh-cf-credentials.yml && bosh update-config --type=runtime --name=juicefs -n deployment/runtime-configs/juicefs-volume-driver.yml -v juicefs_release_version=\"$(bosh interpolate \"$release_manifest\" --path /version)\" && bosh deploy -n -d cf <(bosh manifest -d cf) -o deployment/operations/cf/non-ha-validation.yml -o deployment/operations/cf/enable-volume-services.yml'" + "bash scripts/deploy" ] } } diff --git a/docs/acceptance-testing.md b/docs/acceptance-testing.md index def0673..f3995f6 100644 --- a/docs/acceptance-testing.md +++ b/docs/acceptance-testing.md @@ -77,9 +77,12 @@ Its job properties allow choosing another broker name or plan. Registration runs in JuiceFS with `bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0`. -Before deploying JuiceFS, share CF's endpoint with -`deployment/operations/cf/share-cloud-controller.yml`; the main manifest consumes -that shared link and colocates both registration errands on the broker VM. +Registration calls `https://api.`, the public CF API route, and +does not consume a cross-deployment endpoint link. The main manifest colocates +both registration errands on the broker VM. CI sets the system domain from the +CF manifest and skips API TLS verification only for this disposable test +environment; other deployments can use +`deployment/operations/misc/provide-cf-api-ca.yml` to configure the public API CA. ## Local execution diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index 874c57c..bd1f542 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -159,12 +159,12 @@ bosh update-config --type=runtime --name=juicefs -n \ deployment/runtime-configs/juicefs-volume-driver.yml \ -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ --vars-file azure-vars.yml -v juicefs_release_version="$release_version" -bosh -n -d cf deploy <(bosh -d cf manifest) \ - -o deployment/operations/cf/share-cloud-controller.yml +system_domain=$(bosh -d cf manifest | bosh interpolate - --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) bosh -n -d juicefs deploy deployment/juicefs.yml \ -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ -o deployment/operations/misc/ibosh-cf-credentials.yml \ - -o deployment/operations/use-azure-managed-identities.yml --vars-file azure-vars.yml + -o deployment/operations/use-azure-managed-identities.yml \ + -v juicefs_cf_system_domain="$system_domain" --vars-file azure-vars.yml bosh deploy -n -d cf <(bosh manifest -d cf) \ -o deployment/operations/cf/enable-volume-services.yml ``` diff --git a/jobs/deregister-broker/spec b/jobs/deregister-broker/spec index 9a79f35..688f33e 100644 --- a/jobs/deregister-broker/spec +++ b/jobs/deregister-broker/spec @@ -14,8 +14,6 @@ consumes: - juicefs.broker.port - juicefs.broker.username - juicefs.broker.password -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.broker.name: description: Service broker registration name. @@ -26,6 +24,8 @@ properties: juicefs.cf.password: description: Cloud Controller administrator password. default: "" + juicefs.cf.system_domain: + description: Cloud Foundry system domain used to construct the public API URL. juicefs.cf.skip_ssl_validation: description: Skip Cloud Controller TLS validation. default: false diff --git a/jobs/deregister-broker/templates/config.json.erb b/jobs/deregister-broker/templates/config.json.erb index ffcf5ce..3736dc5 100644 --- a/jobs/deregister-broker/templates/config.json.erb +++ b/jobs/deregister-broker/templates/config.json.erb @@ -1,9 +1,8 @@ <% require 'json' - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), brokerName: p('juicefs.broker.name'), brokerURL: "http://#{link('juicefs_broker').instances.first.address}:#{link('juicefs_broker').p('juicefs.broker.port')}", brokerPort: link('juicefs_broker').p('juicefs.broker.port'), diff --git a/jobs/juicefs-acceptance-tests/spec b/jobs/juicefs-acceptance-tests/spec index 90a48b8..a6bcd84 100644 --- a/jobs/juicefs-acceptance-tests/spec +++ b/jobs/juicefs-acceptance-tests/spec @@ -8,8 +8,6 @@ packages: - juicefs-acceptance-tests - ca-certificates consumes: -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.acceptance.apps_domain: description: Shared CF application domain used for the test route. @@ -24,6 +22,8 @@ properties: default: admin juicefs.cf.password: description: CF administrator password. + juicefs.cf.system_domain: + description: CF system domain used for the public API and app route. juicefs.cf.additional_ca_certificates: description: Additional CA roots for UAA and application routes. default: [] diff --git a/jobs/juicefs-acceptance-tests/templates/config.json.erb b/jobs/juicefs-acceptance-tests/templates/config.json.erb index bcfc2c2..9b2e9a6 100644 --- a/jobs/juicefs-acceptance-tests/templates/config.json.erb +++ b/jobs/juicefs-acceptance-tests/templates/config.json.erb @@ -1,11 +1,10 @@ <% require 'json' - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", username: p('juicefs.cf.username'), password: p('juicefs.cf.password'), - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), appsDomain: p('juicefs.acceptance.apps_domain'), brokerName: p('juicefs.acceptance.broker_name'), planName: p('juicefs.acceptance.plan_name'), diff --git a/jobs/register-broker/spec b/jobs/register-broker/spec index 1236528..8977046 100644 --- a/jobs/register-broker/spec +++ b/jobs/register-broker/spec @@ -14,8 +14,6 @@ consumes: - juicefs.broker.port - juicefs.broker.username - juicefs.broker.password -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.broker.name: description: Service broker registration name. @@ -26,6 +24,8 @@ properties: juicefs.cf.password: description: Cloud Controller administrator password. default: "" + juicefs.cf.system_domain: + description: Cloud Foundry system domain used to construct the public API URL. juicefs.cf.skip_ssl_validation: description: Skip Cloud Controller TLS validation. default: false diff --git a/jobs/register-broker/templates/config.json.erb b/jobs/register-broker/templates/config.json.erb index 7bbbb96..47f04a8 100644 --- a/jobs/register-broker/templates/config.json.erb +++ b/jobs/register-broker/templates/config.json.erb @@ -1,10 +1,9 @@ <% require 'json' broker = link('juicefs_broker') - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), brokerName: p('juicefs.broker.name'), brokerURL: "http://#{broker.instances.first.address}:#{broker.p('juicefs.broker.port')}", brokerPort: broker.p('juicefs.broker.port'), diff --git a/scripts/acceptance/run b/scripts/acceptance/run index 5df7e78..131fb50 100755 --- a/scripts/acceptance/run +++ b/scripts/acceptance/run @@ -60,14 +60,13 @@ case "${1:-}" in domain=$(bosh interpolate "$work/cf.yml" --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) stemcell=$(bosh interpolate "$work/cf.yml" --path /stemcells/alias=default/version) aliases=$(bosh interpolate "$work/cf.yml" --path /releases/name=bosh-dns-aliases/version) - # Publish CF's endpoint before JuiceFS resolves its cross-deployment link. - bosh -d cf deploy "$work/cf.yml" -o deployment/operations/cf/share-cloud-controller.yml - bosh -d cf manifest > "$work/cf.yml" bosh -d juicefs deploy deployment/juicefs.yml \ -o deployment/operations/non-ha-validation.yml \ -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ -o deployment/operations/misc/ibosh-cf-credentials.yml \ -o deployment/operations/misc/ibosh-acceptance.yml \ + -o deployment/operations/misc/skip-cf-api-tls-validation.yml \ + -v juicefs_cf_system_domain="$domain" \ -v juicefs_release_version="$version" \ -v acceptance_stemcell_version="$stemcell" \ -v acceptance_dns_aliases_version="$aliases" diff --git a/scripts/deploy b/scripts/deploy new file mode 100755 index 0000000..05f5523 --- /dev/null +++ b/scripts/deploy @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +root=$(git rev-parse --show-toplevel) +cd "$root" +set -a +source ./bosh.env +set +a + +bosh create-release --force +bosh upload-release +release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) +release_version=$(bosh interpolate "$release_manifest" --path /version) + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT +bosh -d cf manifest > "$work/cf.yml" +system_domain=$(bosh interpolate "$work/cf.yml" --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) + +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/non-ha-validation.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -v juicefs_cf_system_domain="$system_domain" + +bosh update-config --type=runtime --name=juicefs -n \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -v juicefs_release_version="$release_version" + +bosh -n -d cf deploy "$work/cf.yml" \ + -o deployment/operations/cf/non-ha-validation.yml \ + -o deployment/operations/cf/enable-volume-services.yml diff --git a/tests/render_templates.rb b/tests/render_templates.rb index c8a1bd8..6b1f3b7 100644 --- a/tests/render_templates.rb +++ b/tests/render_templates.rb @@ -33,25 +33,29 @@ def render(job, template) end end -class AcceptanceTemplatesTest < Minitest::Test - def test_cf_link_and_extra_roots_are_rendered_without_losing_json_escaping - cc = FakeLink.new({ - 'cc.internal_service_hostname' => 'api.service.cf.internal', - 'cc.public_tls.port' => 9024, - 'cc.public_tls.ca_cert' => "cc-root\n" - }) +class CloudControllerLinkTemplatesTest < Minitest::Test + def test_registration_uses_public_api_route_and_preserves_tls_ca_json_escaping properties = { 'juicefs.cf.password' => "secret\"with\nnewlines", + 'juicefs.cf.system_domain' => 'system.example.com', 'juicefs.cf.additional_ca_certificates' => ['router-root'], 'juicefs.acceptance.apps_domain' => 'apps.example.com' } - rendered = RenderContext.new('juicefs-acceptance-tests', properties, {'cloud_controller_https_endpoint' => cc}).render('juicefs-acceptance-tests', 'config.json.erb') - config = JSON.parse(rendered) - assert_equal 'https://api.service.cf.internal:9024', config['apiURL'] - assert_equal "cc-root\n\nrouter-root", config['caCert'] - assert_equal properties['juicefs.cf.password'], config['password'] - assert_equal 'apps.example.com', config['appsDomain'] - assert_equal false, config['skipSSLValidation'] + broker = FakeLink.new({'juicefs.broker.port' => 8080, 'juicefs.broker.username' => 'broker', 'juicefs.broker.password' => 'secret'}) + links = {'juicefs_broker' => broker} + + %w[register-broker deregister-broker].each do |job| + config = JSON.parse(RenderContext.new(job, properties, links).render(job, 'config.json.erb')) + assert_equal 'https://api.system.example.com', config['apiURL'] + assert_equal 'router-root', config['caCert'] + assert_equal properties['juicefs.cf.password'], config['cfPassword'] + end + + links.delete('juicefs_broker') + acceptance = JSON.parse(RenderContext.new('juicefs-acceptance-tests', properties, links).render('juicefs-acceptance-tests', 'config.json.erb')) + assert_equal 'https://api.system.example.com', acceptance['apiURL'] + assert_equal 'router-root', acceptance['caCert'] + assert_equal properties['juicefs.acceptance.apps_domain'], acceptance['appsDomain'] end end From 75cc2f265c029d57a3934d0a1a8e51387085a13a Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 17:08:12 +0100 Subject: [PATCH 07/11] Deploy JuiceFS before its CF volume driver runtime config --- docs/azure-managed-identities.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index bd1f542..60d072c 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -155,16 +155,16 @@ bosh upload-release Set `release_version` to the explicit version reported by `create-release`, then: ```sh -bosh update-config --type=runtime --name=juicefs -n \ - deployment/runtime-configs/juicefs-volume-driver.yml \ - -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ - --vars-file azure-vars.yml -v juicefs_release_version="$release_version" system_domain=$(bosh -d cf manifest | bosh interpolate - --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) bosh -n -d juicefs deploy deployment/juicefs.yml \ -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ -o deployment/operations/misc/ibosh-cf-credentials.yml \ -o deployment/operations/use-azure-managed-identities.yml \ -v juicefs_cf_system_domain="$system_domain" --vars-file azure-vars.yml +bosh update-config --type=runtime --name=juicefs -n \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ + --vars-file azure-vars.yml -v juicefs_release_version="$release_version" bosh deploy -n -d cf <(bosh manifest -d cf) \ -o deployment/operations/cf/enable-volume-services.yml ``` From 30512e0d7cf88333cc1be7b3256f24f8649ddb2a Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 26 Oct 2026 17:20:24 +0100 Subject: [PATCH 08/11] Clarify CF API CA and Azure deployment instructions --- README.md | 1 + .../operations/misc/provide-cf-api-ca.yml | 5 -- docs/azure-managed-identities.md | 47 ++++++++++++++++--- 3 files changed, 42 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index c300583..d382833 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,7 @@ It also requires `juicefs_cf_password`, `juicefs_cf_system_domain`, and credential ops. Registration calls the public `https://api.` route without a cross-deployment Cloud Controller link. Use `deployment/operations/misc/provide-cf-api-ca.yml` to configure the trusted API CA. +Supply `juicefs_cf_api_ca_certificate` from a vars file or `-v` when applying it. For singleton validation, apply `deployment/operations/non-ha-validation.yml` instead; it supplies both objects, Postgres/WebDAV jobs, credentials, certificates, and backend DNS aliases. Apply its CF companion diff --git a/deployment/operations/misc/provide-cf-api-ca.yml b/deployment/operations/misc/provide-cf-api-ca.yml index d1b59e0..3c45a7e 100644 --- a/deployment/operations/misc/provide-cf-api-ca.yml +++ b/deployment/operations/misc/provide-cf-api-ca.yml @@ -3,8 +3,3 @@ path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/additional_ca_certificates value: - ((juicefs_cf_api_ca_certificate)) -- type: replace - path: /variables/- - value: - name: juicefs_cf_api_ca_certificate - type: value diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index 60d072c..8600ce7 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -152,21 +152,35 @@ devbox run -- bosh create-release --force bosh upload-release ``` -Set `release_version` to the explicit version reported by `create-release`, then: +Set `release_version` to the explicit version reported by `create-release` and +`system_domain` to the CF manifest's `system_domain`. Ensure the CF `api.` +route is reachable from the JuiceFS VM, and configure JuiceFS with the CF admin +password and a CA bundle trusted for that public API route. Install BOSH DNS +using your director's standard runtime-config before deploying JuiceFS. ```sh system_domain=$(bosh -d cf manifest | bosh interpolate - --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) +bosh -n -d cf deploy <(bosh -d cf manifest) \ + -o deployment/operations/cf/enable-volume-services.yml bosh -n -d juicefs deploy deployment/juicefs.yml \ - -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ - -o deployment/operations/misc/ibosh-cf-credentials.yml \ -o deployment/operations/use-azure-managed-identities.yml \ - -v juicefs_cf_system_domain="$system_domain" --vars-file azure-vars.yml + -o deployment/operations/misc/provide-cf-api-ca.yml \ + -v juicefs_cf_system_domain="$system_domain" \ + -v juicefs_cf_password="$cf_admin_password" \ + -v juicefs_cf_api_ca_certificate="$cf_api_ca_certificate" \ + --vars-file azure-vars.yml +bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0 +``` + +After JuiceFS exists, update the runtime-config that deploys the driver to CF +cells. Doing this afterward ensures its `juicefs_mount` link is available: + +```sh bosh update-config --type=runtime --name=juicefs -n \ deployment/runtime-configs/juicefs-volume-driver.yml \ -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ --vars-file azure-vars.yml -v juicefs_release_version="$release_version" -bosh deploy -n -d cf <(bosh manifest -d cf) \ - -o deployment/operations/cf/enable-volume-services.yml +bosh deploy -n -d cf <(bosh -d cf manifest) ``` Use these Azure-specific commands on subsequent deployments too. The default @@ -175,6 +189,27 @@ not the Azure deployment entry point. The bundled PostgreSQL/WebDAV backend group is retained by the example operations; remove unused jobs/resources in your deployment-specific manifest after validating the external backends. +### When using instant-bosh + +The instant-bosh CLI keeps generated CF credentials in its config server and +embeds DNS in CF. Use the following JuiceFS ops to source CF credentials and +reuse the global `/dns_*` certificates already defined by the CF deployment, +instead of supplying the standard-director variables and DNS configuration above: + +```sh +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -o deployment/operations/use-azure-managed-identities.yml \ + --vars-file azure-vars.yml +``` + +The `ibosh-cf-credentials.yml` ops supplies the system domain, admin password, +and router CA from instant-bosh's config server, with TLS verification enabled. +The skip-validation ops file is only for disposable tests. If using instant-bosh +for a production deployment, validate the credential lifecycle, CA trust, and +DNS behavior in that environment before relying on it. + Bootstrap initialization now runs in the supervised mount launcher, rather than `pre-start`, so Monit can start the colocated proxy concurrently. It retries `format --no-update` on `spec.bootstrap` and fails if initialization cannot From d14c695a53649191507d0d89d08edae3f0496e08 Mon Sep 17 00:00:00 2001 From: rkoster Date: Tue, 27 Oct 2026 15:21:36 +0100 Subject: [PATCH 09/11] Simplify acceptance testing documentation --- docs/acceptance-testing.md | 128 ++++--------------------------------- 1 file changed, 11 insertions(+), 117 deletions(-) diff --git a/docs/acceptance-testing.md b/docs/acceptance-testing.md index f3995f6..55f8b3d 100644 --- a/docs/acceptance-testing.md +++ b/docs/acceptance-testing.md @@ -1,121 +1,15 @@ -# Cloud Foundry volume acceptance +# Acceptance testing -`.github/workflows/acceptance.yml` runs on pull requests and manual dispatch. It -creates a disposable Docker-backed [instant-bosh](https://github.com/rkoster/instant-bosh) -director, deploys CF, builds this checkout as a BOSH release, deploys JuiceFS, -updates the CF cells through the named `juicefs` runtime-config, registers the -broker, and runs the `juicefs-acceptance-tests` BOSH errand. +The [acceptance workflow](../.github/workflows/acceptance.yml) runs a disposable +Cloud Foundry environment to check that the JuiceFS volume service works end to +end. -## GitHub setup +It provisions and binds a volume to an app, writes a random value to a file, +restarts the app, and verifies the value is still available. This exercises the +service broker, volume driver, and persistence across app restarts. -The workflow currently uses the standard GitHub-hosted **`ubuntu-latest`** runner; -no runner variable is required. Initial runs will establish how far the full CF -deployment fits within its resources. Diagnostics include host memory, disk, -Docker disk usage, and container statistics to identify capacity limits. -The acceptance job adds an 8 GiB swap file using shell commands, leaving the -runner's existing disk layout intact. +The workflow can also be run locally from a disposable environment. Its commands +are in [`scripts/acceptance/run`](../scripts/acceptance/run). -Local runners need Linux x64, Docker Engine, passwordless sudo, the host FUSE -kernel module/device, `setfacl` (the `acl` package), and support for privileged containers and nested Garden -containers. Use an ephemeral, single-job runner: instant-bosh uses fixed resource -names and ports. - -Startup grants the invoking user a Docker socket ACL: the director's pre-start -hook changes the socket's group to vcap (GID 1000), including on the host through -the bind mount. The ACL preserves runner access after this group change. - -The workflow uses `pull_request` with read-only repository permissions. It does -not need CF credentials: the disposable director generates them. The release's -GCS blobstore must allow anonymous reads of vendored package blobs for fork PRs; -`bosh sync-blobs` fails if those reads are unavailable. Configure public read -access to those release artifacts separately from uploader credentials. - -Tool versions are in `scripts/acceptance/versions.env`. The instant-bosh CLI is -built from a pinned commit because published `v0.3` predates `ibosh cf deploy`. -The director image is pinned by digest. Update these together and verify a full -run. CF release and compiled-stemcell versions come from that CLI's embedded -cf-deployment; the JuiceFS deployment reuses its installed alias release and -stemcell. CI uses a 10 GiB backend disk. Singleton Postgres/WebDAV and their -DNS aliases are enabled with `deployment/operations/non-ha-validation.yml` -and the CF companion `deployment/operations/cf/non-ha-validation.yml`. - -The pinned instant-bosh CLI embeds DNS only in CF. CI applies -`deployment/operations/misc/ibosh-bosh-dns-fix.yml` to JuiceFS to add a DNS -addon using the global certificates already generated by CF. No DNS variable -definitions or changes to CF's DNS configuration are needed. On other directors, -use the upstream DNS runtime-config instead (see `deployment/runtime-configs/README.md`). - -Director startup skips the default `latest` stemcell upload, whose registry -version-tag discovery can be slow. `ibosh cf deploy` uploads the exact stemcell -required by its compiled CF releases instead. - -## What the errand verifies - -The release-owned Go suite uses vendored go-cfclient v3 and Ginkgo/Gomega. It: - -1. Resolves the `shared` service plan under the registered `juicefs` broker. -2. Creates a unique org, space, service instance, app, and route through CAPI. -3. Binds the volume at `/data`, uploads a static Go HTTP app, and stages it with - CF's `binary_buildpack` on `cflinuxfs4` (no runtime downloads in the app). -4. POSTs a random UUID to `/file`; the app writes, syncs, and closes the file. -5. Restarts the app through CAPI and waits for `/identity` to change, proving - requests are reaching a replacement process. -6. GETs `/file` and requires the exact original UUID. -7. Deletes resources in dependency order, awaiting asynchronous CAPI jobs even - after a failed assertion. - -The app endpoints require a per-run bearer token and disable response caching. -The mount directory must already exist; the app never creates a fallback `/data`. -TLS verification uses the Cloud Controller link CA and the CF router CA. - -The errand can be enabled independently on the consolidated CF topology: apply -`deployment/operations/cf/enable-volume-services.yml`, -`deployment/operations/cf/enable-juicefs-acceptance-tests.yml`, supplying -`juicefs_release_version` and `juicefs_acceptance_apps_domain`, then run -`bosh -d cf run-errand juicefs-acceptance-tests` after registering the broker. -Its job properties allow choosing another broker name or plan. - -Registration runs in JuiceFS with -`bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0`. -Registration calls `https://api.`, the public CF API route, and -does not consume a cross-deployment endpoint link. The main manifest colocates -both registration errands on the broker VM. CI sets the system domain from the -CF manifest and skips API TLS verification only for this disposable test -environment; other deployments can use -`deployment/operations/misc/provide-cf-api-ca.yml` to configure the public API CA. - -## Local execution - -Install Go 1.25, BOSH CLI, jq, and the pinned instant-bosh CLI; initialize the -release's submodules. On a disposable Linux Docker host, run from the release -root (inside Devbox if Go is not otherwise on PATH): - -```sh -bash scripts/acceptance/run start -bash scripts/acceptance/run deploy-cf -bash scripts/acceptance/run build-release -bash scripts/acceptance/run deploy-service -bash scripts/acceptance/run test -bash scripts/acceptance/run diagnostics -bash scripts/acceptance/run destroy -``` - -The default work directory is ignored `.acceptance/`; override it with -`ACCEPTANCE_WORK_DIR`. It contains private deployment material, so only its -`artifacts/` subdirectory is uploaded by CI. Do not run against an existing -instant-bosh environment you want to retain: `destroy` removes its resources. - -The workflow always attempts diagnostics and destruction after failure. Download -the `juicefs-acceptance--` artifact for JUnit, errand logs, failed -BOSH task debug output, and full CF and JuiceFS logs (including agent and system -logs). The compute VM bundle is collected first so rep and Garden failures are -captured even if later collection times out. Each bundle collection is bounded -to five minutes. The acceptance VM is kept alive -until destruction so its JUnit report can be retrieved. Cancellation or a runner -loss can interrupt cleanup; ephemeral runners reclaim remaining resources. - -`scripts/test-go` includes the app checks and a local TLS CAPI/UAA/router fixture -that executes the real suite. The fixture verifies broker scoping, executable -app upload, restart, and cleanup on both success and deliberate post-restart -data loss. Live acceptance is skipped unless `JUICEFS_ACCEPTANCE_CONFIG` points -to the rendered job JSON. Template checks run with `ruby tests/render_templates.rb`. +On failure, download the acceptance artifact from the GitHub Actions run summary +for test results and diagnostic logs. From f37ebe1b621a5a686a22f6706900f34418c01ebe Mon Sep 17 00:00:00 2001 From: rkoster Date: Tue, 27 Oct 2026 16:57:49 +0100 Subject: [PATCH 10/11] Keep agent deployment guidance concise and code-linked --- AGENTS.md | 97 ++++++++++++++++--------------------------------------- 1 file changed, 28 insertions(+), 69 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3293891..2afd56e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,9 @@ # Development loop +When changing documented commands, paths, or development constraints, update this +file in the same change. Verify claims against code; link to detailed documentation +rather than duplicating it. + This is the `juicefs` BOSH release. `src/juicefs` is a pinned upstream Git submodule; `packages/juicefs` builds it with the vendored `golang-1.25-linux` BOSH package. `nginx_webdav` is vendored from CAPI release and powers our own singleton @@ -21,8 +25,8 @@ run `scripts/generate-fakes` to regenerate fakes using the pinned vendored tool. errand. Local checks include a TLS CF API fixture; the live suite requires `JUICEFS_ACCEPTANCE_CONFIG`. CI orchestration lives in `scripts/acceptance/run`; see `docs/acceptance-testing.md`. The workflow currently uses the standard -GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together in -`scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts. +GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together +in `scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts. Do not check generated vendor trees for upstream submodules into Git. `packages/juicefs` contains only the upstream client. Release-owned tools have separate packages and Go modules under matching `src/` directories; these compile @@ -47,32 +51,18 @@ after adding blobs. The repository home is director credentials. Do not commit it or print its contents. Refresh it from the instant-bosh repo when the director credentials change. -From the release root: - -```sh -devbox run deploy -``` +From the release root, run `devbox run deploy` for the instant-bosh validation +environment. `scripts/deploy` defines its deployment sequence. See +`README.md#deploy-and-register` and `docs/acceptance-testing.md` for details. -This sources `bosh.env`, runs `bosh create-release --force`, uploads that release, -updates the named `juicefs` runtime-config with `bosh update-config` using the -newly built release's explicit version, deploys the JuiceFS release, then runs -`bosh deploy -n -d cf <(bosh manifest -d cf)` so pending runtime-config changes -are applied. `deployment/juicefs.yml` contains the `juicefs-broker` group and accepts -`juicefs_metadata` and `juicefs_storage` objects for external backends. -`deployment/operations/non-ha-validation.yml` adds singleton Postgres/WebDAV, -their variables and DNS aliases, and overrides those backend objects. Apply it -before `misc/ibosh-acceptance.yml`. Its backend persistent disk is 50 GiB. -The CF aliases are in -`deployment/operations/cf/non-ha-validation.yml`. Instant-bosh embeds DNS only -in CF; `deployment/operations/misc/ibosh-bosh-dns-fix.yml` adds it to JuiceFS, -reusing global `/dns_*` certificates without declaring variables. Other directors -should use the upstream DNS runtime-config. See `deployment/runtime-configs/README.md`. -The manifest uses `version: latest` for the -`juicefs` release. The director must have an -`ubuntu-noble` stemcell and cloud-config with `z1`, `default` network and -`default` VM type. Postgres and BPM are pinned external releases; BPM is used by -the blobstore job. The JuiceFS FUSE client runs directly on the VM under Monit, -not in BPM, so it can access the host `/dev/fuse` and publish its mount normally. +The base `deployment/juicefs.yml` accepts `juicefs_metadata` and +`juicefs_storage` objects. `deployment/operations/non-ha-validation.yml` adds +singleton Postgres/WebDAV, their DNS aliases, and the 50 GiB backend disk. +Instant-bosh DNS and credential adjustments live in +`deployment/operations/misc/`; other directors should use their normal DNS and +credential configuration. The director needs an `ubuntu-noble` stemcell and +cloud-config with `z1`, the `default` network, and `default` VM type. The JuiceFS +FUSE client runs directly on the VM under Monit so it can access host `/dev/fuse`. The singleton WebDAV blobstore keeps objects on its persistent disk at `/var/vcap/store/juicefs-blobstore/`. The job provides a `webdav` BOSH link; @@ -138,46 +128,15 @@ records make failed quota application retryable; completed provision retries do not change quotas. Existing pre-quota directories retain the unlimited shared plan. The broker's instance metadata lives on JuiceFS, not on its ephemeral disk. -`deployment/operations/cf/enable-volume-services.yml` enables CAPI volume services. -The main JuiceFS manifest colocates `register-broker` and `deregister-broker` -as errands on `juicefs-broker`. They consume the local broker link and CF's shared -API system domain, constructing `https://api.` without a -cross-deployment endpoint link. `misc/ibosh-cf-credentials.yml` references the -existing CF domain and credentials in instant-bosh; other environments supply -`juicefs_cf_password`, `juicefs_cf_system_domain`, and -`juicefs_cf_ca_certificates`. `misc/provide-cf-api-ca.yml` configures the trusted -API CA; TLS verification can be skipped for disposable tests with -`misc/skip-cf-api-tls-validation.yml`. The CF deployment is separate -from the JuiceFS deployment. The JuiceFS deployment marks Postgres, WebDAV, the -full-root mountpoint, and broker as shared link providers. The CF runtime-config -consumes the broker mounter's shared `juicefs_mount` link from deployment -`juicefs`. That mounter is authoritative for `juicefs.metadata` (postgres or -mysql), `juicefs.storage` (webdav, s3 or wasb/Azure Blob), and the filesystem name. The Diego -driver is link-only, with no independent backend overrides. Quota -commands reuse the mounter's rendered configuration. The colocated broker -consumes its mountpoint link. `deployment/operations/use-mysql-s3.yml` shows external -backend configuration; `deployment/operations/use-azure-blob.yml` configures Azure Blob. -Changing properties does not migrate existing data. The CF control group has -no full-root mount; only the broker and cell driver mount JuiceFS. - -The runtime-config is `deployment/runtime-configs/juicefs-volume-driver.yml`. It targets the -live `compute` group for the isolated -volume driver. Broker registration errands consume the shared broker link. The -broker provisions -directories beneath `/cf-volumes/`; each cell mount uses -JuiceFS `--subdir` for only that directory. Bindings are synchronous and stateless, -with `bindings_retrievable: false`; binding retrieval is not implemented. -The optional binding parameter `mount` selects a clean absolute container path -(for example, `cf bind-service cf-env foobar -c '{"mount":"/data"}'`). Without it, -the path is `/var/vcap/data/`. Cloud Controller stores the -returned mount instructions. This view restriction is not strong -isolation against hostile users who can access shared backend credentials. -The broker-registration errands currently use the configured CF admin -credential; replace that with a dedicated account authorized to manage service -brokers and service access before using this POC beyond a test environment. +See `README.md#cloud-foundry-volume-service-poc` and +`docs/acceptance-testing.md` for the volume-service deployment and validation +details. `scripts/deploy` is the source of truth for the instant-bosh deployment +sequence. The CF runtime-config consumes the broker mounter's shared +`juicefs_mount` link; the mounter defines the backend configuration and +filesystem name. The driver mounts only the service-instance subdirectory. -Build and validate from this release root with `bosh create-release --force`, -then interpolate the CF manifest with the ops file and the runtime-config with -the required backend variables. After applying both to the director, -run `bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0`, then verify CAPI provision/bind and app-container -read/write/unmount behavior before relying on deprovision cleanup. +Service bindings are synchronous and stateless; binding retrieval is not +implemented. The optional `mount` binding parameter selects an absolute container +path, defaulting to `/var/vcap/data/`. The broker currently +uses the configured CF administrator credential; replace it with a dedicated +broker-management account before using this POC beyond a test environment. From e29182ba21b450d6fde861338f355aedcee7479d Mon Sep 17 00:00:00 2001 From: rkoster Date: Wed, 28 Oct 2026 09:35:01 +0100 Subject: [PATCH 11/11] Use CF manifest domain for instant-bosh credentials --- README.md | 2 +- deployment/operations/misc/ibosh-cf-credentials.yml | 6 ++---- docs/azure-managed-identities.md | 5 +++-- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index d382833..62ab86b 100644 --- a/README.md +++ b/README.md @@ -148,7 +148,7 @@ This command: `deployment/operations/non-ha-validation.yml` and the instant-bosh DNS workaround `deployment/operations/misc/ibosh-bosh-dns-fix.yml`. The `misc/ibosh-cf-credentials.yml` ops reuses CF credentials in instant-bosh's - config server. + config server; `scripts/deploy` supplies the system domain from CF's manifest. 3. Updates the named `juicefs` runtime-config with the built release's explicit version and redeploys CF, applying `deployment/operations/cf/non-ha-validation.yml`, diff --git a/deployment/operations/misc/ibosh-cf-credentials.yml b/deployment/operations/misc/ibosh-cf-credentials.yml index 87695eb..a2b0b57 100644 --- a/deployment/operations/misc/ibosh-cf-credentials.yml +++ b/deployment/operations/misc/ibosh-cf-credentials.yml @@ -1,7 +1,5 @@ -# Reuse CF credentials from instant-bosh's config-server namespace. -- type: replace - path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/system_domain - value: ((/instant-bosh/cf/system_domain)) +# Reuse CF credentials from instant-bosh's config-server namespace. The caller +# supplies juicefs_cf_system_domain from the generated CF manifest. - type: replace path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/password value: ((/instant-bosh/cf/cf_admin_password)) diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index 8600ce7..71755e5 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -204,8 +204,9 @@ bosh -n -d juicefs deploy deployment/juicefs.yml \ --vars-file azure-vars.yml ``` -The `ibosh-cf-credentials.yml` ops supplies the system domain, admin password, -and router CA from instant-bosh's config server, with TLS verification enabled. +The `ibosh-cf-credentials.yml` ops supplies the admin password and router CA from +instant-bosh's config server. Set `juicefs_cf_system_domain` from the generated +CF manifest; TLS verification remains enabled. The skip-validation ops file is only for disposable tests. If using instant-bosh for a production deployment, validate the credential lifecycle, CA trust, and DNS behavior in that environment before relying on it.