diff --git a/AGENTS.md b/AGENTS.md index 92d6391..2afd56e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,9 @@ # Development loop +When changing documented commands, paths, or development constraints, update this +file in the same change. Verify claims against code; link to detailed documentation +rather than duplicating it. + This is the `juicefs` BOSH release. `src/juicefs` is a pinned upstream Git submodule; `packages/juicefs` builds it with the vendored `golang-1.25-linux` BOSH package. `nginx_webdav` is vendored from CAPI release and powers our own singleton @@ -21,8 +25,8 @@ run `scripts/generate-fakes` to regenerate fakes using the pinned vendored tool. errand. Local checks include a TLS CF API fixture; the live suite requires `JUICEFS_ACCEPTANCE_CONFIG`. CI orchestration lives in `scripts/acceptance/run`; see `docs/acceptance-testing.md`. The workflow currently uses the standard -GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together in -`scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts. +GitHub-hosted `ubuntu-latest` runner. Keep instant-bosh CLI/image pins together +in `scripts/acceptance/versions.env`. Run the orchestration only on disposable hosts. Do not check generated vendor trees for upstream submodules into Git. `packages/juicefs` contains only the upstream client. Release-owned tools have separate packages and Go modules under matching `src/` directories; these compile @@ -47,23 +51,18 @@ after adding blobs. The repository home is director credentials. Do not commit it or print its contents. Refresh it from the instant-bosh repo when the director credentials change. -From the release root: - -```sh -devbox run deploy -``` +From the release root, run `devbox run deploy` for the instant-bosh validation +environment. `scripts/deploy` defines its deployment sequence. See +`README.md#deploy-and-register` and `docs/acceptance-testing.md` for details. -This sources `bosh.env`, runs `bosh create-release --force`, uploads that release, -updates the named `juicefs` runtime-config with `bosh update-config` using the -newly built release's explicit version, deploys the JuiceFS release, then runs -`bosh deploy -n -d cf <(bosh manifest -d cf)` so pending runtime-config changes -are applied. The JuiceFS manifest contains -`juicefs-backends` and `juicefs-broker` instance groups and uses `version: latest` for the -`juicefs` release. The director must have an -`ubuntu-noble` stemcell and cloud-config with `z1`, `default` network and -`default` VM type. Postgres and BPM are pinned external releases; BPM is used by -the blobstore job. The JuiceFS FUSE client runs directly on the VM under Monit, -not in BPM, so it can access the host `/dev/fuse` and publish its mount normally. +The base `deployment/juicefs.yml` accepts `juicefs_metadata` and +`juicefs_storage` objects. `deployment/operations/non-ha-validation.yml` adds +singleton Postgres/WebDAV, their DNS aliases, and the 50 GiB backend disk. +Instant-bosh DNS and credential adjustments live in +`deployment/operations/misc/`; other directors should use their normal DNS and +credential configuration. The director needs an `ubuntu-noble` stemcell and +cloud-config with `z1`, the `default` network, and `default` VM type. The JuiceFS +FUSE client runs directly on the VM under Monit so it can access host `/dev/fuse`. The singleton WebDAV blobstore keeps objects on its persistent disk at `/var/vcap/store/juicefs-blobstore/`. The job provides a `webdav` BOSH link; @@ -129,38 +128,15 @@ records make failed quota application retryable; completed provision retries do not change quotas. Existing pre-quota directories retain the unlimited shared plan. The broker's instance metadata lives on JuiceFS, not on its ephemeral disk. -`operations/enable-juicefs-cf-volume-service.yml` enables CAPI volume services -and defines registration/deregistration errands. The CF deployment is separate -from the JuiceFS deployment. The JuiceFS deployment marks Postgres, WebDAV, the -full-root mountpoint, and broker as shared link providers. The CF runtime-config -consumes the broker mounter's shared `juicefs_mount` link from deployment -`juicefs`. That mounter is authoritative for `juicefs.metadata` (postgres or -mysql), `juicefs.storage` (webdav, s3 or wasb/Azure Blob), and the filesystem name. The Diego -driver is link-only, with no independent backend overrides. Quota -commands reuse the mounter's rendered configuration. The colocated broker -consumes its mountpoint link. `operations/use-mysql-s3.yml` shows external -backend configuration; `operations/use-azure-blob.yml` configures Azure Blob. -Changing properties does not migrate existing data. The CF control group has -no full-root mount; only the broker and cell driver mount JuiceFS. - -The runtime-config is `runtime-config/juicefs-volume-driver.yml`. It targets the -live `compute` group for the isolated -volume driver. Broker registration errands consume the shared broker link. The -broker provisions -directories beneath `/cf-volumes/`; each cell mount uses -JuiceFS `--subdir` for only that directory. Bindings are synchronous and stateless, -with `bindings_retrievable: false`; binding retrieval is not implemented. -The optional binding parameter `mount` selects a clean absolute container path -(for example, `cf bind-service cf-env foobar -c '{"mount":"/data"}'`). Without it, -the path is `/var/vcap/data/`. Cloud Controller stores the -returned mount instructions. This view restriction is not strong -isolation against hostile users who can access shared backend credentials. -The broker-registration errands currently use the configured CF admin -credential; replace that with a dedicated account authorized to manage service -brokers and service access before using this POC beyond a test environment. +See `README.md#cloud-foundry-volume-service-poc` and +`docs/acceptance-testing.md` for the volume-service deployment and validation +details. `scripts/deploy` is the source of truth for the instant-bosh deployment +sequence. The CF runtime-config consumes the broker mounter's shared +`juicefs_mount` link; the mounter defines the backend configuration and +filesystem name. The driver mounts only the service-instance subdirectory. -Build and validate from this release root with `bosh create-release --force`, -then interpolate the CF manifest with the ops file and the runtime-config with -the required backend variables. After applying both to the director, -run `register-juicefs-broker`, then verify CAPI provision/bind and app-container -read/write/unmount behavior before relying on deprovision cleanup. +Service bindings are synchronous and stateless; binding retrieval is not +implemented. The optional `mount` binding parameter selects an absolute container +path, defaulting to `/var/vcap/data/`. The broker currently +uses the configured CF administrator credential; replace it with a dedicated +broker-management account before using this POC beyond a test environment. diff --git a/README.md b/README.md index d9aaa13..62ab86b 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,9 @@ A proof-of-concept [JuiceFS](https://juicefs.com/) volume service for Cloud Foundry. The release includes a service broker, a Diego volume driver, and a shared filesystem with PostgreSQL or MySQL metadata and WebDAV, S3, or Azure Blob object storage. -The supplied deployment uses PostgreSQL and persistent WebDAV storage. +The base deployment accepts external metadata and object-storage configuration. +The `deployment/operations/non-ha-validation.yml` ops file adds singleton +PostgreSQL and persistent WebDAV backends for validation. Applications can provision a directory, bind it at a chosen container path, and keep files across container restarts. BOSH-configurable service plans apply @@ -39,7 +41,7 @@ Diego cell → JuiceFS volume driver → subdirectory FUSE mount → app contain └─ WebDAV / S3 / Azure Blob: persistent data blocks ``` -The `juicefs` deployment has two singleton instance groups: +With the non-HA validation ops file, `juicefs` has two singleton instance groups: | Group | Jobs | Storage | | --- | --- | --- | @@ -56,7 +58,7 @@ The CF runtime-config adds the volume driver to `compute`. It consumes the broker VM's shared `juicefs_mount` BOSH link for the filesystem name and all backend settings, credentials, and trust roots. The broker's volume mounter is the authoritative configuration source; -registration errands run in CF. +registration errands are colocated on `juicefs-broker` in the JuiceFS deployment. FUSE clients run directly under Monit so they can use the host `/dev/fuse` and mount namespace. In the default WebDAV setup, clients connect directly to @@ -142,22 +144,42 @@ devbox run deploy This command: 1. Sources `bosh.env` and creates and uploads a development release. -2. Updates the named `juicefs` runtime-config with the built release's explicit - version. -3. Deploys `manifests/juicefs.yml`. -4. Redeploys CF from its director manifest, applying - `operations/enable-juicefs-cf-volume-service.yml`. - -The ops file enables CAPI volume services and adds registration/deregistration -errands. Registration is a separate step, also needed after catalog changes: +2. Reads CF's system domain and deploys `deployment/juicefs.yml` with + `deployment/operations/non-ha-validation.yml` and the instant-bosh DNS + workaround `deployment/operations/misc/ibosh-bosh-dns-fix.yml`. The + `misc/ibosh-cf-credentials.yml` ops reuses CF credentials in instant-bosh's + config server; `scripts/deploy` supplies the system domain from CF's manifest. +3. Updates the named `juicefs` runtime-config with the built release's explicit + version and redeploys CF, applying + `deployment/operations/cf/non-ha-validation.yml`, + `deployment/operations/cf/enable-volume-services.yml`. + +The main JuiceFS manifest colocates `register-broker` and `deregister-broker` +as errands on the broker VM. Registration is a separate step, also needed after catalog changes: ```sh source bosh.env -bosh -n -d cf run-errand register-juicefs-broker +bosh -n -d juicefs run-errand register-broker --instance=juicefs-broker/0 cf service-brokers cf marketplace -e juicefs ``` +The base `deployment/juicefs.yml` requires `juicefs_metadata` and +`juicefs_storage` objects in a vars file, or an external-backend ops file. +It also requires `juicefs_cf_password`, `juicefs_cf_system_domain`, and +`juicefs_cf_ca_certificates` for registration, unless using the instant-bosh +credential ops. Registration calls the public `https://api.` +route without a cross-deployment Cloud Controller link. Use +`deployment/operations/misc/provide-cf-api-ca.yml` to configure the trusted API CA. +Supply `juicefs_cf_api_ca_certificate` from a vars file or `-v` when applying it. +For singleton validation, apply `deployment/operations/non-ha-validation.yml` +instead; it supplies both objects, Postgres/WebDAV jobs, credentials, certificates, +and backend DNS aliases. Apply its CF companion +`deployment/operations/cf/non-ha-validation.yml` to CF. +For instant-bosh, the misc DNS workaround adds a JuiceFS addon that reuses +CF's global certificates. For other directors, use the upstream DNS +runtime-config; see [DNS setup](deployment/runtime-configs/README.md). + The registration errand creates or updates the broker and enables access to its plans. It trusts both Cloud Controller's linked CA and the configured router CA used by the public authentication endpoint. The supplied ops file uses CF's @@ -296,11 +318,11 @@ link. CF's volume driver consumes that link directly from deployment `juicefs`; they have no independent backend overrides. Broker quota operations reuse the mounter's rendered configuration, including its metadata engine. -`operations/use-mysql-s3.yml` provides an example deployment ops file with -external backend variables. PostgreSQL/WebDAV remains the default. Provision +`deployment/operations/use-mysql-s3.yml` provides an example deployment ops file with +external backend variables. PostgreSQL/WebDAV is selected by the non-HA validation ops file. Provision the external database, object bucket, and access credentials before initializing -a new filesystem. The bundled backend jobs are not removed by that example ops -file. Changing properties is not a data migration: `format --no-update` does +a new filesystem. External backend ops can be applied directly to the base +manifest without deploying singleton backend jobs. Changing properties is not a data migration: `format --no-update` does not rewrite an existing filesystem's storage settings or copy metadata/objects. The linked PostgreSQL/WebDAV configuration is validated on the live POC; @@ -308,7 +330,7 @@ MySQL/S3/Azure Blob support has configuration tests but has not been integration ### Azure Blob Storage -Use `operations/use-azure-blob.yml`, or configure the mounter's storage object: +Use `deployment/operations/use-azure-blob.yml`, or configure the mounter's storage object: ```yaml juicefs: @@ -357,8 +379,8 @@ under `/var/vcap/sys/log/juicefs-volume/`; broker logs are under PostgreSQL metadata and WebDAV blocks share the backend persistent disk (50 GiB in the example manifest). Monitor its free space: a full object store can also prevent PostgreSQL checkpoints, causing metadata EIO errors across all mounts. -`operations/expand-backend-disk.yml` expands the example backend disk to 50 GiB -when applied to the deployed manifest; increase that value for larger workloads. +`deployment/operations/non-ha-validation.yml` sets the backend persistent disk +to 50 GiB; increase its `persistent_disk` value for larger workloads. Preserve the disk and recover the database before reconnecting affected clients. If an app retains a disconnected FUSE mount, stop it and wait for its volume unmount to finish before starting it again; an immediate restart can race the diff --git a/deployment/juicefs.yml b/deployment/juicefs.yml new file mode 100644 index 0000000..a480ea4 --- /dev/null +++ b/deployment/juicefs.yml @@ -0,0 +1,87 @@ +--- +name: juicefs + +releases: +- name: juicefs + version: latest + +stemcells: +- alias: default + os: ubuntu-noble + version: latest + +update: + canaries: 1 + max_in_flight: 1 + canary_watch_time: 30000-60000 + update_watch_time: 30000-60000 + +instance_groups: +- name: juicefs-broker + migrated_from: + - name: juicefs-mount-broker + az: z1 + azs: [z1] + instances: 1 + vm_type: default + stemcell: default + networks: + - name: default + jobs: + - name: juicefs-volume + release: juicefs + properties: + juicefs: + volume: + initialize: true + metadata: ((juicefs_metadata)) + storage: ((juicefs_storage)) + provides: + juicefs_mount: {shared: true} + consumes: + juicefs_volume: nil + - name: juicefs-broker + release: juicefs + provides: + juicefs_broker: {shared: true} + properties: + juicefs: + broker: + password: ((juicefs-broker-password)) + plans: + - id: 40bf3e21-885a-4c2a-99b4-d4e8186c467e + name: shared + description: Shared JuiceFS directory without a quota + free: true + capacity_gib: 0 + inodes: 0 + - id: 5d3c1f42-eaa0-4d1b-88ce-b26cf6545cb1 + name: small + description: Shared JuiceFS directory with 1 GiB and 1000 inodes + free: true + capacity_gib: 1 + inodes: 1000 + consumes: + juicefs_mount: {from: juicefs_mount} + - name: register-broker + release: juicefs + lifecycle: errand + consumes: + juicefs_broker: {from: juicefs_broker, ip_addresses: true} + - name: deregister-broker + release: juicefs + lifecycle: errand + consumes: + juicefs_broker: {from: juicefs_broker, ip_addresses: true} + properties: + juicefs: + cf: + username: admin + system_domain: ((juicefs_cf_system_domain)) + password: ((juicefs_cf_password)) + additional_ca_certificates: ((juicefs_cf_ca_certificates)) + skip_ssl_validation: false + +variables: +- name: juicefs-broker-password + type: password diff --git a/operations/enable-juicefs-acceptance-tests.yml b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml similarity index 73% rename from operations/enable-juicefs-acceptance-tests.yml rename to deployment/operations/cf/enable-juicefs-acceptance-tests.yml index b41ebe8..14d7be4 100644 --- a/operations/enable-juicefs-acceptance-tests.yml +++ b/deployment/operations/cf/enable-juicefs-acceptance-tests.yml @@ -1,3 +1,4 @@ +# Apply to the CF deployment to install the acceptance-test errand. - type: replace path: /instance_groups/name=juicefs-acceptance-tests? value: @@ -11,8 +12,6 @@ jobs: - name: juicefs-acceptance-tests release: juicefs - consumes: - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} properties: juicefs: acceptance: @@ -20,8 +19,11 @@ cf: username: admin password: ((cf_admin_password)) + system_domain: ((juicefs_acceptance_apps_domain)) additional_ca_certificates: [((router_ssl.ca))] skip_ssl_validation: false - type: replace - path: /releases/name=juicefs/version - value: ((juicefs_release_version)) + path: /releases/name=juicefs? + value: + name: juicefs + version: ((juicefs_release_version)) diff --git a/deployment/operations/cf/enable-volume-services.yml b/deployment/operations/cf/enable-volume-services.yml new file mode 100644 index 0000000..7241010 --- /dev/null +++ b/deployment/operations/cf/enable-volume-services.yml @@ -0,0 +1,9 @@ +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/cc/volume_services_enabled? + value: true +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_worker/properties/cc/volume_services_enabled? + value: true +- type: replace + path: /instance_groups/name=control/jobs/name=cloud_controller_clock/properties/cc/volume_services_enabled? + value: true diff --git a/deployment/operations/cf/non-ha-validation.yml b/deployment/operations/cf/non-ha-validation.yml new file mode 100644 index 0000000..a957724 --- /dev/null +++ b/deployment/operations/cf/non-ha-validation.yml @@ -0,0 +1,13 @@ +# Apply to CF when using the singleton backends in ../non-ha-validation.yml. +- type: replace + path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=blobstore.service.juicefs.internal? + value: + domain: blobstore.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} +- type: replace + path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=postgres.service.juicefs.internal? + value: + domain: postgres.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} diff --git a/operations/instant-bosh-acceptance.yml b/deployment/operations/misc/ibosh-acceptance.yml similarity index 54% rename from operations/instant-bosh-acceptance.yml rename to deployment/operations/misc/ibosh-acceptance.yml index a5d6599..df75672 100644 --- a/operations/instant-bosh-acceptance.yml +++ b/deployment/operations/misc/ibosh-acceptance.yml @@ -1,13 +1,12 @@ -# Reuse the stemcell and DNS releases installed by ibosh cf deploy. +# Apply to JuiceFS after ../non-ha-validation.yml. Reuse the CF stemcell and alias release. - type: replace path: /stemcells/alias=default/version value: ((acceptance_stemcell_version)) - type: replace - path: /releases/name=bosh-dns/version - value: ((acceptance_dns_version)) -- type: replace - path: /releases/name=bosh-dns-aliases/version - value: ((acceptance_dns_aliases_version)) + path: /releases/name=bosh-dns-aliases + value: + name: bosh-dns-aliases + version: ((acceptance_dns_aliases_version)) - type: replace path: /releases/name=juicefs/version value: ((juicefs_release_version)) diff --git a/deployment/operations/misc/ibosh-bosh-dns-fix.yml b/deployment/operations/misc/ibosh-bosh-dns-fix.yml new file mode 100644 index 0000000..e3d1792 --- /dev/null +++ b/deployment/operations/misc/ibosh-bosh-dns-fix.yml @@ -0,0 +1,28 @@ +# instant-bosh embeds DNS in CF instead of installing a director runtime-config. +# Reuse the global certificates already generated by the CF deployment. +- type: replace + path: /releases/name=bosh-dns? + value: + name: bosh-dns + version: 1.39.34 + url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.39.34 + sha1: 7561733fb62595b0ccf6b73d771902da4bc519be +- type: replace + path: /addons?/name=ibosh-bosh-dns? + value: + name: ibosh-bosh-dns + jobs: + - name: bosh-dns + release: bosh-dns + properties: + configure_systemd_resolved: true + override_nameserver: false + disable_recursors: true + cache: {enabled: true} + api: + client: {tls: ((/dns_api_client_tls))} + server: {tls: ((/dns_api_server_tls))} + health: + enabled: true + client: {tls: ((/dns_healthcheck_client_tls))} + server: {tls: ((/dns_healthcheck_server_tls))} diff --git a/deployment/operations/misc/ibosh-cf-credentials.yml b/deployment/operations/misc/ibosh-cf-credentials.yml new file mode 100644 index 0000000..a2b0b57 --- /dev/null +++ b/deployment/operations/misc/ibosh-cf-credentials.yml @@ -0,0 +1,8 @@ +# Reuse CF credentials from instant-bosh's config-server namespace. The caller +# supplies juicefs_cf_system_domain from the generated CF manifest. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/password + value: ((/instant-bosh/cf/cf_admin_password)) +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/additional_ca_certificates + value: [((/instant-bosh/cf/router_ssl.ca))] diff --git a/deployment/operations/misc/provide-cf-api-ca.yml b/deployment/operations/misc/provide-cf-api-ca.yml new file mode 100644 index 0000000..3c45a7e --- /dev/null +++ b/deployment/operations/misc/provide-cf-api-ca.yml @@ -0,0 +1,5 @@ +# Supply the CA certificate that signs CF's public API route certificate. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/additional_ca_certificates + value: + - ((juicefs_cf_api_ca_certificate)) diff --git a/deployment/operations/misc/skip-cf-api-tls-validation.yml b/deployment/operations/misc/skip-cf-api-tls-validation.yml new file mode 100644 index 0000000..d91e3c2 --- /dev/null +++ b/deployment/operations/misc/skip-cf-api-tls-validation.yml @@ -0,0 +1,4 @@ +# For disposable test environments only. Prefer a trusted CA bundle. +- type: replace + path: /instance_groups/name=juicefs-broker/properties/juicefs/cf/skip_ssl_validation + value: true diff --git a/deployment/operations/non-ha-validation.yml b/deployment/operations/non-ha-validation.yml new file mode 100644 index 0000000..6f7059c --- /dev/null +++ b/deployment/operations/non-ha-validation.yml @@ -0,0 +1,120 @@ +# Disposable validation backends. Each singleton owns a separate persistent disk; +# increasing instances does not create HA or shared object storage. +- type: replace + path: /releases/- + value: + name: postgres + version: '57.0.0' + url: https://bosh.io/d/github.com/cloudfoundry/postgres-release?v=57.0.0 + sha1: sha256:fc5f73f57a6b9e7208e4293a7fb7faf11e4fb8fc8bb79db80d0d97e41ab1ee2a +- type: replace + path: /releases/name=bpm? + value: + name: bpm + version: '1.4.41' + url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 + sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 +- type: replace + path: /releases/- + value: + name: bosh-dns-aliases + version: 0.0.5 + url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-aliases-release?v=0.0.5 + sha1: sha256:a140a3e0aef083c1b356970ddd27fe6a84d7d9272a4dded7c3cb4e55d0e4247e +- type: replace + path: /addons?/name=juicefs-validation-aliases? + value: + name: juicefs-validation-aliases + jobs: + - name: bosh-dns-aliases + release: bosh-dns-aliases + properties: + aliases: + - domain: postgres.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} + - domain: blobstore.service.juicefs.internal + targets: + - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} +- type: replace + path: /instance_groups/0:before + value: + name: juicefs-backends + azs: [z1] + instances: 1 + vm_type: default + stemcell: default + persistent_disk: 51200 + networks: + - name: default + jobs: + - name: postgres + release: postgres + provides: + postgres: {as: postgres, shared: true} + properties: + databases: + databases: + - name: juicefs + roles: + - name: juicefs + password: ((juicefs_postgres_password)) + - name: bpm + release: bpm + - name: blobstore + release: juicefs + provides: + blobstore: {shared: true} + properties: + blobstore: + admin_users: + - username: juicefs + password: ((juicefs_blobstore_password)) + tls: + cert: ((juicefs_blobstore_server_tls.certificate)) + private_key: ((juicefs_blobstore_server_tls.private_key)) + ca: ((juicefs_blobstore_ca.certificate)) + hostname: blobstore.service.juicefs.internal +- type: replace + path: /instance_groups/name=juicefs-broker/jobs/name=juicefs-volume/properties/juicefs/metadata + value: + type: postgres + host: postgres.service.juicefs.internal + port: 5432 + database: juicefs + user: juicefs + password: ((juicefs_postgres_password)) + options: sslmode=disable +- type: replace + path: /instance_groups/name=juicefs-broker/jobs/name=juicefs-volume/properties/juicefs/storage + value: + type: webdav + bucket: https://blobstore.service.juicefs.internal:4443/admin/ + access_key: juicefs + secret_key: ((juicefs_blobstore_password)) + ca_cert: ((juicefs_blobstore_ca.certificate)) +- type: replace + path: /variables/- + value: {name: juicefs_postgres_password, type: password} +- type: replace + path: /variables/- + value: {name: juicefs_blobstore_password, type: password} +- type: replace + path: /variables/- + value: + name: juicefs_blobstore_ca + type: certificate + options: + is_ca: true + common_name: juicefs-blobstore-ca +- type: replace + path: /variables/- + value: + name: juicefs_blobstore_server_tls + type: certificate + update_mode: converge + options: + ca: juicefs_blobstore_ca + common_name: blobstore.service.juicefs.internal + alternative_names: [blobstore.service.juicefs.internal] + extended_key_usage: [server_auth] diff --git a/operations/use-azure-blob-managed-identity.yml b/deployment/operations/use-azure-blob-managed-identity.yml similarity index 100% rename from operations/use-azure-blob-managed-identity.yml rename to deployment/operations/use-azure-blob-managed-identity.yml diff --git a/operations/use-azure-blob.yml b/deployment/operations/use-azure-blob.yml similarity index 100% rename from operations/use-azure-blob.yml rename to deployment/operations/use-azure-blob.yml diff --git a/operations/use-azure-managed-identities.yml b/deployment/operations/use-azure-managed-identities.yml similarity index 82% rename from operations/use-azure-managed-identities.yml rename to deployment/operations/use-azure-managed-identities.yml index 5c00523..7c6c2aa 100644 --- a/operations/use-azure-managed-identities.yml +++ b/deployment/operations/use-azure-managed-identities.yml @@ -1,4 +1,11 @@ -# Apply to manifests/juicefs.yml. Apply the matching runtime-config ops to CF. +# Apply to deployment/juicefs.yml. Apply the matching runtime-config ops to CF. +- type: replace + path: /releases/name=bpm? + value: + name: bpm + version: '1.4.41' + url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 + sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 - type: replace path: /instance_groups/name=juicefs-broker/jobs/- value: diff --git a/operations/use-mysql-s3.yml b/deployment/operations/use-mysql-s3.yml similarity index 100% rename from operations/use-mysql-s3.yml rename to deployment/operations/use-mysql-s3.yml diff --git a/deployment/runtime-configs/README.md b/deployment/runtime-configs/README.md new file mode 100644 index 0000000..f7db8b3 --- /dev/null +++ b/deployment/runtime-configs/README.md @@ -0,0 +1,45 @@ +# Runtime configs + +## JuiceFS volume driver + +[`juicefs-volume-driver.yml`](juicefs-volume-driver.yml) installs the JuiceFS +volume driver on Cloud Foundry Diego cells. It adds the `juicefs-volume-driver` +job to the `compute` instance group in deployment `cf`, enabling cells to mount +the service-instance directories provisioned by the JuiceFS broker. Adjust the +addon's `include` filters if your CF deployment or cell group has another name. + +The driver consumes the shared `juicefs_mount` link from deployment `juicefs`. +That link supplies the filesystem name, backend configuration, credentials, and +trust roots; the cell driver does not need independent backend settings. Deploy +JuiceFS first so this link is available. + +Apply the runtime-config with the explicit uploaded JuiceFS release version: + +```sh +bosh update-config --type=runtime --name=juicefs -n \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -v juicefs_release_version="$release_version" +bosh deploy -n -d cf <(bosh manifest -d cf) +``` + +Updating the runtime-config records the configuration on the director; +redeploying CF installs or updates the addon jobs on the cells. + +### Azure managed identities + +Apply [`operations/use-azure-managed-identities.yml`](operations/use-azure-managed-identities.yml) +to the runtime-config to colocate the Azure PostgreSQL authentication proxy on +each targeted cell and connect the driver's local proxy link. BPM must already +be present on those cells. See [Azure managed identities](../../docs/azure-managed-identities.md) +for backend configuration, required variables, and deployment commands. + +## BOSH DNS prerequisite + +Install BOSH DNS through the upstream +[`bosh-deployment/runtime-configs/dns.yml`](https://github.com/cloudfoundry/bosh-deployment/blob/master/runtime-configs/dns.yml). +This is separate from the JuiceFS volume-driver runtime-config. + +For instant-bosh development environments, DNS is currently embedded only in CF. +Apply [`../operations/misc/ibosh-bosh-dns-fix.yml`](../operations/misc/ibosh-bosh-dns-fix.yml) +to JuiceFS to reuse CF's global DNS certificates. Omit this workaround when the +director already supplies DNS through a runtime-config. diff --git a/runtime-config/juicefs-volume-driver.yml b/deployment/runtime-configs/juicefs-volume-driver.yml similarity index 100% rename from runtime-config/juicefs-volume-driver.yml rename to deployment/runtime-configs/juicefs-volume-driver.yml diff --git a/operations/azure-managed-identities-runtime-config.yml b/deployment/runtime-configs/operations/use-azure-managed-identities.yml similarity index 86% rename from operations/azure-managed-identities-runtime-config.yml rename to deployment/runtime-configs/operations/use-azure-managed-identities.yml index 538cbf3..0cef7a9 100644 --- a/operations/azure-managed-identities-runtime-config.yml +++ b/deployment/runtime-configs/operations/use-azure-managed-identities.yml @@ -1,4 +1,4 @@ -# Apply to runtime-config/juicefs-volume-driver.yml; requires BPM on compute. +# Apply to deployment/runtime-configs/juicefs-volume-driver.yml; requires BPM on compute. - type: replace path: /addons/name=juicefs-volume-driver/jobs/- value: diff --git a/devbox.json b/devbox.json index 1777925..cc1fc9c 100644 --- a/devbox.json +++ b/devbox.json @@ -7,7 +7,7 @@ ], "scripts": { "deploy": [ - "bash -euo pipefail -c 'set -a; source ./bosh.env; set +a; bosh create-release --force && bosh upload-release && release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) && bosh update-config --type=runtime --name=juicefs -n runtime-config/juicefs-volume-driver.yml -v juicefs_release_version=\"$(bosh interpolate \"$release_manifest\" --path /version)\" && bosh -n -d juicefs deploy manifests/juicefs.yml && bosh deploy -n -d cf <(bosh manifest -d cf) -o operations/enable-juicefs-cf-volume-service.yml'" + "bash scripts/deploy" ] } } diff --git a/docs/acceptance-testing.md b/docs/acceptance-testing.md index 593b72b..55f8b3d 100644 --- a/docs/acceptance-testing.md +++ b/docs/acceptance-testing.md @@ -1,104 +1,15 @@ -# Cloud Foundry volume acceptance +# Acceptance testing -`.github/workflows/acceptance.yml` runs on pull requests and manual dispatch. It -creates a disposable Docker-backed [instant-bosh](https://github.com/rkoster/instant-bosh) -director, deploys CF, builds this checkout as a BOSH release, deploys JuiceFS, -updates the CF cells through the named `juicefs` runtime-config, registers the -broker, and runs the `juicefs-acceptance-tests` BOSH errand. +The [acceptance workflow](../.github/workflows/acceptance.yml) runs a disposable +Cloud Foundry environment to check that the JuiceFS volume service works end to +end. -## GitHub setup +It provisions and binds a volume to an app, writes a random value to a file, +restarts the app, and verifies the value is still available. This exercises the +service broker, volume driver, and persistence across app restarts. -The workflow currently uses the standard GitHub-hosted **`ubuntu-latest`** runner; -no runner variable is required. Initial runs will establish how far the full CF -deployment fits within its resources. Diagnostics include host memory, disk, -Docker disk usage, and container statistics to identify capacity limits. -The acceptance job adds an 8 GiB swap file using shell commands, leaving the -runner's existing disk layout intact. +The workflow can also be run locally from a disposable environment. Its commands +are in [`scripts/acceptance/run`](../scripts/acceptance/run). -Local runners need Linux x64, Docker Engine, passwordless sudo, the host FUSE -kernel module/device, `setfacl` (the `acl` package), and support for privileged containers and nested Garden -containers. Use an ephemeral, single-job runner: instant-bosh uses fixed resource -names and ports. - -Startup grants the invoking user a Docker socket ACL: the director's pre-start -hook changes the socket's group to vcap (GID 1000), including on the host through -the bind mount. The ACL preserves runner access after this group change. - -The workflow uses `pull_request` with read-only repository permissions. It does -not need CF credentials: the disposable director generates them. The release's -GCS blobstore must allow anonymous reads of vendored package blobs for fork PRs; -`bosh sync-blobs` fails if those reads are unavailable. Configure public read -access to those release artifacts separately from uploader credentials. - -Tool versions are in `scripts/acceptance/versions.env`. The instant-bosh CLI is -built from a pinned commit because published `v0.3` predates `ibosh cf deploy`. -The director image is pinned by digest. Update these together and verify a full -run. CF release and compiled-stemcell versions come from that CLI's embedded -cf-deployment; the JuiceFS deployment reuses its installed DNS releases and -stemcell. CI uses a 10 GiB backend disk. - -Director startup skips the default `latest` stemcell upload, whose registry -version-tag discovery can be slow. `ibosh cf deploy` uploads the exact stemcell -required by its compiled CF releases instead. - -## What the errand verifies - -The release-owned Go suite uses vendored go-cfclient v3 and Ginkgo/Gomega. It: - -1. Resolves the `shared` service plan under the registered `juicefs` broker. -2. Creates a unique org, space, service instance, app, and route through CAPI. -3. Binds the volume at `/data`, uploads a static Go HTTP app, and stages it with - CF's `binary_buildpack` on `cflinuxfs4` (no runtime downloads in the app). -4. POSTs a random UUID to `/file`; the app writes, syncs, and closes the file. -5. Restarts the app through CAPI and waits for `/identity` to change, proving - requests are reaching a replacement process. -6. GETs `/file` and requires the exact original UUID. -7. Deletes resources in dependency order, awaiting asynchronous CAPI jobs even - after a failed assertion. - -The app endpoints require a per-run bearer token and disable response caching. -The mount directory must already exist; the app never creates a fallback `/data`. -TLS verification uses the Cloud Controller link CA and the CF router CA. - -The errand can be enabled independently on the consolidated CF topology: apply -`operations/enable-juicefs-cf-volume-service.yml` followed by -`operations/enable-juicefs-acceptance-tests.yml`, supplying -`juicefs_release_version` and `juicefs_acceptance_apps_domain`, then run -`bosh -d cf run-errand juicefs-acceptance-tests` after registering the broker. -Its job properties allow choosing another broker name or plan. - -## Local execution - -Install Go 1.25, BOSH CLI, jq, and the pinned instant-bosh CLI; initialize the -release's submodules. On a disposable Linux Docker host, run from the release -root (inside Devbox if Go is not otherwise on PATH): - -```sh -bash scripts/acceptance/run start -bash scripts/acceptance/run deploy-cf -bash scripts/acceptance/run build-release -bash scripts/acceptance/run deploy-service -bash scripts/acceptance/run test -bash scripts/acceptance/run diagnostics -bash scripts/acceptance/run destroy -``` - -The default work directory is ignored `.acceptance/`; override it with -`ACCEPTANCE_WORK_DIR`. It contains private deployment material, so only its -`artifacts/` subdirectory is uploaded by CI. Do not run against an existing -instant-bosh environment you want to retain: `destroy` removes its resources. - -The workflow always attempts diagnostics and destruction after failure. Download -the `juicefs-acceptance--` artifact for JUnit, errand logs, failed -BOSH task debug output, and full CF and JuiceFS logs (including agent and system -logs). The compute VM bundle is collected first so rep and Garden failures are -captured even if later collection times out. Each bundle collection is bounded -to five minutes. The acceptance VM is kept alive -until destruction so its JUnit report can be retrieved. Cancellation or a runner -loss can interrupt cleanup; ephemeral runners reclaim remaining resources. - -`scripts/test-go` includes the app checks and a local TLS CAPI/UAA/router fixture -that executes the real suite. The fixture verifies broker scoping, executable -app upload, restart, and cleanup on both success and deliberate post-restart -data loss. Live acceptance is skipped unless `JUICEFS_ACCEPTANCE_CONFIG` points -to the rendered job JSON. Template checks run with `ruby tests/render_templates.rb`. +On failure, download the acceptance artifact from the GitHub Actions run summary +for test results and diagnostic logs. diff --git a/docs/azure-managed-identities.md b/docs/azure-managed-identities.md index 74d668b..71755e5 100644 --- a/docs/azure-managed-identities.md +++ b/docs/azure-managed-identities.md @@ -105,9 +105,9 @@ The JuiceFS `wasb` bucket hostname starts with the **container**, not the accoun domains can be configured, but the PostgreSQL proxy's token audience is currently hardcoded for public Azure; sovereign-cloud PostgreSQL requires upstream changes. -`operations/use-azure-managed-identities.yml` adds BPM and the proxy to the +`deployment/operations/use-azure-managed-identities.yml` adds BPM and the proxy to the broker group and replaces the mounter's backend settings. The matching -`operations/azure-managed-identities-runtime-config.yml` adds a proxy to each +`deployment/runtime-configs/operations/use-azure-managed-identities.yml` adds a proxy to each targeted CF compute VM. **BPM must already be colocated on compute** (as in cf-deployment); add it in your CF manifest if absent. The proxy shares the host network, as required for loopback clients, DNS, and IMDS. @@ -133,7 +133,7 @@ requires `wasb` and an HTTPS bucket. Custom CA bundles must include all roots needed by that process. Public trust is packaged from pinned certifi sources, not loaded from stemcell certificate files. -For Blob only, apply `operations/use-azure-blob-managed-identity.yml` instead; +For Blob only, apply `deployment/operations/use-azure-blob-managed-identity.yml` instead; no proxy or runtime-config modification is needed. For PostgreSQL only, colocate the proxy/BPM and consume its link while retaining your existing storage object. The proxy's properties live under `azure_postgres_auth_proxy`: `host`, @@ -152,17 +152,35 @@ devbox run -- bosh create-release --force bosh upload-release ``` -Set `release_version` to the explicit version reported by `create-release`, then: +Set `release_version` to the explicit version reported by `create-release` and +`system_domain` to the CF manifest's `system_domain`. Ensure the CF `api.` +route is reachable from the JuiceFS VM, and configure JuiceFS with the CF admin +password and a CA bundle trusted for that public API route. Install BOSH DNS +using your director's standard runtime-config before deploying JuiceFS. + +```sh +system_domain=$(bosh -d cf manifest | bosh interpolate - --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) +bosh -n -d cf deploy <(bosh -d cf manifest) \ + -o deployment/operations/cf/enable-volume-services.yml +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/use-azure-managed-identities.yml \ + -o deployment/operations/misc/provide-cf-api-ca.yml \ + -v juicefs_cf_system_domain="$system_domain" \ + -v juicefs_cf_password="$cf_admin_password" \ + -v juicefs_cf_api_ca_certificate="$cf_api_ca_certificate" \ + --vars-file azure-vars.yml +bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0 +``` + +After JuiceFS exists, update the runtime-config that deploys the driver to CF +cells. Doing this afterward ensures its `juicefs_mount` link is available: ```sh bosh update-config --type=runtime --name=juicefs -n \ - runtime-config/juicefs-volume-driver.yml \ - -o operations/azure-managed-identities-runtime-config.yml \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -o deployment/runtime-configs/operations/use-azure-managed-identities.yml \ --vars-file azure-vars.yml -v juicefs_release_version="$release_version" -bosh -n -d juicefs deploy manifests/juicefs.yml \ - -o operations/use-azure-managed-identities.yml --vars-file azure-vars.yml -bosh deploy -n -d cf <(bosh manifest -d cf) \ - -o operations/enable-juicefs-cf-volume-service.yml +bosh deploy -n -d cf <(bosh -d cf manifest) ``` Use these Azure-specific commands on subsequent deployments too. The default @@ -171,6 +189,28 @@ not the Azure deployment entry point. The bundled PostgreSQL/WebDAV backend group is retained by the example operations; remove unused jobs/resources in your deployment-specific manifest after validating the external backends. +### When using instant-bosh + +The instant-bosh CLI keeps generated CF credentials in its config server and +embeds DNS in CF. Use the following JuiceFS ops to source CF credentials and +reuse the global `/dns_*` certificates already defined by the CF deployment, +instead of supplying the standard-director variables and DNS configuration above: + +```sh +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -o deployment/operations/use-azure-managed-identities.yml \ + --vars-file azure-vars.yml +``` + +The `ibosh-cf-credentials.yml` ops supplies the admin password and router CA from +instant-bosh's config server. Set `juicefs_cf_system_domain` from the generated +CF manifest; TLS verification remains enabled. +The skip-validation ops file is only for disposable tests. If using instant-bosh +for a production deployment, validate the credential lifecycle, CA trust, and +DNS behavior in that environment before relying on it. + Bootstrap initialization now runs in the supervised mount launcher, rather than `pre-start`, so Monit can start the colocated proxy concurrently. It retries `format --no-update` on `spec.bootstrap` and fails if initialization cannot diff --git a/jobs/deregister-broker/spec b/jobs/deregister-broker/spec index 9a79f35..688f33e 100644 --- a/jobs/deregister-broker/spec +++ b/jobs/deregister-broker/spec @@ -14,8 +14,6 @@ consumes: - juicefs.broker.port - juicefs.broker.username - juicefs.broker.password -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.broker.name: description: Service broker registration name. @@ -26,6 +24,8 @@ properties: juicefs.cf.password: description: Cloud Controller administrator password. default: "" + juicefs.cf.system_domain: + description: Cloud Foundry system domain used to construct the public API URL. juicefs.cf.skip_ssl_validation: description: Skip Cloud Controller TLS validation. default: false diff --git a/jobs/deregister-broker/templates/config.json.erb b/jobs/deregister-broker/templates/config.json.erb index ffcf5ce..3736dc5 100644 --- a/jobs/deregister-broker/templates/config.json.erb +++ b/jobs/deregister-broker/templates/config.json.erb @@ -1,9 +1,8 @@ <% require 'json' - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), brokerName: p('juicefs.broker.name'), brokerURL: "http://#{link('juicefs_broker').instances.first.address}:#{link('juicefs_broker').p('juicefs.broker.port')}", brokerPort: link('juicefs_broker').p('juicefs.broker.port'), diff --git a/jobs/juicefs-acceptance-tests/spec b/jobs/juicefs-acceptance-tests/spec index 90a48b8..a6bcd84 100644 --- a/jobs/juicefs-acceptance-tests/spec +++ b/jobs/juicefs-acceptance-tests/spec @@ -8,8 +8,6 @@ packages: - juicefs-acceptance-tests - ca-certificates consumes: -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.acceptance.apps_domain: description: Shared CF application domain used for the test route. @@ -24,6 +22,8 @@ properties: default: admin juicefs.cf.password: description: CF administrator password. + juicefs.cf.system_domain: + description: CF system domain used for the public API and app route. juicefs.cf.additional_ca_certificates: description: Additional CA roots for UAA and application routes. default: [] diff --git a/jobs/juicefs-acceptance-tests/templates/config.json.erb b/jobs/juicefs-acceptance-tests/templates/config.json.erb index bcfc2c2..9b2e9a6 100644 --- a/jobs/juicefs-acceptance-tests/templates/config.json.erb +++ b/jobs/juicefs-acceptance-tests/templates/config.json.erb @@ -1,11 +1,10 @@ <% require 'json' - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", username: p('juicefs.cf.username'), password: p('juicefs.cf.password'), - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), appsDomain: p('juicefs.acceptance.apps_domain'), brokerName: p('juicefs.acceptance.broker_name'), planName: p('juicefs.acceptance.plan_name'), diff --git a/jobs/register-broker/spec b/jobs/register-broker/spec index 1236528..8977046 100644 --- a/jobs/register-broker/spec +++ b/jobs/register-broker/spec @@ -14,8 +14,6 @@ consumes: - juicefs.broker.port - juicefs.broker.username - juicefs.broker.password -- name: cloud_controller_https_endpoint - type: cloud_controller_https_endpoint properties: juicefs.broker.name: description: Service broker registration name. @@ -26,6 +24,8 @@ properties: juicefs.cf.password: description: Cloud Controller administrator password. default: "" + juicefs.cf.system_domain: + description: Cloud Foundry system domain used to construct the public API URL. juicefs.cf.skip_ssl_validation: description: Skip Cloud Controller TLS validation. default: false diff --git a/jobs/register-broker/templates/config.json.erb b/jobs/register-broker/templates/config.json.erb index 7bbbb96..47f04a8 100644 --- a/jobs/register-broker/templates/config.json.erb +++ b/jobs/register-broker/templates/config.json.erb @@ -1,10 +1,9 @@ <% require 'json' broker = link('juicefs_broker') - cc = link('cloud_controller_https_endpoint') config = { - apiURL: "https://#{cc.p('cc.internal_service_hostname')}:#{cc.p('cc.public_tls.port')}", - caCert: ([cc.p('cc.public_tls.ca_cert')] + p('juicefs.cf.additional_ca_certificates')).join("\n"), + apiURL: "https://api.#{p('juicefs.cf.system_domain')}", + caCert: p('juicefs.cf.additional_ca_certificates').join("\n"), brokerName: p('juicefs.broker.name'), brokerURL: "http://#{broker.instances.first.address}:#{broker.p('juicefs.broker.port')}", brokerPort: broker.p('juicefs.broker.port'), diff --git a/manifests/juicefs.yml b/manifests/juicefs.yml deleted file mode 100644 index 0a1fd04..0000000 --- a/manifests/juicefs.yml +++ /dev/null @@ -1,174 +0,0 @@ ---- -name: juicefs - -releases: -- name: juicefs - version: latest -- name: postgres - version: '57.0.0' - url: https://bosh.io/d/github.com/cloudfoundry/postgres-release?v=57.0.0 - sha1: sha256:fc5f73f57a6b9e7208e4293a7fb7faf11e4fb8fc8bb79db80d0d97e41ab1ee2a -- name: bpm - version: '1.4.41' - url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.4.41 - sha1: sha256:7b445508d63d2171d892ba33f836087e1e519b9782f44c5fa5290323616ca8e6 -- name: bosh-dns - version: 1.39.30 -- name: bosh-dns-aliases - version: 0.0.4 - -addons: -- name: juicefs-dns - jobs: - - name: bosh-dns - release: bosh-dns - properties: - configure_systemd_resolved: true - override_nameserver: false - disable_recursors: true - cache: {enabled: true} - api: - client: {tls: ((/dns_api_client_tls))} - server: {tls: ((/dns_api_server_tls))} - health: - enabled: true - client: {tls: ((/dns_healthcheck_client_tls))} - server: {tls: ((/dns_healthcheck_server_tls))} - - name: bosh-dns-aliases - release: bosh-dns-aliases - properties: - aliases: - - domain: postgres.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} - - domain: blobstore.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} - -stemcells: -- alias: default - os: ubuntu-noble - version: latest - -update: - canaries: 1 - max_in_flight: 1 - canary_watch_time: 30000-60000 - update_watch_time: 30000-60000 - -instance_groups: -- name: juicefs-backends - azs: [z1] - instances: 1 - vm_type: default - stemcell: default - persistent_disk: 51200 - networks: - - name: default - jobs: - - name: postgres - release: postgres - provides: - postgres: {as: postgres, shared: true} - properties: - databases: - databases: - - name: juicefs - roles: - - name: juicefs - password: ((juicefs_postgres_password)) - - name: bpm - release: bpm - - name: blobstore - release: juicefs - provides: - blobstore: {shared: true} - properties: - blobstore: - admin_users: - - username: juicefs - password: ((juicefs_blobstore_password)) - tls: - cert: ((juicefs_blobstore_server_tls.certificate)) - private_key: ((juicefs_blobstore_server_tls.private_key)) - ca: ((juicefs_blobstore_ca.certificate)) - hostname: blobstore.service.juicefs.internal -- name: juicefs-broker - migrated_from: - - name: juicefs-mount-broker - az: z1 - azs: [z1] - instances: 1 - vm_type: default - stemcell: default - networks: - - name: default - jobs: - - name: juicefs-volume - release: juicefs - properties: - juicefs: - volume: - initialize: true - metadata: - type: postgres - host: postgres.service.juicefs.internal - port: 5432 - database: juicefs - user: juicefs - password: ((juicefs_postgres_password)) - options: sslmode=disable - storage: - type: webdav - bucket: https://blobstore.service.juicefs.internal:4443/admin/ - access_key: juicefs - secret_key: ((juicefs_blobstore_password)) - ca_cert: ((juicefs_blobstore_ca.certificate)) - provides: - juicefs_mount: {shared: true} - consumes: - juicefs_volume: nil - - name: juicefs-broker - release: juicefs - provides: - juicefs_broker: {shared: true} - properties: - juicefs: - broker: - password: ((juicefs-broker-password)) - plans: - - id: 40bf3e21-885a-4c2a-99b4-d4e8186c467e - name: shared - description: Shared JuiceFS directory without a quota - free: true - capacity_gib: 0 - inodes: 0 - - id: 5d3c1f42-eaa0-4d1b-88ce-b26cf6545cb1 - name: small - description: Shared JuiceFS directory with 1 GiB and 1000 inodes - free: true - capacity_gib: 1 - inodes: 1000 - consumes: - juicefs_mount: {from: juicefs_mount} -variables: -- name: juicefs-broker-password - type: password -- name: juicefs_postgres_password - type: password -- name: juicefs_blobstore_password - type: password -- name: juicefs_blobstore_ca - type: certificate - options: - is_ca: true - common_name: juicefs-blobstore-ca -- name: juicefs_blobstore_server_tls - type: certificate - update_mode: converge - options: - ca: juicefs_blobstore_ca - common_name: blobstore.service.juicefs.internal - alternative_names: - - blobstore.service.juicefs.internal - extended_key_usage: [server_auth] diff --git a/operations/enable-juicefs-cf-volume-service.yml b/operations/enable-juicefs-cf-volume-service.yml deleted file mode 100644 index 4f7beed..0000000 --- a/operations/enable-juicefs-cf-volume-service.yml +++ /dev/null @@ -1,76 +0,0 @@ -- type: replace - path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=blobstore.service.juicefs.internal? - value: - domain: blobstore.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/domain=postgres.service.juicefs.internal? - value: - domain: postgres.service.juicefs.internal - targets: - - {query: '*', instance_group: juicefs-backends, deployment: juicefs, network: default, domain: bosh} -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_worker/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /instance_groups/name=control/jobs/name=cloud_controller_clock/properties/cc/volume_services_enabled? - value: true -- type: replace - path: /instance_groups/name=register-juicefs-broker? - value: - azs: [z1] - instances: 1 - jobs: - - name: register-broker - release: juicefs - consumes: - juicefs_broker: {from: juicefs_broker, deployment: juicefs, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} - lifecycle: errand - name: register-juicefs-broker - networks: [{name: default}] - stemcell: default - vm_type: default - properties: - juicefs: - broker: - name: juicefs - cf: - username: admin - password: ((cf_admin_password)) - additional_ca_certificates: [((router_ssl.ca))] - skip_ssl_validation: false -- type: replace - path: /instance_groups/name=deregister-juicefs-broker? - value: - azs: [z1] - instances: 1 - jobs: - - name: deregister-broker - release: juicefs - consumes: - juicefs_broker: {from: juicefs_broker, deployment: juicefs, ip_addresses: true} - cloud_controller_https_endpoint: {from: cloud_controller_https_endpoint} - lifecycle: errand - name: deregister-juicefs-broker - networks: [{name: default}] - stemcell: default - vm_type: default - properties: - juicefs: - broker: - name: juicefs - cf: - username: admin - password: ((cf_admin_password)) - additional_ca_certificates: [((router_ssl.ca))] - skip_ssl_validation: false -- type: replace - path: /releases/name=juicefs? - value: - name: juicefs - version: latest diff --git a/operations/expand-backend-disk.yml b/operations/expand-backend-disk.yml deleted file mode 100644 index f913aca..0000000 --- a/operations/expand-backend-disk.yml +++ /dev/null @@ -1,3 +0,0 @@ -- type: replace - path: /instance_groups/name=juicefs-backends/persistent_disk - value: 51200 diff --git a/scripts/acceptance/run b/scripts/acceptance/run index aa3bcdc..131fb50 100755 --- a/scripts/acceptance/run +++ b/scripts/acceptance/run @@ -59,22 +59,26 @@ case "${1:-}" in bosh -d cf manifest > "$work/cf.yml" domain=$(bosh interpolate "$work/cf.yml" --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) stemcell=$(bosh interpolate "$work/cf.yml" --path /stemcells/alias=default/version) - dns=$(bosh interpolate "$work/cf.yml" --path /releases/name=bosh-dns/version) aliases=$(bosh interpolate "$work/cf.yml" --path /releases/name=bosh-dns-aliases/version) - bosh -d juicefs deploy manifests/juicefs.yml \ - -o operations/instant-bosh-acceptance.yml \ + bosh -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/non-ha-validation.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -o deployment/operations/misc/ibosh-acceptance.yml \ + -o deployment/operations/misc/skip-cf-api-tls-validation.yml \ + -v juicefs_cf_system_domain="$domain" \ -v juicefs_release_version="$version" \ -v acceptance_stemcell_version="$stemcell" \ - -v acceptance_dns_version="$dns" \ -v acceptance_dns_aliases_version="$aliases" - bosh update-config --type=runtime --name=juicefs runtime-config/juicefs-volume-driver.yml \ + bosh update-config --type=runtime --name=juicefs deployment/runtime-configs/juicefs-volume-driver.yml \ -v juicefs_release_version="$version" bosh -d cf deploy "$work/cf.yml" \ - -o operations/enable-juicefs-cf-volume-service.yml \ - -o operations/enable-juicefs-acceptance-tests.yml \ + -o deployment/operations/cf/non-ha-validation.yml \ + -o deployment/operations/cf/enable-volume-services.yml \ + -o deployment/operations/cf/enable-juicefs-acceptance-tests.yml \ -v juicefs_release_version="$version" \ -v juicefs_acceptance_apps_domain="$domain" - bosh -d cf run-errand register-juicefs-broker --download-logs --logs-dir="$artifacts" + bosh -d juicefs run-errand register-broker --instance=juicefs-broker/0 --download-logs --logs-dir="$artifacts" ;; test) load_env diff --git a/scripts/deploy b/scripts/deploy new file mode 100755 index 0000000..05f5523 --- /dev/null +++ b/scripts/deploy @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +root=$(git rev-parse --show-toplevel) +cd "$root" +set -a +source ./bosh.env +set +a + +bosh create-release --force +bosh upload-release +release_manifest=$(ls -v dev_releases/juicefs/juicefs-*.yml | tail -n 1) +release_version=$(bosh interpolate "$release_manifest" --path /version) + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT +bosh -d cf manifest > "$work/cf.yml" +system_domain=$(bosh interpolate "$work/cf.yml" --path /instance_groups/name=control/jobs/name=cloud_controller_ng/properties/system_domain) + +bosh -n -d juicefs deploy deployment/juicefs.yml \ + -o deployment/operations/non-ha-validation.yml \ + -o deployment/operations/misc/ibosh-bosh-dns-fix.yml \ + -o deployment/operations/misc/ibosh-cf-credentials.yml \ + -v juicefs_cf_system_domain="$system_domain" + +bosh update-config --type=runtime --name=juicefs -n \ + deployment/runtime-configs/juicefs-volume-driver.yml \ + -v juicefs_release_version="$release_version" + +bosh -n -d cf deploy "$work/cf.yml" \ + -o deployment/operations/cf/non-ha-validation.yml \ + -o deployment/operations/cf/enable-volume-services.yml diff --git a/tests/render_templates.rb b/tests/render_templates.rb index c8a1bd8..6b1f3b7 100644 --- a/tests/render_templates.rb +++ b/tests/render_templates.rb @@ -33,25 +33,29 @@ def render(job, template) end end -class AcceptanceTemplatesTest < Minitest::Test - def test_cf_link_and_extra_roots_are_rendered_without_losing_json_escaping - cc = FakeLink.new({ - 'cc.internal_service_hostname' => 'api.service.cf.internal', - 'cc.public_tls.port' => 9024, - 'cc.public_tls.ca_cert' => "cc-root\n" - }) +class CloudControllerLinkTemplatesTest < Minitest::Test + def test_registration_uses_public_api_route_and_preserves_tls_ca_json_escaping properties = { 'juicefs.cf.password' => "secret\"with\nnewlines", + 'juicefs.cf.system_domain' => 'system.example.com', 'juicefs.cf.additional_ca_certificates' => ['router-root'], 'juicefs.acceptance.apps_domain' => 'apps.example.com' } - rendered = RenderContext.new('juicefs-acceptance-tests', properties, {'cloud_controller_https_endpoint' => cc}).render('juicefs-acceptance-tests', 'config.json.erb') - config = JSON.parse(rendered) - assert_equal 'https://api.service.cf.internal:9024', config['apiURL'] - assert_equal "cc-root\n\nrouter-root", config['caCert'] - assert_equal properties['juicefs.cf.password'], config['password'] - assert_equal 'apps.example.com', config['appsDomain'] - assert_equal false, config['skipSSLValidation'] + broker = FakeLink.new({'juicefs.broker.port' => 8080, 'juicefs.broker.username' => 'broker', 'juicefs.broker.password' => 'secret'}) + links = {'juicefs_broker' => broker} + + %w[register-broker deregister-broker].each do |job| + config = JSON.parse(RenderContext.new(job, properties, links).render(job, 'config.json.erb')) + assert_equal 'https://api.system.example.com', config['apiURL'] + assert_equal 'router-root', config['caCert'] + assert_equal properties['juicefs.cf.password'], config['cfPassword'] + end + + links.delete('juicefs_broker') + acceptance = JSON.parse(RenderContext.new('juicefs-acceptance-tests', properties, links).render('juicefs-acceptance-tests', 'config.json.erb')) + assert_equal 'https://api.system.example.com', acceptance['apiURL'] + assert_equal 'router-root', acceptance['caCert'] + assert_equal properties['juicefs.acceptance.apps_domain'], acceptance['appsDomain'] end end