Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
d5d845c
Record the deployed template version in bundle evidence
JayDoubleu Oct 23, 2026
ddf81ce
Fix CycleCloud VM action outputs (#5159)
JayDoubleu Oct 10, 2026
0c7ae5d
Validate isolated CycleCloud server lifecycle (#5159)
JayDoubleu Oct 10, 2026
d7bd64a
Deploy new CycleCloud servers with the Standard_D4s_v5 VM size
JayDoubleu Oct 22, 2026
70e5590
Ignore the firewall collections of other services in the CycleCloud c…
JayDoubleu Oct 26, 2026
0f60ce4
Repair OHDSI private bootstrap and firewall cleanup (#5149)
JayDoubleu Oct 10, 2026
f76a7f3
Validate private OHDSI lifecycle and bounded cleanup (#5149)
JayDoubleu Oct 10, 2026
2931377
Upload the OHDSI Atlas UI config with the storage account key
JayDoubleu Oct 22, 2026
094aa86
Create the OHDSI web apps after the PostgreSQL subnet writes (#5149)
JayDoubleu Oct 22, 2026
d7f28fc
Declare the Microsoft.Storage endpoint on the OHDSI PostgreSQL subnet
JayDoubleu Oct 22, 2026
916e8b1
Ignore the firewall collections of other services in the OHDSI check …
JayDoubleu Oct 26, 2026
16e2de4
Give the OHDSI Web API identity a role on its own secrets only
JayDoubleu Oct 25, 2026
f9b3e95
Expect the OHDSI roles on the secrets in the E2E test
JayDoubleu Oct 26, 2026
ec75b50
Make sure that the OHDSI Web API resolves its Key Vault references
JayDoubleu Oct 26, 2026
1a10c55
Use one OHDSI version bump
JayDoubleu Oct 30, 2026
23ec913
Fix review points of the CycleCloud and OHDSI change
JayDoubleu Oct 31, 2026
365f248
Make the OHDSI role test fail on the old script and correct two texts…
JayDoubleu Nov 1, 2026
32b0d55
Add network rules to the CycleCloud server
JayDoubleu Nov 2, 2026
e3fb24f
Check the CycleCloud network rules in the E2E test
JayDoubleu Nov 2, 2026
c227777
Describe the CycleCloud network rules and the upgrade step
JayDoubleu Nov 2, 2026
b08d7bd
Use the network address of each cluster range and keep a range until …
JayDoubleu Nov 2, 2026
ffc589b
Name CycleCloud 1.0.0 in place of 0.7.10
JayDoubleu Nov 2, 2026
6dfc56f
Add the issue and pull request links to the new CHANGELOG entries
JayDoubleu Nov 2, 2026
36396b3
Add cluster_address_spaces to the CycleCloud parameter set
JayDoubleu Nov 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/deploy_tre_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1072,6 +1072,13 @@ jobs:
# The job timeout is 300 minutes. Each step below sets an absolute deadline 270 minutes
# (270 * 60 seconds) after the job start. The lifecycle helper keeps the last 120 minutes
# before the deadline for the clean-up. The 30 minutes after the deadline are for the report.
- name: Reserve CycleCloud cleanup and reporting time
if: inputs.e2eBundle == 'tre-shared-service-cyclecloud'
run: echo "CYCLECLOUD_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV"
- name: Reserve OHDSI cleanup and reporting time
if: inputs.e2eBundle == 'tre-workspace-service-ohdsi'
run: echo "OHDSI_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV"

- name: Reserve SQL cleanup and reporting time
if: inputs.e2eBundle == 'tre-workspace-service-azuresql'
run: echo "SQL_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV"
Expand Down Expand Up @@ -1113,6 +1120,8 @@ jobs:
SQL_VALIDATION_DEADLINE=${{ env.SQL_VALIDATION_DEADLINE }}
MYSQL_VALIDATION_DEADLINE=${{ env.MYSQL_VALIDATION_DEADLINE }}
GITEA_VALIDATION_DEADLINE=${{ env.GITEA_VALIDATION_DEADLINE }}
CYCLECLOUD_VALIDATION_DEADLINE=${{ env.CYCLECLOUD_VALIDATION_DEADLINE }}
OHDSI_VALIDATION_DEADLINE=${{ env.OHDSI_VALIDATION_DEADLINE }}
TEST_BUNDLE_REQUEST_FILE=e2e_tests/bundle-validation-request.json make test-e2e-bundle
DEVCONTAINER_TAG: ${{ inputs.DEVCONTAINER_TAG }}
CI_CACHE_ACR_NAME: ${{ secrets.CI_CACHE_ACR_NAME}}
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@
Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048))
* `tre-workspace-service-gitea` 2.0.0 keeps the Git repositories on its file share. The upgrade restarts the web app, so the repositories on the container disk are lost.
Mirror the repositories before the upgrade, and rotate the workspace client secret after it. See the [Gitea upgrade steps](docs/tre-templates/workspace-services/gitea.md#upgrading-to-version-200). The upgrade needs a [force version update](docs/tre-admins/upgrading-resources.md#force-version-update). ([#5283](https://github.com/microsoft/AzureTRE/issues/5283))
* `tre-shared-service-cyclecloud` 1.0.0 adds network rules to the CycleCloud server. After the upgrade, cluster nodes in workspaces cannot connect to the server on TCP port 9443 until you set the new `cluster_address_spaces` property.
The upgrade needs a [force version update](docs/tre-admins/upgrading-resources.md#force-version-update). After the upgrade, update the CycleCloud shared service and add the address space of each workspace that has a cluster to `cluster_address_spaces`.
See the [CycleCloud upgrade steps](docs/tre-templates/shared-services/cyclecloud.md#upgrade-to-version-100). ([#5301](https://github.com/microsoft/AzureTRE/issues/5301))

ENHANCEMENTS:
* Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150))
* Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148))
* Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145))
* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository.
The VM reads the file again after an update, a change of the OIDC client ID, a stop and a start of the app. The check also makes sure that the OIDC change stops Gitea one time and that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141))
* Add an independent CycleCloud server lifecycle with marketplace checks, VM power actions and bounded cleanup. ([#5159](https://github.com/microsoft/AzureTRE/issues/5159))
* Add independent private OHDSI lifecycle validation with callback, firewall, shared DNS preservation and bounded cleanup checks. ([#5149](https://github.com/microsoft/AzureTRE/issues/5149))
* Add a bundle evidence helper that records the deployed template version of a resource that a test gives to it, without the resource properties. ([#5302](https://github.com/microsoft/AzureTRE/pull/5302))
* Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255))
* Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050))
* Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111))
Expand Down Expand Up @@ -56,6 +62,15 @@ BUG FIXES:
The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 2.0.0, `tre-shared-service-gitea` 1.2.6) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283))
* Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283))
* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283))
* Export the CycleCloud VM ID for start and stop actions. (`tre-shared-service-cyclecloud` 1.0.0) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Deploy new CycleCloud servers with the `Standard_D4s_v5` VM size. Azure does not offer `Standard_DS3_v2` to all subscriptions in all regions, for example Switzerland North, so the CycleCloud server failed to deploy there. An upgrade keeps the size of an existing server. (`tre-shared-service-cyclecloud` 1.0.0) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Add network rules to the network card of the CycleCloud server. The rules allow TCP ports 443 and 22 from the core virtual network, and TCP port 9443 from the address spaces in the new `cluster_address_spaces` property, which is empty by default.
The rules deny other inbound traffic from virtual networks. Before this change, the server had the default access of the Azure network rules, which allow inbound traffic from all peered virtual networks. (`tre-shared-service-cyclecloud` 1.0.0) ([#5301](https://github.com/microsoft/AzureTRE/issues/5301))
* Give the OHDSI WebAPI identity the Key Vault role only on the five secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-ohdsi` 0.3.14) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Fix OHDSI database credentials, bound WebAPI startup checks, disable public access and remove owned firewall rules on uninstall. (`tre-workspace-service-ohdsi` 0.3.14) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Upload the OHDSI Atlas UI config with the storage account key, as the share and its web app mount already do. Since azurerm 4, the upload used Azure AD and failed with HTTP 403, because the resource processor has no file data role. (`tre-workspace-service-ohdsi` 0.3.14) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Create the OHDSI Atlas UI web app after the PostgreSQL server, and the WebAPI web app after the Atlas UI web app. Azure rejected the web app subnet link with ReferencedResourceNotProvisioned while a PostgreSQL subnet write ran in the same VNet. (`tre-workspace-service-ohdsi` 0.3.14) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Declare the `Microsoft.Storage` service endpoint on the OHDSI PostgreSQL subnet. The PostgreSQL service adds it during the server create, and each upgrade removed it again. (`tre-workspace-service-ohdsi` 0.3.14) ([#5300](https://github.com/microsoft/AzureTRE/issues/5300))
* Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274))
* Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270))
* Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266))
Expand Down
59 changes: 59 additions & 0 deletions docs/tre-developers/end-to-end-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,66 @@ The **E2E Test Results (Bundle)** artefact contains:

The JSON report copies only selected identifiers and operation states. It excludes resource properties, tokens and failure tracebacks.
Source versions describe the checked-out manifests; verify deployed versions separately when reusing an environment.
A test can call `record_deployed_resource` for a resource that it reads. The report then has a `deployed_template_version` record for that resource.
`passed_selected_cases` means the selected assertions and their teardown passed. The report always leaves `full_lifecycle_proven` false for a separate evidence review.
An entirely skipped selection exits unsuccessfully. Individual skipped cases retain their reasons, and a mixed pass/skip result is marked `partial`.

Unavailable bundles fail with the gap recorded in the map. Adding a selector does not implement missing tests or fix bundle publication and registration gaps.

### CycleCloud server validation

Select `tre-shared-service-cyclecloud` with the branch workflow's bundle input, or use:

```bash
TEST_BUNDLE=tre-shared-service-cyclecloud make test-e2e-bundle
```

Use a dedicated AzureCloud environment with one writer and no existing CycleCloud service or DNS zone.
Publish and register the checkout firewall and CycleCloud bundles. Deploy the firewall before validation.
Register the CycleCloud bundle, version 1.0.0 or later, with `start` and `stop` actions.
Accept the CycleCloud server image terms separately before the run. The test reads the agreement and regional image list. It does not accept terms.
Check `Standard_D4s_v5` capacity and quota in the core region before deployment. Image availability does not prove VM capacity.

The case creates one server and checks its private VM address, DNS configuration, blob private endpoint, managed identity and firewall rules.
It also checks the network security group of the VM network card. By default, the inbound allow rules must be for the core address space and the `AzureLoadBalancer` service tag only, and the lowest-priority rule must deny other `VirtualNetwork` traffic.
It invokes stop/start through TRE and verifies the VM power states. It changes the overview to exercise a same-version Terraform upgrade and checks that the VM identity remains stable.
The same update sets `cluster_address_spaces` to one test range outside the core address space. The test then checks that the TCP 9443 rule allows exactly that range.
The test records the existing firewall as reused. Smoke completes before the bundle job starts.

The job sets `CYCLECLOUD_VALIDATION_DEADLINE` before container setup. The test reserves 120 minutes for cleanup within its 270-minute deadline.
It owns an accepted service before polling. Cleanup waits for terminal operations, verifies ownership and then uses TRE deletion.
It verifies removal of the VM, disk, NIC, network security group, storage, blob private endpoint, DNS zone and captured identity's roles. It also checks that the original firewall rules are restored.
Failed cleanup fails the test. Core Key Vault credentials can remain soft-deleted. The test does not retrieve or purge them.

This case does not configure the CycleCloud application or create an HPC cluster. Browser access, application TLS trust, private DNS reachability and prior-release upgrades remain unproven.
The case reads the network rules from Azure Resource Manager. It does not send traffic from a workspace to the server.
The existing bundle grants subscription Contributor to its VM and permits SharedSubnet access to ARM and GitHub. It does not disable the storage public endpoint.
A private endpoint and a successful server lifecycle do not prove workspace isolation or storage access restrictions.

### Private OHDSI validation

Select the bundle from a branch that contains its repaired bootstrap and validation case:

```bash
TEST_BUNDLE=tre-workspace-service-ohdsi make test-e2e-bundle
```

Use a dedicated AzureCloud environment with one writer and matching deployment and authentication tenants.
Publish and register the checkout base-workspace, firewall and OHDSI bundles. Deploy the firewall before the test.
Check PostgreSQL 14 capacity for `B_Standard_B1ms`, 32 GiB storage and availability zone 1 in the selected region.
The case creates a fresh Automatic workspace. It disables backups and sets `configure_data_source=false` without data-source properties.

The case checks both private web apps, their pinned images and PostgreSQL networking.
It also checks that the WebAPI identity has a role on each of its five Key Vault secrets and no role on the vault, and that each Key Vault reference resolves.
Successful installation includes the bundle's private WebAPI readiness check and local administrator setup.
A metadata update exercises a same-version upgrade. The test checks the callback and firewall collections after each operation.
It waits for terminal operations before updates or deletion. It owns accepted resources before evidence writes or client closure.

The job sets `OHDSI_VALIDATION_DEADLINE` before container setup. The case reserves 120 minutes for cleanup within 270 minutes.
Cleanup removes the service before its workspace. It checks Azure resource removal, app DNS records, callback removal and firewall restoration.
If service cleanup fails, the case retains the workspace and fails. The shared core PostgreSQL DNS link remains in place.
Key Vault items can remain soft-deleted under the existing retention policy.

Browser sign-in, independent private HTTP and DNS probes, database data persistence and previous-version upgrades remain unproven.
Synapse provisioning, CDM queries and clinical workflows need separate cases. Retained non-empty data-source properties are outside this case.
The Synapse script does not currently inspect `configure_data_source`; empty properties avoid that path.
34 changes: 34 additions & 0 deletions docs/tre-templates/shared-services/cyclecloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ az vm image terms accept --urn almalinux:almalinux-hpc:8_5-hpc:latest

Deploy the CycleCloud server using UI or API.

The CycleCloud server uses the `Standard_D4s_v5` VM size. Before you deploy, make sure that your subscription can use this size in the TRE region. An upgrade from a version before 1.0.0 keeps the size of the existing server.

To connect to the CycleCloud server, the TRE Administrator must connect to the CycleCloud server from the administration jumpbox. Use Azure Bastion to connect to the jumpbox a with the username `admin` and the select the password located in your core KeyVault. Connect to the CycleCloud server at the URL: `https://cyclecloud-{TRE_ID}.{LOCATION}.cloudapp.azure.com/`.

- Provide a name for the cyclecloud server instance.
Expand All @@ -37,9 +39,41 @@ To connect to the CycleCloud server, the TRE Administrator must connect to the C

- Hit Save, and then "Back to Clusters"

## Network rules

The CycleCloud server has a network security group on its network card. The network security group has these inbound rules:

| Priority | Source | Ports | Access |
| --- | --- | --- | --- |
| 100 | The address space of the core virtual network | TCP 443 and 22 | Allow |
| 110 | The address spaces in the `cluster_address_spaces` property | TCP 9443 | Allow |
| 120 | The `AzureLoadBalancer` service tag | All | Allow |
| 4096 | The `VirtualNetwork` service tag | All | Deny |

- The admin jumpbox is in the core virtual network. Administrators connect to the CycleCloud server from the jumpbox, as the steps above describe.
- The `VirtualNetwork` service tag includes the peered workspace virtual networks. By default, workspaces cannot connect to the CycleCloud server.
- The bundle creates the rule for TCP port 9443 only when `cluster_address_spaces` has one or more address spaces. The default is an empty list.
- The outbound rules are the Azure default rules.

Cluster nodes connect to the CycleCloud server on TCP port 9443 when the cluster does not use a return proxy. For the ports that CycleCloud uses, see [CycleCloud required ports and traffic](https://learn.microsoft.com/azure/cyclecloud/how-to/network-security?view=cyclecloud-8#required-ports-and-traffic).

To let the nodes of a cluster connect, update the CycleCloud shared service and add the address space of the workspace to `cluster_address_spaces`. Use the CIDR format, for example `10.1.4.0/24`. If a range has host bits, for example `10.1.4.5/24`, the bundle uses its network address, `10.1.4.0/24`.

All clusters in one workspace use the same address space. Keep the address space of a workspace in the list until you delete the last cluster in that workspace.

### Upgrade to version 1.0.0

Version 1.0.0 adds the network rules. After the upgrade, cluster nodes in workspaces cannot connect to the CycleCloud server until you set `cluster_address_spaces`.

1. Find the address space of each workspace that has a CycleCloud cluster. The `address_spaces` property of the workspace has this value.
1. Upgrade the CycleCloud shared service to version 1.0.0. The upgrade needs a [force version update](../../tre-admins/upgrading-resources.md#force-version-update).
1. Update the CycleCloud shared service. Set `cluster_address_spaces` to the address spaces from step 1.

## Create a Cluster
- Before you start creating the cluster retrieve the last 4 digits of the workspace ID that you want to deploy the cluster into.

- Update the CycleCloud shared service. Add the address space of the workspace to `cluster_address_spaces`, so that the cluster nodes can connect to the CycleCloud server. See [Network rules](#network-rules).

- Create a user in CycleCloud as per <https://docs.microsoft.com/en-us/azure/cyclecloud/concepts/user-management?view=cyclecloud-8#adding-new-users-to-cyclecloud> . The SSH key for the user will need to be created within the workspace and public key exporting. We suggest using the 4 digits retrieved in step 1 as part of the user account.

- Select your cluster type, we have tested Slurm and Grid Engine using the methods documented here.
Expand Down
Loading