Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a0fbd13
Clean up backup test workspaces after failure (#5135)
JayDoubleu Oct 10, 2026
71bf455
Make one Azure Backup registration attempt before Terraform registers…
JayDoubleu Oct 22, 2026
bd131e1
Try the backup registration warm-up up to five times (#5135)
JayDoubleu Oct 25, 2026
8b21541
Wrap a long CHANGELOG entry (#5135)
JayDoubleu Oct 25, 2026
82083d0
Validate OpenAI model prerequisites and private lifecycle (#5150)
JayDoubleu Oct 10, 2026
32448c7
Create the OpenAI private endpoint after the model deployment (#5249)
JayDoubleu Oct 21, 2026
8b04660
Disable public network access on the MySQL server (#5148)
JayDoubleu Oct 23, 2026
c786c7a
Add a private MySQL lifecycle check (#5148)
JayDoubleu Oct 23, 2026
a3f3b99
Give the MySQL probe script a shebang (#5148)
JayDoubleu Oct 23, 2026
503733b
Run the Linux VM actions in the Guacamole Linux VM check (#5145)
JayDoubleu Oct 23, 2026
f0f9045
Retry the backup registration reads and the unregister of the warm-up…
JayDoubleu Oct 28, 2026
73e5b3e
Use one workspace owner token for the MySQL and Linux VM cleanup (#5148)
JayDoubleu Oct 28, 2026
48fe038
Stop pinning the OpenAI bundle version in the unit tests (#5150)
JayDoubleu Oct 28, 2026
26e0c8b
Use the bundle cloud and login mode in the backup warm-up and wait fo…
JayDoubleu Oct 28, 2026
96661f4
Check the MySQL settings after the SKU change and keep the VM actions…
JayDoubleu Oct 28, 2026
c8b8aae
Stop the backup warm-up when an accepted registration cannot be follo…
JayDoubleu Oct 28, 2026
c210b54
Wait for the backup registration status when no operation is returned…
JayDoubleu Oct 29, 2026
c1ebf9e
Unregister a warm-up registration only after it ended (#5135)
JayDoubleu Oct 29, 2026
7e0d78b
Stop the backup warm-up when a PUT gets no answer from Azure (#5135)
JayDoubleu Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/deploy_tre_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -718,6 +718,8 @@ jobs:
BUNDLE_DIR: "./templates/workspace_services/ohdsi"
- BUNDLE_TYPE: "workspace_service"
BUNDLE_DIR: "./templates/workspace_services/azuresql"
- BUNDLE_TYPE: "workspace_service"
BUNDLE_DIR: "./templates/workspace_services/openai"

environment: ${{ inputs.environmentName }}
steps:
Expand Down Expand Up @@ -1071,6 +1073,10 @@ jobs:
if: inputs.e2eBundle == 'tre-workspace-service-azuresql'
run: echo "SQL_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV"

- name: Reserve MySQL cleanup and reporting time
if: inputs.e2eBundle == 'tre-workspace-service-mysql'
run: echo "MYSQL_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV"

- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
Expand Down Expand Up @@ -1098,6 +1104,7 @@ jobs:
COMMAND: >-
TEST_RUN_CERTIFICATE_TESTS_ON_WEEKENDS=${{ inputs.runCertificateTestsOnWeekends }}
SQL_VALIDATION_DEADLINE=${{ env.SQL_VALIDATION_DEADLINE }}
MYSQL_VALIDATION_DEADLINE=${{ env.MYSQL_VALIDATION_DEADLINE }}
TEST_BUNDLE_REQUEST_FILE=e2e_tests/bundle-validation-request.json make test-e2e-bundle
DEVCONTAINER_TAG: ${{ inputs.DEVCONTAINER_TAG }}
CI_CACHE_ACR_NAME: ${{ secrets.CI_CACHE_ACR_NAME}}
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048))

ENHANCEMENTS:
* Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150))
* Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148))
* Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145))
* Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255))
* Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050))
* Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111))
Expand Down Expand Up @@ -35,6 +38,11 @@ BUG FIXES:
(`core` 0.18.16, `tre-workspace-base` 2.12.3, `tre-workspace-unrestricted` 0.15.2, `tre-workspace-airlock-import-review` 0.18.2) ([#3193](https://github.com/microsoft/AzureTRE/issues/3193))
* Link App Insights to the Azure Monitor private link scope after the private endpoint exists. Before this fix, App Insights rejected all private ingestion with HTTP 403, so the core and workspace App Insights received no telemetry.
Rename the scoped service, so that existing deployments replace the link on upgrade. (`core` 0.18.16, `tre-workspace-base` 2.12.3, `tre-workspace-unrestricted` 0.15.2, `tre-workspace-airlock-import-review` 0.18.2) ([#5280](https://github.com/microsoft/AzureTRE/issues/5280))
* Try the storage registration up to five times before Terraform registers the workspace storage account with Azure Backup. Azure Backup usually fails the first registration in a new vault with an internal error, so most backup-enabled workspaces failed to deploy. Terraform registers after an attempt succeeds, or after five failed attempts.
(`tre-workspace-base` 2.12.4, `tre-workspace-unrestricted` 0.15.3, `tre-workspace-airlock-import-review` 0.18.3) ([#5135](https://github.com/microsoft/AzureTRE/issues/5135))
* Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135))
* Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249))
* Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148))
* Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274))
* Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270))
* Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266))
Expand Down
37 changes: 37 additions & 0 deletions docs/tre-developers/end-to-end-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,25 @@ That case requires an owned workspace so cascade cleanup removes its new service
Some shared-service cases replace existing services, so use an isolated validation environment.
Check the selected entry's prerequisites and limitations before running it.

### OpenAI selection

Select `tre-workspace-service-openai` to run `test_openai.py::test_private_openai_lifecycle`.
The branch workflow registers the OpenAI bundle. The test also replaces its generic case in the `workspace_services` group.
The test requests `gpt-5.1 | 2025-11-13`, private access and one regional `Standard` capacity unit.
It does not switch models or deployment types when Azure cannot satisfy those inputs.

The prerequisite check uses the workspace resource group's actual region and the current subscription model and usage APIs.
It requires a generally available chat model, a current Standard tier and at least one free capacity unit.
It rejects retired model/tier dates, fine-tuning quotas and incomplete or blocked quota data.
The CI identity needs permission to read model and quota data at subscription scope.
The check does not grant roles or reserve capacity. Azure deployment can still fail after it passes.

Prerequisite failures fail the selected test before service creation. They do not count as skipped or passing coverage.
Service creation has a 45-minute deadline. Verification has a separate five-minute deadline.
Failed creation and teardown use the existing helper's separate cleanup deadline.
A configured workspace is retained. A fixture-created workspace uses the fixture's cleanup.
Private inference, access control, upgrades and complete physical cleanup remain separate checks.

### Certificate and Nexus selection

Select `tre-shared-service-certs` to install, uninstall and reinstall the certificate shared service with the same hostname and certificate name.
Expand Down Expand Up @@ -124,6 +143,14 @@ Select `e2eBundle=tre-shared-service-certs` in `deploy_tre_branch.yml` and leave
PR slash commands use workflow definitions from `main`. `/test` runs smoke tests and `/test-shared-services` excludes this certificate-only case.
A test-only PR without the new certificate bundle cannot validate reuse. Run Nexus and Airlock only after this case and its cleanup pass.

### Linux VM selection

Select `tre-service-guacamole-linuxvm` to run `test_guacamole_linuxvm.py::test_create_guacamole_linux_vm`.
The case creates a Linux VM and waits for its Nexus-backed bootstrap. Thus the run needs a suitable Nexus instance or `acceptNexusEula=true`.
The case then runs the `stop`, `start` and `reset_password` actions.
After `stop` and `start`, the case reads the VM power state from Azure. After `reset_password`, it makes sure that the Key Vault secret has a new version.
The case does not sign in to the VM, so it does not prove interactive access or the new password.

### Base workspace lifecycle

Select `tre-workspace-base` to run `test_workspace_base.py::test_base_workspace_lifecycle` independently.
Expand All @@ -149,6 +176,16 @@ Failed-create recovery and temporary probe cleanup also count towards the lifecy
The final 30 minutes are reserved for results and job overhead. Cleanup failures still fail the test.
For a local run, the 270-minute lifecycle budget starts when the SQL case begins.

### MySQL selection

Select `tre-workspace-service-mysql` to run `test_mysql.py::test_private_mysql_query_survives_sku_change`.
The case creates its own workspace, Guacamole service, Linux VM and MySQL service.
The Linux VM installs the MySQL client through Nexus. Thus the run needs a suitable Nexus instance or `acceptNexusEula=true`.
The case writes a row through private DNS with the Burstable SKU. It then changes the SKU to General Purpose and reads the row again.
The VM reads the administrator password from the workspace Key Vault through a temporary role assignment on that secret only.
The MySQL job uses the SQL time limits above with its own `MYSQL_VALIDATION_DEADLINE`. If the case creates Nexus, it removes Nexus after the VM.
These checks do not prove previous-version upgrades or researcher access.

### Run the branch workflow before merge

1. Push the reviewed changes to a branch in `microsoft/AzureTRE`.
Expand Down
2 changes: 1 addition & 1 deletion docs/tre-developers/github-pr-bot-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The table describes the selected assertions. A passing command does not prove ev
| `/test-airlock` | Import requests, access checks, the import review VM flow and export request storage routing | No export review VM or migration proof. |
| `/test-backups` | Base workspace deployment with backups enabled and disabled | No backup restore or retention proof. |

The `workspace_services`, `linux_vm`, `airlock_validation` and `sql_validation` markers have no dedicated slash command.
The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation` and `mysql_validation` markers have no dedicated slash command.
Use the branch workflow or the local test runner for these selections.
The `airlock_validation` marker selects two Airlock cases. It does not select the full `airlock` suite.
See [bundle selection](end-to-end-tests.md#validate-one-bundle) for exact cases and declared coverage gaps.
Expand Down
4 changes: 4 additions & 0 deletions docs/tre-templates/workspace-services/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ See: [MySQL Azure](https://learn.microsoft.com/en-GB/azure/mysql/)

`make workspace_service_bundle BUNDLE=mysql`

## Network access

The MySQL server has public network access disabled. Clients connect through its private endpoint, for example from a VM in the workspace.

## Authenticating to MySQL

1. Navigate to the MySQL workspace service using the `Mysql fqdn` from the details tab.
Expand Down
24 changes: 23 additions & 1 deletion docs/tre-templates/workspace-services/openai.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,28 @@ See: [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/

## Properties
- `is_exposed_externally` - If `True`, the OpenAI workspace is accessible from outside of the workspace virtual network.
- `openai_model` - The model to use for the OpenAI deployment `<model name> | <model version>`. The default is `gpt-35-turbo | 0301`.
- `openai_model` - The model to use for the OpenAI deployment `<model name> | <model version>`. The default is `gpt-5.1 | 2025-11-13`.
- Important note: Models are subject to different quota and region availability and the deployment may fail if you don't have the correct quota.
Please review this link on current limits and how to request increases: [Open AI Quotas](https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits)


## Model prerequisites

The bundle uses the regional `Standard` deployment type with one capacity unit.
Check availability and quota for the exact model version in the workspace region before deployment.
The older model values remain in the schema for existing resource records. Their presence does not mean Azure can deploy them.
Changing the template default does not change the model value stored on an existing service.
An explicit model change can replace the deployment and needs separate upgrade validation.

The OpenAI bundle currently requires the workspace and core in the same subscription.
Private network access remains the default. The bundle does not add researcher RBAC roles or change key authentication.

See [model availability](https://learn.microsoft.com/en-us/rest/api/aiservices/accountmanagement/models/list?view=rest-aiservices-accountmanagement-2024-10-01)
and [subscription quota](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/quota).

## Independent validation

Select `tre-workspace-service-openai` through the [bundle validation procedure](../../tre-developers/end-to-end-tests.md#validate-one-bundle).
The test checks current model and quota prerequisites, deploys a private service, checks its model and network settings, and uninstalls it.
It confirms removal of the API record and active Azure account.
Private inference, researcher access, model upgrades, secret removal and soft-deleted account purge require separate evidence.
42 changes: 32 additions & 10 deletions e2e_tests/bundle_coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -325,10 +325,13 @@
"coverage": [
"install",
"bootstrap-completion",
"stop action with deallocated power state",
"start action with running power state",
"reset_password action with a new Key Vault secret version",
"uninstall"
],
"limitations": [
"Requires a suitable Nexus instance or explicit Nexus EULA consent to create prerequisites. VM availability and interactive access need separate validation."
"Requires a suitable Nexus instance or explicit Nexus EULA consent to create prerequisites. Interactive access, sign-in with the new password and other VM sizes need separate validation."
],
"unavailable_reason": ""
},
Expand Down Expand Up @@ -376,18 +379,27 @@
"manifest": "templates/workspace_services/mysql/porter.yaml",
"family": "workspace_service",
"prerequisites": [
"tre-workspace-base"
"tre-workspace-base",
"tre-service-guacamole",
"tre-service-guacamole-linuxvm",
"tre-shared-service-certs",
"tre-shared-service-sonatype-nexus"
],
"tests": [
"test_mysql.py::test_private_mysql_query_survives_sku_change"
],
"related_tests": [
"test_workspace_services.py::test_install_workspace_service[tre-workspace-service-mysql]"
],
"related_tests": [],
"coverage": [
"install",
"private MySQL write/read",
"Burstable-to-General Purpose SKU change with retained data",
"uninstall"
],
"limitations": [
"Functional, upgrade and custom-action coverage must be assessed separately."
"AzureCloud only. Requires a suitable Nexus instance or explicit Nexus EULA consent, permission to run commands on a test-created workspace VM and a temporary secret-scoped role assignment.",
"The SKU change uses the current bundle version. Previous-version migration remains separate (#5128). The MySQL bundle has no custom actions."
],
"unavailable_reason": ""
},
Expand All @@ -411,15 +423,25 @@
"prerequisites": [
"tre-workspace-base"
],
"tests": [],
"related_tests": [
"test_workspace_services.py::test_install_workspace_service[tre-workspace-service-openai]"
"tests": [
"test_openai.py::test_private_openai_lifecycle"
],
"related_tests": [],
"coverage": [
"model-prerequisites",
"install",
"deployment-status",
"private-network-setting",
"uninstall",
"api-removal",
"active-account-removal"
],
"coverage": [],
"limitations": [
"Functional, upgrade and custom-action coverage must be assessed separately."
"Requires AzureCloud, a registered 1.1.0 or later template, regional Standard model availability and subscription quota read access. Preflight does not reserve Azure capacity.",
"Uses the workspace subscription and actual resource-group region. A workspace in a different subscription from ARM_SUBSCRIPTION_ID is unsupported by this bundle.",
"Does not prove private inference, researcher access, model upgrades, private endpoint connectivity, secret removal or soft-deleted account purge."
],
"unavailable_reason": "The branch deployment does not register this bundle (#5150)."
"unavailable_reason": ""
},
"tre-workspace-airlock-import-review": {
"manifest": "templates/workspaces/airlock-import-review/porter.yaml",
Expand Down
2 changes: 2 additions & 0 deletions e2e_tests/pytest.ini
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ markers =
airlock_validation: draft-container deletion and review-VM flow, including Nexus prerequisites and cleanup
workspace_services
workspace_validation: opt-in base-workspace creation, scoped API access and deletion
openai_validation: opt-in private OpenAI deployment with model and quota preflight
sql_validation: Private SQL query and configuration upgrade lifecycle
mysql_validation: opt-in private MySQL query and SKU change lifecycle from a Nexus-backed Linux VM

asyncio_mode = auto
asyncio_default_fixture_loop_scope = session
Expand Down
Loading
Loading