From f4ff72df0b8e0d6874af997160dc229ad679f76e Mon Sep 17 00:00:00 2001 From: Jay W Date: Mon, 26 Oct 2026 08:42:50 +0000 Subject: [PATCH 01/28] Keep a new Health FHIR service in the Terraform state (#5147) A live run installed a Health service with FHIR. Azure created the FHIR service, but the install failed with "waiting for Fhir Service ... to settle down: unexpected state 'Accepted', wanted target 'Succeeded'". After the create, azurerm 4.58.0 waits until the service reports Succeeded 12 times. The wait accepts only Creating, Updating and Verifying as other states, so a short Accepted state stops it. The provider saves the resource ID only after this wait, so the state did not get the FHIR service. Then the next update failed with "already exists", and the delete failed because Azure does not delete a healthcare workspace that has a FHIR service. azurerm 5.3.0 removed this wait and saves the ID before it polls the create. No 4.x release has this change. The Health bundle now uses azurerm 5.11.0. The 5.0 upgrade guide has no change for the resources of the bundle. The provider block keeps the resource provider registrations of version 4. Terraform removed the unused external provider from the lock file. A new unit test makes sure that the bundle uses azurerm 5.3.0 or later and that the lock file has the same version. The Health documentation tells how to repair a service that has this failure. --- CHANGELOG.md | 1 + .../workspace-services/health_services.md | 7 +++ .../test_health_bundle_providers.py | 28 ++++++++++ .../health-services/porter.yaml | 2 +- .../terraform/.terraform.lock.hcl | 52 ++++++------------- .../health-services/terraform/providers.tf | 5 +- 6 files changed, 57 insertions(+), 38 deletions(-) create mode 100644 e2e_tests/unit_tests/test_health_bundle_providers.py diff --git a/CHANGELOG.md b/CHANGELOG.md index decc945754..5b019932ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,7 @@ BUG FIXES: * Fix Guacamole login failures in large workspaces: raise the user-resources API timeout from 5s to 30s, batch template version lookups into one Cosmos query, and query VM power states concurrently. (`tre-service-guacamole` 0.14.6, `API` 0.27.31) ([#5099](https://github.com/microsoft/AzureTRE/issues/5099)) * Fix API resource update retries so a successful retry returns the patched resource to the original caller. (`API` 0.27.32) ([#5025](https://github.com/microsoft/AzureTRE/pull/5025)) * Fix the health check falsely returning OK when Cosmos is down or inaccessible. ([#4926](https://github.com/microsoft/AzureTRE/issues/4926)) +* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 5.3.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) ## (0.29.1) **BREAKING CHANGES** diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index e93c63999b..3bc98092b0 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -13,6 +13,13 @@ In addition, when creating this workspace service you can choose to deploy [FHIR ![Healthcare Service](images/hs_details.png) +## Install failures before version 0.3.9 + +Before version 0.3.9, an install can fail after Azure creates the FHIR service. +If Azure reports the FHIR service as `Accepted` after the create, the azurerm provider stops its wait, and Terraform does not save the FHIR service. +After this failure, each update and each delete of the Health service fails. +To repair the service, delete its FHIR service in the Azure portal, and then delete the Health service again. + ## Authentication Learn more about authentication and application roles in [this doc](https://learn.microsoft.com/en-us/azure/healthcare-apis/authentication-authorization). Make sure to assign your users/apps with the required role and follow the guidelines to retrieve a token. diff --git a/e2e_tests/unit_tests/test_health_bundle_providers.py b/e2e_tests/unit_tests/test_health_bundle_providers.py new file mode 100644 index 0000000000..4b2b4efe3d --- /dev/null +++ b/e2e_tests/unit_tests/test_health_bundle_providers.py @@ -0,0 +1,28 @@ +"""Check the provider versions of the Health bundle.""" + +from pathlib import Path +import re +import unittest + +ROOT = Path(__file__).resolve().parents[2] +BUNDLE = ROOT / "templates/workspace_services/health-services/terraform" + + +def version(text): + return tuple(int(part) for part in text.split(".")) + + +class HealthBundleProviderTests(unittest.TestCase): + def test_azurerm_keeps_a_new_fhir_service_in_the_state(self): + # Before 5.3.0, azurerm waits for Succeeded after the FHIR create and saves the ID after this + # wait. If Azure reports Accepted, the create fails, and the state does not get the service. + providers = (BUNDLE / "providers.tf").read_text() + lock = (BUNDLE / ".terraform.lock.hcl").read_text() + azurerm = re.search(r'source\s*=\s*"hashicorp/azurerm"\s*\n\s*version\s*=\s*"=\s*([\d.]+)"', providers).group(1) + locked = re.search(r'"registry.terraform.io/hashicorp/azurerm" \{\s*version\s*=\s*"([\d.]+)"', lock).group(1) + self.assertGreaterEqual(version(azurerm), (5, 3, 0)) + self.assertEqual(locked, azurerm) + + +if __name__ == "__main__": + unittest.main() diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 3724596fc8..4dc4af76cd 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.8 +version: 0.3.9 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl index 80913233fb..23c631c6bc 100644 --- a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl +++ b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl @@ -2,43 +2,23 @@ # Manual edits may be lost in future updates. provider "registry.terraform.io/hashicorp/azurerm" { - version = "4.58.0" - constraints = "4.58.0" + version = "5.11.0" + constraints = "5.11.0" hashes = [ - "h1:6lb8gT8vG3vgraxqjePXVqOUUxP6OVOPzz1CxI2Kbj8=", - "h1:MESGDsXHZy6+1pCObte7/WSXaIzH9FUFy/Vp8MPVxiI=", - "h1:bPPD3lFdhTksJqlZlo+piO7OhuX5v9PgCSvWyQRlgPg=", - "zh:041c2a778ab4dd5a9af174b1d6f75409e5aabfc359cb386dfea3fb09e3f32709", - "zh:0a302531a61e7383acf99a6202d7984b2ea559306f45021381665c827a830d46", - "zh:0c69f132c7609683d907e87b89210a298d84c5b0121b62278949931bc54ca952", - "zh:0cadf48e9d2d9daed43212a3c9d886d7faaf68787b6e955456cbe4f43e4a17ec", - "zh:35ef4293d7731f6ff1f8bcba2c4529f987b7fac243c1ac1c154bbc02c9703c25", - "zh:3cb2679e1d56865e0ee0cf4c5d1404dbad0db42d11425e7bf0580a026cc64287", - "zh:4e56411f5119042d4962acff5c6d64224a49a69154ba80e6df63fa57b1e6d284", + "h1:6OVoAYFOZ5OPkpJT5lS4Pt14o5a0LRjJgDPGBt/1VfE=", + "h1:QldFxdezZrp3E1bKnWS7xcU63Zmol1K7siWsXS2S3pg=", + "h1:pZJ3HOERxQUnw4kbPZ0KHRihAIrxiE2+duDOnxFrJgg=", + "zh:2345a6104bb485141afb8f0047a48a5f27c3bb4e6eeb360c7955f381227d88dd", + "zh:34e86c5ceeba746fed119775cd37d7314ea8b7c2196ed8b8da8d8d71e10b96df", + "zh:358f30581df0670719fc8511066ef9d05eb2612934c147bfe1b1b1d67712d2f0", + "zh:3edb6681e17535f16ad1f2b7588325d86d7af57df9eed82fa4a607fc03e8afc7", + "zh:6a5b9d6dd86a759f8bfd81e15e7e688531c85ee9660bd5923f99e993a1f15c62", "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:ca4626411a111720c220f9849c7d2e1fcd5d380f56459e096d835a9dbf9e6e13", - "zh:d31c4e65dcb096974479b2d548fffb86fc9a5262aff1b01fe62ef442ce536c6b", - "zh:d9631602999c1853e53ee2c5aef7476e23c7787beddc3599c10dbaa4891ba166", - "zh:f31ba7c9341037ceb7d49467946c01b2b0930404ed1d5643c1451f734a613a03", - ] -} - -provider "registry.terraform.io/hashicorp/external" { - version = "2.3.5" - constraints = "2.3.5" - hashes = [ - "h1:smKSos4zs57pJjQrNuvGBpSWth2el9SgePPbPHo0aps=", - "zh:6e89509d056091266532fa64de8c06950010498adf9070bf6ff85bc485a82562", - "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:86868aec05b58dc0aa1904646a2c26b9367d69b890c9ad70c33c0d3aa7b1485a", - "zh:a2ce38fda83a62fa5fb5a70e6ca8453b168575feb3459fa39803f6f40bd42154", - "zh:a6c72798f4a9a36d1d1433c0372006cc9b904e8cfd60a2ae03ac5b7d2abd2398", - "zh:a8a3141d2fc71c86bf7f3c13b0b3be8a1b0f0144a47572a15af4dfafc051e28a", - "zh:aa20a1242eb97445ad26ebcfb9babf2cd675bdb81cac5f989268ebefa4ef278c", - "zh:b58a22445fb8804e933dcf835ab06c29a0f33148dce61316814783ee7f4e4332", - "zh:cb5626a661ee761e0576defb2a2d75230a3244799d380864f3089c66e99d0dcc", - "zh:d1acb00d20445f682c4e705c965e5220530209c95609194c2dc39324f3d4fcce", - "zh:d91a254ba77b69a29d8eae8ed0e9367cbf0ea6ac1a85b58e190f8cb096a40871", - "zh:f6592327673c9f85cdb6f20336faef240abae7621b834f189c4a62276ea5db41", + "zh:a470dc95f15dcded2e06dd45c44c3b50a22a45b87c9ba7bd19dc6b25022660b7", + "zh:b179f04830c1d8d7ec15ae4be4434042d87679bb23b7d60d7ebd751420bf646c", + "zh:b183835acbcf39047d195f933caebd1902065d2437df17e964ceb4a358441f41", + "zh:b3d3555ac1f16a59baaab894fe9e839baedc809aef05a6d8e6c0cdaba9c92749", + "zh:cc29f59113d468757e9f1a3db774c73d8827f8359816216b7b82ddef5ff02d66", + "zh:e37d3649320c50110675156174f476b3b12ca5c1f267f1cadb56ad26e7f3553e", ] } diff --git a/templates/workspace_services/health-services/terraform/providers.tf b/templates/workspace_services/health-services/terraform/providers.tf index fc3dcaeab4..b2f9f931fc 100644 --- a/templates/workspace_services/health-services/terraform/providers.tf +++ b/templates/workspace_services/health-services/terraform/providers.tf @@ -1,9 +1,10 @@ # Azure Provider source and version being used terraform { required_providers { + # Before 5.3.0, a FHIR create that Azure reports as Accepted fails, and the state does not get the service. azurerm = { source = "hashicorp/azurerm" - version = "= 4.58.0" + version = "= 5.11.0" } } @@ -11,6 +12,8 @@ terraform { } provider "azurerm" { + # Version 5 registers no resource providers by default. Keep the registrations of version 4. + resource_provider_registrations = "legacy" features { key_vault { # Don't purge on destroy (this would fail due to purge protection being enabled on keyvault) From 52919cb82c45043df232512a12a9f539c7d0b33f Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 14:38:31 +0100 Subject: [PATCH 02/28] Add a private Health lifecycle check (#5147) The Health bundle selection ran only the generic install test with the default inputs, so it deployed no FHIR or DICOM service. The new case deploys FHIR R4 with a Windows VM in the same workspace. From the VM, it makes sure that the FHIR host resolves privately and that the FHIR metadata answers. FHIR must reject a data request without a token with HTTP 401, and from the VM identity without a role with HTTP 403. The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service, writes a Patient, adds DICOM with an update and reads the Patient again. The DICOM host must also be private and need a token. The case checks the public network access of the Health workspace, FHIR and DICOM last, so that a failure there keeps the functional evidence. --- .github/workflows/deploy_tre_reusable.yml | 4 + CHANGELOG.md | 1 + docs/tre-developers/end-to-end-tests.md | 10 ++ docs/tre-developers/github-pr-bot-commands.md | 1 + e2e_tests/bundle_coverage.json | 19 ++- e2e_tests/pytest.ini | 1 + e2e_tests/resources/health_probe.ps1 | 92 +++++++++++++ e2e_tests/resources/health_probe.py | 93 ++++++++++++++ e2e_tests/test_health.py | 111 ++++++++++++++++ e2e_tests/unit_tests/test_health_probe.py | 121 ++++++++++++++++++ 10 files changed, 448 insertions(+), 5 deletions(-) create mode 100644 e2e_tests/resources/health_probe.ps1 create mode 100644 e2e_tests/resources/health_probe.py create mode 100644 e2e_tests/test_health.py create mode 100644 e2e_tests/unit_tests/test_health_probe.py diff --git a/.github/workflows/deploy_tre_reusable.yml b/.github/workflows/deploy_tre_reusable.yml index 65ebe15392..5f00fe5509 100644 --- a/.github/workflows/deploy_tre_reusable.yml +++ b/.github/workflows/deploy_tre_reusable.yml @@ -1083,6 +1083,9 @@ jobs: - name: Reserve Gitea cleanup and reporting time if: inputs.e2eBundle == 'tre-workspace-service-gitea' run: echo "GITEA_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" + - name: Reserve Health cleanup and reporting time + if: inputs.e2eBundle == 'tre-workspace-service-health' + run: echo "HEALTH_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" - name: Checkout uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 @@ -1113,6 +1116,7 @@ jobs: SQL_VALIDATION_DEADLINE=${{ env.SQL_VALIDATION_DEADLINE }} MYSQL_VALIDATION_DEADLINE=${{ env.MYSQL_VALIDATION_DEADLINE }} GITEA_VALIDATION_DEADLINE=${{ env.GITEA_VALIDATION_DEADLINE }} + HEALTH_VALIDATION_DEADLINE=${{ env.HEALTH_VALIDATION_DEADLINE }} TEST_BUNDLE_REQUEST_FILE=e2e_tests/bundle-validation-request.json make test-e2e-bundle DEVCONTAINER_TAG: ${{ inputs.DEVCONTAINER_TAG }} CI_CACHE_ACR_NAME: ${{ secrets.CI_CACHE_ACR_NAME}} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b019932ae..aff8d22c80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Setting to `false` will delete existing airlock storage accounts and must only b ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) +* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) * Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 306f4ca158..7755ef00b2 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -204,6 +204,16 @@ The last start has the same OIDC settings, so the log must show only one stop fo The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. +### Health selection + +Select `tre-workspace-service-health` to run `test_health.py::test_private_fhir_access_and_dicom_update`. +The case creates its own workspace, Guacamole service, Windows VM and Health service with FHIR R4. +From the VM, the case makes sure that the FHIR host resolves to a private address and that the FHIR metadata is available. +It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). +The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. +An update adds DICOM. The case reads the Patient again and makes sure that the DICOM host is private and needs a token. +At the end, the case makes sure that the Health workspace, FHIR and DICOM have public network access disabled. +The Health job uses the SQL time limits with its own `HEALTH_VALIDATION_DEADLINE`. ### Run the branch workflow before merge diff --git a/docs/tre-developers/github-pr-bot-commands.md b/docs/tre-developers/github-pr-bot-commands.md index da6f973b54..61da508abc 100644 --- a/docs/tre-developers/github-pr-bot-commands.md +++ b/docs/tre-developers/github-pr-bot-commands.md @@ -23,6 +23,7 @@ The table describes the selected assertions. A passing command does not prove ev | `/test-backups` | Base workspace deployment with backups enabled and disabled | No backup restore or retention proof. | The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `gitea_validation` markers have no dedicated slash command. +The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `health_validation` markers have no dedicated slash command. Use the branch workflow or the local test runner for these selections. The `airlock_validation` marker selects two Airlock cases. It does not select the full `airlock` suite. See [bundle selection](end-to-end-tests.md#validate-one-bundle) for exact cases and declared coverage gaps. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index 49194451cd..e395cef64c 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -372,19 +372,28 @@ "manifest": "templates/workspace_services/health-services/porter.yaml", "family": "workspace_service", "prerequisites": [ - "tre-workspace-base" + "tre-workspace-base", + "tre-service-guacamole", + "tre-service-guacamole-windowsvm" ], "tests": [ + "test_health.py::test_private_fhir_access_and_dicom_update" + ], + "related_tests": [ "test_workspace_services.py::test_install_workspace_service[tre-workspace-service-health]" ], - "related_tests": [], "coverage": [ - "install", + "install with FHIR R4", + "private FHIR metadata", + "anonymous and unassigned FHIR requests denied", + "private FHIR write/read with a temporary data role", + "update that adds DICOM with retained FHIR data", + "public network access disabled", "uninstall" ], "limitations": [ - "Functional, upgrade and custom-action coverage must be assessed separately.", - "Default inputs do not enable FHIR or DICOM. Endpoint validation requires an explicit configuration." + "AzureCloud only. Requires permission to run commands on a test-created workspace VM and create a temporary service-scoped role assignment.", + "The case does not use DICOM data, FHIR STU3 or researcher identities. The update uses the current bundle version. Previous-version migration remains separate (#5128)." ], "unavailable_reason": "" }, diff --git a/e2e_tests/pytest.ini b/e2e_tests/pytest.ini index ea391b6c83..f037ef6fc2 100644 --- a/e2e_tests/pytest.ini +++ b/e2e_tests/pytest.ini @@ -19,6 +19,7 @@ markers = sql_validation: Private SQL query and configuration upgrade lifecycle mysql_validation: opt-in private MySQL query and SKU change lifecycle from a Nexus-backed Linux VM gitea_validation: opt-in private Gitea repository write/read across an update and an app restart + health_validation: opt-in private FHIR access control, write/read and DICOM update lifecycle asyncio_mode = auto asyncio_default_fixture_loop_scope = session diff --git a/e2e_tests/resources/health_probe.ps1 b/e2e_tests/resources/health_probe.ps1 new file mode 100644 index 0000000000..2b5f35444b --- /dev/null +++ b/e2e_tests/resources/health_probe.ps1 @@ -0,0 +1,92 @@ +param( + [Parameter(Mandatory)][ValidatePattern('^https://[a-z0-9-]+\.fhir\.azurehealthcareapis\.com$')][string]$FhirUrl, + [ValidatePattern('^(https://[a-z0-9-]+\.dicom\.azurehealthcareapis\.com)?$')][string]$DicomUrl = '', + [Parameter(Mandatory)][ValidatePattern('^[a-f0-9]{32}$')][string]$ProbeId, + [Parameter(Mandatory)][ValidateSet('access', 'write', 'read')][string]$Phase +) +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 +$stage = 'private-dns' + +function Assert-PrivateDns([string]$Url) { + $addresses = [System.Net.Dns]::GetHostAddresses(([uri]$Url).Host) + if ($addresses.Count -eq 0) { throw 'No address' } + foreach ($address in $addresses) { + $bytes = $address.GetAddressBytes() + $private = $bytes.Length -eq 4 -and ( + $bytes[0] -eq 10 -or + ($bytes[0] -eq 172 -and $bytes[1] -ge 16 -and $bytes[1] -le 31) -or + ($bytes[0] -eq 192 -and $bytes[1] -eq 168) + ) + if (-not $private) { throw 'DNS is not private' } + } +} + +function Get-Status([string]$Uri, [hashtable]$Headers = @{}) { + try { + return [int](Invoke-WebRequest -UseBasicParsing -Uri $Uri -Headers $Headers -TimeoutSec 30).StatusCode + } catch { + # Windows PowerShell and PowerShell 7 use different exception types, but both have a Response. + $response = $_.Exception.Response + if ($null -eq $response) { throw } + return [int]$response.StatusCode + } +} + +function Get-FhirHeaders { + $resource = [uri]::EscapeDataString($FhirUrl) + $identity = Invoke-RestMethod -Headers @{Metadata = 'true'} -Uri "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=$resource" -TimeoutSec 15 + return @{Authorization = "Bearer $($identity.access_token)"} +} + +function Get-ProbeCount([hashtable]$Headers) { + $search = Invoke-RestMethod -Headers $Headers -Uri "$FhirUrl/Patient?identifier=urn:tre:e2e|$ProbeId" -TimeoutSec 30 + if ($search.resourceType -ne 'Bundle') { throw 'Unexpected search result' } + return @($search.entry).Where({ $null -ne $_ }).Count +} + +try { + Assert-PrivateDns $FhirUrl + $result = [ordered]@{phase = $Phase; probe = $ProbeId; private_dns = $true} + if ($Phase -eq 'access') { + $stage = 'fhir-metadata' + $metadata = Invoke-RestMethod -Uri "$FhirUrl/metadata" -TimeoutSec 30 + if ($metadata.resourceType -ne 'CapabilityStatement') { throw 'Unexpected metadata' } + $result.fhir_version = [string]$metadata.fhirVersion + $stage = 'fhir-anonymous' + $result.anonymous = Get-Status "$FhirUrl/Patient?_count=1" + $stage = 'fhir-unassigned' + $result.unassigned = Get-Status "$FhirUrl/Patient?_count=1" (Get-FhirHeaders) + } else { + $stage = 'fhir-role' + # A new role assignment can take some minutes to apply. + $deadline = [DateTime]::UtcNow.AddMinutes(10) + while ((Get-Status "$FhirUrl/Patient?_count=1" (Get-FhirHeaders)) -ne 200) { + if ([DateTime]::UtcNow -ge $deadline) { throw 'Role did not apply' } + Start-Sleep -Seconds 20 + } + $headers = Get-FhirHeaders + if ($Phase -eq 'write') { + $stage = 'fhir-write' + $patient = @{resourceType = 'Patient'; identifier = @(@{system = 'urn:tre:e2e'; value = $ProbeId})} | ConvertTo-Json -Depth 4 + $created = Invoke-RestMethod -Method Post -Headers $headers -ContentType 'application/fhir+json' -Uri "$FhirUrl/Patient" -Body $patient -TimeoutSec 30 + if ($created.resourceType -ne 'Patient') { throw 'Unexpected create result' } + } + $stage = 'fhir-read' + $result.count = Get-ProbeCount $headers + if ($DicomUrl) { + $stage = 'dicom-access' + Assert-PrivateDns $DicomUrl + $result.dicom_anonymous = Get-Status "$DicomUrl/v2/studies" + } + } + Write-Output ('HEALTH_PROBE_RESULT=' + ($result | ConvertTo-Json -Compress)) +} catch { + # Do not emit exception details or tokens. + Write-Output "HEALTH_PROBE_FAILED=$stage" + exit 1 +} finally { + $identity = $null + $headers = $null +} diff --git a/e2e_tests/resources/health_probe.py b/e2e_tests/resources/health_probe.py new file mode 100644 index 0000000000..7708b89889 --- /dev/null +++ b/e2e_tests/resources/health_probe.py @@ -0,0 +1,93 @@ +"""Run a private FHIR probe from a test-created workspace Windows VM.""" + +import asyncio +from contextlib import asynccontextmanager +import json +from pathlib import Path +from uuid import uuid4 + +from e2e_tests.resources.sql_probe import COMPUTE_API, ROLE_API, delete_after + +SCRIPT = Path(__file__).with_name("health_probe.ps1") +RESULT = "HEALTH_PROBE_RESULT=" +FAILED = "HEALTH_PROBE_FAILED=" +FHIR_DATA_CONTRIBUTOR = "5a1fc7df-4bf1-4951-a576-89034ee01acd" + + +def expected_result(phase, probe_id, dicom): + expected = {"phase": phase, "probe": probe_id, "private_dns": True} + if phase == "access": + return {**expected, "fhir_version": "4.0.1", "anonymous": 401, "unassigned": 403} + expected["count"] = 1 + if dicom: + expected["dicom_anonymous"] = 401 + return expected + + +def probe_result(resource, phase, probe_id, dicom=False): + properties = resource.get("properties", {}) + view = properties.get("instanceView", {}) + state = view.get("executionState", "") + lines = view.get("output", "").splitlines() + if properties.get("provisioningState") in ("Failed", "Canceled") or state in ("Failed", "Canceled", "TimedOut"): + stages = [line[len(FAILED) :] for line in lines if line.startswith(FAILED)] + raise AssertionError( + f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, stage={','.join(stages) or 'unknown'}" + ) + if state != "Succeeded": + return False + if view.get("exitCode") != 0: + raise AssertionError("Health probe returned a non-zero exit code") + results = [line[len(RESULT) :] for line in lines if line.startswith(RESULT)] + if len(results) != 1: + raise AssertionError("Health probe returned no single result") + actual = json.loads(results[0]) + if actual != expected_result(phase, probe_id, dicom): + raise AssertionError(f"Health probe result does not match: {actual}") + return True + + +@asynccontextmanager +async def role_access(arm, scope, role, principal_id, subscription_id): + """Grant only this VM a data role on one service, then remove the assignment.""" + assignment = f"{scope}/providers/Microsoft.Authorization/roleAssignments/{uuid4()}" + body = { + "properties": { + "roleDefinitionId": f"/subscriptions/{subscription_id}/providers/Microsoft.Authorization/roleDefinitions/{role}", + "principalId": principal_id, + "principalType": "ServicePrincipal", + } + } + # The PUT may have succeeded even if its response was lost. + async with delete_after(arm, assignment, ROLE_API): + await arm.request("PUT", assignment, ROLE_API, body=body) + yield + + +async def run_probe(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): + parameters = {"FhirUrl": fhir_url, "ProbeId": probe_id, "Phase": phase} + if dicom_url: + parameters["DicomUrl"] = dicom_url + command = f"{vm_id}/runCommands/health-probe-{uuid4().hex}" + # Removing our command also terminates it if its deadline expired. + async with delete_after(arm, command, COMPUTE_API): + async with asyncio.timeout(20 * 60): + await arm.request( + "PUT", + command, + COMPUTE_API, + body={ + "location": location, + "properties": { + "source": {"script": SCRIPT.read_text()}, + "parameters": [{"name": k, "value": v} for k, v in parameters.items()], + "timeoutInSeconds": 900, + "treatFailureAsDeploymentFailure": True, + }, + }, + ) + while True: + result = await arm.request("GET", command, COMPUTE_API, expand=True) + if probe_result(result, phase, probe_id, bool(dicom_url)): + return + await asyncio.sleep(10) diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py new file mode 100644 index 0000000000..8a72af6da7 --- /dev/null +++ b/e2e_tests/test_health.py @@ -0,0 +1,111 @@ +"""Exercise private FHIR access, access control and a DICOM update.""" + +import asyncio +from functools import partial +import logging +import os +from uuid import uuid4 + +import pytest + +from e2e_tests import config +from e2e_tests.conftest import get_workspace_owner_token +from e2e_tests.resources import strings +from e2e_tests.resources.health_probe import FHIR_DATA_CONTRIBUTOR, role_access, run_probe +from e2e_tests.resources.resource import get_resource, post_resource +from e2e_tests.resources.sql_lifecycle import sql_lifecycle +from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, validate_vm + +LOGGER = logging.getLogger(__name__) +HEALTH_API = "2024-03-31" +HEALTH_PAYLOAD = { + "templateName": strings.HEALTH_SERVICE, + "properties": { + "display_name": "Health lifecycle test", + "description": "Private FHIR and DICOM validation", + "deploy_fhir": True, + "fhir_kind": "R4", + "deploy_dicom": False, + }, +} +VM_PAYLOAD = { + "templateName": strings.GUACAMOLE_WINDOWS_USER_RESOURCE, + "properties": { + "display_name": "Health test client", + "description": "Temporary private FHIR validation client", + "os_image": "Windows 11", + "vm_size": "2 CPU | 8GB RAM", + "admin_username": "researcher", + "shared_storage_access": False, + **{ + f"install_{tool}": False + for tool in ("azure_cli", "vscode", "storage_explorer", "git", "python_tools", "r_tools") + }, + }, +} + + +@pytest.mark.asyncio(loop_scope="session") +@pytest.mark.health_validation +@pytest.mark.timeout(285 * 60) +async def test_private_fhir_access_and_dicom_update(verify): + if os.environ.get("AZURE_ENVIRONMENT", "AzureCloud") != "AzureCloud": + raise ValueError("The health probe currently supports AzureCloud only") + async with sql_lifecycle(verify, deadline_variable="HEALTH_VALIDATION_DEADLINE") as resources: + workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup() + new_token = partial(get_workspace_owner_token, workspace_id, verify) + health_path, health_id = await resources.create( + HEALTH_PAYLOAD, f"/api{workspace_path}/workspace-services", await new_token(), cleanup_token=new_token + ) + vm_path, vm_id = await resources.create( + VM_PAYLOAD, f"/api{guacamole_path}/user-resources", await new_token(), cleanup_token=new_token + ) + token = await new_token() + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] + vm_properties = (await get_resource(f"/api{vm_path}", token, verify))["userResource"]["properties"] + subscription_id = vm_properties.get("workspace_subscription_id") or os.environ["ARM_SUBSCRIPTION_ID"] + suffix = f"{workspace_id[-4:]}svc{health_id[-4:]}" + fhir_url = f"https://hs{suffix}-fhir{suffix}.fhir.azurehealthcareapis.com" + dicom_url = f"https://hs{suffix}-dicom{suffix}.dicom.azurehealthcareapis.com" + assert health["properties"]["fhir_url"] == fhir_url + assert not health["properties"].get("dicom_url") + azure_vm_id = vm_properties["azure_resource_id"] + async with arm_client() as arm: + vm = await arm.request("GET", azure_vm_id, COMPUTE_API) + group = validate_vm(vm, azure_vm_id, subscription_id, config.TRE_ID, workspace_id, guacamole_id, vm_id) + workspace_arm_id = f"{group}/providers/Microsoft.HealthcareApis/workspaces/hs{suffix}" + fhir_id = f"{workspace_arm_id}/fhirservices/fhir{suffix}" + fhir = await arm.request("GET", fhir_id, HEALTH_API) + assert fhir["properties"]["provisioningState"] == "Succeeded" + probe_id = uuid4().hex + probe = partial(run_probe, arm, azure_vm_id, vm["location"], fhir_url=fhir_url, probe_id=probe_id) + await probe(phase="access") + LOGGER.info("FHIR metadata is private, and data requests need a token and a role") + async with role_access(arm, fhir_id, FHIR_DATA_CONTRIBUTOR, vm["identity"]["principalId"], subscription_id): + await probe(phase="write") + token = await new_token() + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] + async with asyncio.timeout(60 * 60): + await post_resource( + {"properties": {"deploy_dicom": True}}, + f"/api{health_path}", + token, + verify, + method="PATCH", + etag=health["_etag"], + ) + health = (await get_resource(f"/api{health_path}", await new_token(), verify))["workspaceService"] + assert health["properties"]["dicom_url"] == dicom_url + dicom = await arm.request("GET", f"{workspace_arm_id}/dicomservices/dicom{suffix}", HEALTH_API) + assert dicom["properties"]["provisioningState"] == "Succeeded" + await probe(phase="read", dicom_url=dicom_url) + LOGGER.info("FHIR data survived the update that added DICOM") + # Check the network settings last, so that a failure here keeps the functional evidence above. + workspace_record = await arm.request("GET", workspace_arm_id, HEALTH_API) + network = { + "workspace": workspace_record["properties"].get("publicNetworkAccess"), + "fhir": fhir["properties"].get("publicNetworkAccess"), + "dicom": dicom["properties"].get("publicNetworkAccess"), + } + LOGGER.info("Health public network access: %s", network) + assert network == dict.fromkeys(network, "Disabled"), f"Health services allow public access: {network}" diff --git a/e2e_tests/unit_tests/test_health_probe.py b/e2e_tests/unit_tests/test_health_probe.py new file mode 100644 index 0000000000..acd1cbd038 --- /dev/null +++ b/e2e_tests/unit_tests/test_health_probe.py @@ -0,0 +1,121 @@ +"""Check the health probe's evidence, role scope and cleanup boundaries.""" + +import json +from pathlib import Path +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock + +from jsonschema import validate +import yaml + +from api_app.services.schema_service import enrich_user_resource_template, enrich_workspace_service_template +from e2e_tests import test_health as health +from e2e_tests.resources import health_probe as probe + + +ROOT = Path(__file__).resolve().parents[2] +GROUP = "/subscriptions/sub/resourceGroups/rg-tretest-ws-abcd" +VM_ID = f"{GROUP}/providers/Microsoft.Compute/virtualMachines/windowsvm1234" +FHIR_ID = f"{GROUP}/providers/Microsoft.HealthcareApis/workspaces/hsabcdsvc1234/fhirservices/fhirabcdsvc1234" +FHIR_URL = "https://hsabcdsvc1234-fhirabcdsvc1234.fhir.azurehealthcareapis.com" +PROBE_ID = "0123456789abcdef0123456789abcdef" + + +def result(evidence, state="Succeeded", code=0, output=None): + return { + "properties": { + "provisioningState": "Succeeded", + "instanceView": { + "executionState": state, + "exitCode": code, + "output": output if output is not None else "HEALTH_PROBE_RESULT=" + json.dumps(evidence), + }, + } + } + + +class ProbeEvidenceTests(unittest.TestCase): + def test_access_phase_requires_private_metadata_and_denied_requests(self): + evidence = probe.expected_result("access", PROBE_ID, False) + self.assertEqual(evidence["anonymous"], 401) + self.assertEqual(evidence["unassigned"], 403) + self.assertTrue(probe.probe_result(result(evidence), "access", PROBE_ID)) + for change in ({"anonymous": 200}, {"unassigned": 200}, {"private_dns": False}, {"fhir_version": "3.0.2"}): + with self.subTest(change=change), self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, **change}), "access", PROBE_ID) + + def test_data_phases_require_the_written_row(self): + for phase, dicom in (("write", False), ("read", True)): + evidence = probe.expected_result(phase, PROBE_ID, dicom) + with self.subTest(phase=phase): + self.assertTrue(probe.probe_result(result(evidence), phase, PROBE_ID, dicom)) + with self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, "count": 0}), phase, PROBE_ID, dicom) + evidence = probe.expected_result("read", PROBE_ID, False) + with self.assertRaises(AssertionError): + probe.probe_result(result(evidence), "read", PROBE_ID, True) + + def test_running_command_is_not_a_pass_and_failure_names_the_stage(self): + running = result({}) + running["properties"]["instanceView"] = {} + self.assertFalse(probe.probe_result(running, "access", PROBE_ID)) + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-role\n") + with self.assertRaisesRegex(AssertionError, "stage=fhir-role"): + probe.probe_result(failed, "write", PROBE_ID) + + def test_payloads_match_enriched_api_schemas(self): + cases = ( + (health.HEALTH_PAYLOAD, "health-services", enrich_workspace_service_template), + (health.VM_PAYLOAD, "guacamole/user_resources/guacamole-azure-windowsvm", enrich_user_resource_template), + ) + for payload, folder, enrich in cases: + with self.subTest(bundle=payload["templateName"]): + schema = json.loads( + (ROOT / "templates/workspace_services" / folder / "template_schema.json").read_text() + ) + validate(payload["properties"], enrich(SimpleNamespace(model_dump=lambda **_: schema))) + self.assertFalse(any(v for k, v in health.VM_PAYLOAD["properties"].items() if k.startswith("install_"))) + + def test_deadline_precedes_container(self): + jobs = yaml.safe_load((ROOT / ".github/workflows/deploy_tre_reusable.yml").read_text())["jobs"] + job = jobs["e2e_tests_bundle"] + reserve = next(s for s in job["steps"] if "HEALTH_VALIDATION_DEADLINE" in s.get("run", "")) + run = next(s for s in job["steps"] if s["name"] == "Run E2E Tests") + self.assertLess(job["steps"].index(reserve), job["steps"].index(run)) + self.assertEqual(reserve["if"], "inputs.e2eBundle == 'tre-workspace-service-health'") + self.assertIn("HEALTH_VALIDATION_DEADLINE=${{ env.HEALTH_VALIDATION_DEADLINE }}", run["with"]["COMMAND"]) + + +class ProbeRecoveryTests(unittest.IsolatedAsyncioTestCase): + async def test_role_is_scoped_to_one_service_and_removed_after_failure(self): + arm = SimpleNamespace(request=AsyncMock(return_value={}), delete=AsyncMock()) + with self.assertRaisesRegex(RuntimeError, "write failed"): + async with probe.role_access(arm, FHIR_ID, probe.FHIR_DATA_CONTRIBUTOR, "principal", "sub"): + raise RuntimeError("write failed") + assignment = arm.request.call_args.args[1] + self.assertTrue(assignment.startswith(FHIR_ID + "/providers/Microsoft.Authorization/roleAssignments/")) + body = arm.request.call_args.kwargs["body"]["properties"] + self.assertEqual(body["principalId"], "principal") + self.assertTrue(body["roleDefinitionId"].endswith("/roleDefinitions/" + probe.FHIR_DATA_CONTRIBUTOR)) + arm.delete.assert_awaited_once_with(assignment, probe.ROLE_API) + + async def test_failed_command_is_removed_and_dicom_is_optional(self): + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-metadata\n") + for dicom_url in ("", "https://hsabcdsvc1234-dicomabcdsvc1234.dicom.azurehealthcareapis.com"): + arm = SimpleNamespace(request=AsyncMock(side_effect=[{}, failed]), delete=AsyncMock()) + with self.subTest(dicom_url=dicom_url), self.assertRaisesRegex(AssertionError, "fhir-metadata"): + await probe.run_probe( + arm, + VM_ID, + "switzerlandnorth", + fhir_url=FHIR_URL, + probe_id=PROBE_ID, + phase="read", + dicom_url=dicom_url, + ) + command = arm.request.call_args_list[0].args[1] + self.assertTrue(command.startswith(VM_ID + "/runCommands/health-probe-")) + arm.delete.assert_awaited_once_with(command, probe.COMPUTE_API) + names = [p["name"] for p in arm.request.call_args_list[0].kwargs["body"]["properties"]["parameters"]] + self.assertEqual(names, ["FhirUrl", "ProbeId", "Phase"] + (["DicomUrl"] if dicom_url else [])) From 38768da46b82819a27152e7517c2104742101135 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 18:43:32 +0100 Subject: [PATCH 03/28] Create an Automatic workspace for the Health check (#5147) The Health bundle needs the workspace owners group. Since #4844, its Terraform refuses an empty workspace_owners_group_id. CI sets TEST_WORKSPACE_APP_ID, so the SQL lifecycle setup created a Manual workspace, which has no Entra ID groups. The first Health run failed on the install with "workspace_owners_group_id must be provided". SQLResources.setup now takes groups=True. Then it creates an Automatic workspace and its own Guacamole service, and it does not use the configured workspace, service or workspace application. The Health case uses this option and stops before the install if the workspace has no owners group. --- docs/tre-developers/end-to-end-tests.md | 2 ++ e2e_tests/resources/sql_lifecycle.py | 21 +++++++++++++-------- e2e_tests/test_health.py | 5 ++++- e2e_tests/unit_tests/test_sql_deadlines.py | 22 ++++++++++++++++++++++ 4 files changed, 41 insertions(+), 9 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 7755ef00b2..c8da2a18a0 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -208,6 +208,8 @@ The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. Select `tre-workspace-service-health` to run `test_health.py::test_private_fhir_access_and_dicom_update`. The case creates its own workspace, Guacamole service, Windows VM and Health service with FHIR R4. +The Health bundle needs the workspace owners group, and only an Automatic workspace gets Entra ID groups. +So the case always creates an Automatic workspace. It does not use `TEST_WORKSPACE_ID`, `TEST_WORKSPACE_SERVICE_ID` or `TEST_WORKSPACE_APP_ID`. From the VM, the case makes sure that the FHIR host resolves to a private address and that the FHIR metadata is available. It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. diff --git a/e2e_tests/resources/sql_lifecycle.py b/e2e_tests/resources/sql_lifecycle.py index 8b19ae9986..0f36a2b91d 100644 --- a/e2e_tests/resources/sql_lifecycle.py +++ b/e2e_tests/resources/sql_lifecycle.py @@ -65,22 +65,27 @@ async def enter(self, name, context): await context.__aenter__() self.own(name, context.__aexit__, None, None, None) - async def setup(self): + async def setup(self, groups=False): # Keep these resources inside the SQL budget rather than session fixtures. + # Only an Automatic workspace gets Entra ID groups, so a case that needs + # them creates its own workspace and does not use the configured ones. + manual = bool(config.TEST_WORKSPACE_APP_ID) and not groups + existing_workspace = "" if groups else config.TEST_WORKSPACE_ID + existing_service = "" if groups else config.TEST_WORKSPACE_SERVICE_ID workspace_path, workspace_id = await create_or_get_test_workspace( - auth_type="Manual" if config.TEST_WORKSPACE_APP_ID else "Automatic", + auth_type="Manual" if manual else "Automatic", verify=self.verify, - pre_created_workspace_id=config.TEST_WORKSPACE_ID, - client_id=config.TEST_WORKSPACE_APP_ID, - client_secret=config.TEST_WORKSPACE_APP_SECRET, + pre_created_workspace_id=existing_workspace, + client_id=config.TEST_WORKSPACE_APP_ID if manual else "", + client_secret=config.TEST_WORKSPACE_APP_SECRET if manual else "", ) - if not config.TEST_WORKSPACE_ID: + if not existing_workspace: self.own(workspace_path, clean_up_test_workspace, "", workspace_path, self.verify) token = await get_workspace_owner_token(workspace_id, self.verify) guacamole_path, guacamole_id = await create_or_get_test_workpace_service( - workspace_path, token, config.TEST_WORKSPACE_SERVICE_ID, self.verify + workspace_path, token, existing_service, self.verify ) - if not config.TEST_WORKSPACE_SERVICE_ID: + if not existing_service: self.own(guacamole_path, clean_up_test_workspace_service, "", guacamole_path, workspace_id, self.verify) return workspace_path, workspace_id, guacamole_path, guacamole_id diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py index 8a72af6da7..2aa5973f1f 100644 --- a/e2e_tests/test_health.py +++ b/e2e_tests/test_health.py @@ -52,8 +52,11 @@ async def test_private_fhir_access_and_dicom_update(verify): if os.environ.get("AZURE_ENVIRONMENT", "AzureCloud") != "AzureCloud": raise ValueError("The health probe currently supports AzureCloud only") async with sql_lifecycle(verify, deadline_variable="HEALTH_VALIDATION_DEADLINE") as resources: - workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup() + # The bundle refuses to install without the workspace owners group. + workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup(groups=True) new_token = partial(get_workspace_owner_token, workspace_id, verify) + workspace = (await get_resource(f"/api{workspace_path}", await new_token(), verify))["workspace"] + assert workspace["properties"].get("workspace_owners_group_id"), "The workspace has no owners group" health_path, health_id = await resources.create( HEALTH_PAYLOAD, f"/api{workspace_path}/workspace-services", await new_token(), cleanup_token=new_token ) diff --git a/e2e_tests/unit_tests/test_sql_deadlines.py b/e2e_tests/unit_tests/test_sql_deadlines.py index 4bcdd4b23e..aa2e1f2635 100644 --- a/e2e_tests/unit_tests/test_sql_deadlines.py +++ b/e2e_tests/unit_tests/test_sql_deadlines.py @@ -121,6 +121,28 @@ async def test_existing_workspace_and_service_are_not_owned(self): await owned.setup() self.assertEqual(owned.finalisers, []) + async def test_group_setup_creates_and_owns_an_automatic_workspace(self): + create_workspace = AsyncMock(return_value=("/workspaces/ws", "ws")) + create_service = AsyncMock(return_value=("/service", "service")) + with ( + self.budgets(), + patch.object(lifecycle.config, "TEST_WORKSPACE_APP_ID", "app"), + patch.object(lifecycle.config, "TEST_WORKSPACE_APP_SECRET", "secret"), + patch.object(lifecycle.config, "TEST_WORKSPACE_ID", "existing"), + patch.object(lifecycle.config, "TEST_WORKSPACE_SERVICE_ID", "existing-service"), + patch.object(lifecycle, "create_or_get_test_workspace", create_workspace), + patch.object(lifecycle, "get_workspace_owner_token", AsyncMock(return_value="token")), + patch.object(lifecycle, "create_or_get_test_workpace_service", create_service), + ): + async with lifecycle.sql_lifecycle(True) as owned: + await owned.setup(groups=True) + self.assertEqual([name for name, _ in owned.finalisers], ["/workspaces/ws", "/service"]) + owned.finalisers.clear() + create_workspace.assert_awaited_once_with( + auth_type="Automatic", verify=True, pre_created_workspace_id="", client_id="", client_secret="" + ) + create_service.assert_awaited_once_with("/workspaces/ws", "token", "", True) + async def test_job_start_deadline_reduces_work_time(self): with patch.dict(os.environ, SQL_VALIDATION_DEADLINE=str(time.time() + 125 * 60)): work, finish = lifecycle.lifecycle_deadlines() From f920cfdec4a3304608a3f86f4defe0a4e02ea55d Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 19:02:33 +0100 Subject: [PATCH 04/28] Remove Health from the generic workspace service install case (#5147) test_install_workspace_service uses the shared workspace fixture. CI always sets TEST_WORKSPACE_APP_ID, so that workspace is Manual and has no Entra ID groups. Since #4844, the Health bundle refuses an empty workspace_owners_group_id, so the Health parameter of this case cannot pass in CI. The first Health run on another fork showed the same Terraform error with a Manual workspace. The private Health case creates an Automatic workspace and covers the install, an update and the uninstall, so the generic case drops Health. The coverage map no longer lists the generic case as a related test. --- CHANGELOG.md | 1 + e2e_tests/bundle_coverage.json | 4 +--- e2e_tests/test_workspace_services.py | 1 - 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aff8d22c80..c49e6e409c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Setting to `false` will delete existing airlock storage accounts and must only b ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) * Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. The generic workspace service install case no longer installs Health, because its CI workspace has no Entra ID groups. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) * Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index e395cef64c..237d8b1157 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -379,9 +379,7 @@ "tests": [ "test_health.py::test_private_fhir_access_and_dicom_update" ], - "related_tests": [ - "test_workspace_services.py::test_install_workspace_service[tre-workspace-service-health]" - ], + "related_tests": [], "coverage": [ "install with FHIR R4", "private FHIR metadata", diff --git a/e2e_tests/test_workspace_services.py b/e2e_tests/test_workspace_services.py index 3ce22c5614..bcb7efcd07 100644 --- a/e2e_tests/test_workspace_services.py +++ b/e2e_tests/test_workspace_services.py @@ -11,7 +11,6 @@ strings.AZUREML_SERVICE, strings.GITEA_SERVICE, strings.MYSQL_SERVICE, - strings.HEALTH_SERVICE, strings.AZURESQL_SERVICE, ] From 61d93978fd76a52a2c34cda3fb1d085b1fd03fd9 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 20:51:29 +0100 Subject: [PATCH 05/28] Replace the Health private endpoint when its services change (#5147) The Health service creates one private endpoint for the healthcare workspace. Azure writes the private DNS records when it creates the endpoint. The second Health run on another fork installed FHIR, then added DICOM with an update. The Activity Log showed A records for the workspace and FHIR only, and no record in the DICOM zone in the 16 minutes before the delete. Thus the DICOM host did not resolve to a private address from the workspace VM. Microsoft documents that missing private link DNS entries can occur, and that a new private endpoint fixes them: https://learn.microsoft.com/en-us/azure/healthcare-apis/configure-private-link A terraform_data resource now holds the deploy_fhir and deploy_dicom values, and the endpoint is replaced when they change. A new terraform_data resource does not replace the endpoint of an existing deployment, so the docs give the repair for a service that got FHIR or DICOM before this version. --- CHANGELOG.md | 1 + .../workspace-services/health_services.md | 11 ++++++++++ .../unit_tests/test_health_bundle_contract.py | 22 +++++++++++++++++++ .../health-services/terraform/main.tf | 15 ++++++++++++- 4 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 e2e_tests/unit_tests/test_health_bundle_contract.py diff --git a/CHANGELOG.md b/CHANGELOG.md index c49e6e409c..fb1a418ca3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,6 +49,7 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) +* Replace the Health private endpoint when an update adds or removes FHIR or DICOM. Azure wrote private DNS records only for the services that existed when it created the endpoint, so a DICOM service that an update added resolved to a public address. An existing service that got FHIR or DICOM after its install keeps the missing record until the endpoint is created again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 3bc98092b0..f9e86ddde2 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -20,6 +20,17 @@ If Azure reports the FHIR service as `Accepted` after the create, the azurerm pr After this failure, each update and each delete of the Health service fails. To repair the service, delete its FHIR service in the Azure portal, and then delete the Health service again. +## Private network access + +The service connects the healthcare workspace to the workspace network through one private endpoint. +Azure writes the private DNS records for FHIR and DICOM when it creates this endpoint. +If an update adds or removes FHIR or DICOM, Terraform replaces the endpoint, so that each service gets a private DNS record. +During this replacement, private access to the services stops for some minutes. + +Before version 0.3.9, an update did not replace the endpoint. +If you added FHIR or DICOM to an existing service before this version, the new host can resolve to a public address. +To repair the service, delete its private endpoint `pe-hs` in Azure, then upgrade the service. Terraform then creates the endpoint again. + ## Authentication Learn more about authentication and application roles in [this doc](https://learn.microsoft.com/en-us/azure/healthcare-apis/authentication-authorization). Make sure to assign your users/apps with the required role and follow the guidelines to retrieve a token. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py new file mode 100644 index 0000000000..a1ac504dee --- /dev/null +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -0,0 +1,22 @@ +"""Check that the Health private endpoint has a DNS record for each service.""" + +from pathlib import Path +import unittest + +ROOT = Path(__file__).resolve().parents[2] +BUNDLE = ROOT / "templates/workspace_services/health-services/terraform" + + +def block(source, header): + return source.split(header + " {", 1)[1].split("\n}", 1)[0] + + +class HealthBundleContractTests(unittest.TestCase): + def test_endpoint_is_replaced_when_the_services_change(self): + # Azure does not always add a DNS record for a service that joins after the endpoint. + source = (BUNDLE / "main.tf").read_text() + services = block(source, 'resource "terraform_data" "private_endpoint_services"') + self.assertRegex(services, r"\n fhir\s*=\s*var\.deploy_fhir\n") + self.assertRegex(services, r"\n dicom\s*=\s*var\.deploy_dicom\n") + endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') + self.assertRegex(endpoint, r"replace_triggered_by\s*=\s*\[terraform_data\.private_endpoint_services\]") diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 0189b5e456..ddd23ff038 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -43,6 +43,16 @@ resource "azurerm_healthcare_dicom_service" "dicom" { lifecycle { ignore_changes = [tags] } } +# Azure writes the private DNS records when it creates the endpoint. It does not +# always add a record for a service that joins the workspace later, so the +# endpoint is replaced when the set of services changes. +resource "terraform_data" "private_endpoint_services" { + input = { + fhir = var.deploy_fhir + dicom = var.deploy_dicom + } +} + resource "azurerm_private_endpoint" "health_services_private_endpoint" { name = "pe-${azurerm_healthcare_workspace.healthcare_workspace.name}" location = data.azurerm_resource_group.ws.location @@ -67,5 +77,8 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { azurerm_healthcare_dicom_service.dicom ] - lifecycle { ignore_changes = [tags] } + lifecycle { + ignore_changes = [tags] + replace_triggered_by = [terraform_data.private_endpoint_services] + } } From 566c305aa4a520f17e180f0302116c78749874e2 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 22:28:53 +0100 Subject: [PATCH 06/28] Turn off public network access on the Health DICOM service (#5147) The azurerm provider creates a DICOM service with public network access on unless the configuration sets it. In a live run, a DICOM service that an update added had public network access on for 22 minutes, until Azure turned it off when the private endpoint changed. The provider then sees a difference, so the next upgrade turns it on again. Set public_network_access_enabled = false. An upgrade changes the setting in place. Release this as tre-workspace-service-health 0.3.9. --- CHANGELOG.md | 1 + docs/tre-templates/workspace-services/health_services.md | 4 ++++ e2e_tests/unit_tests/test_health_bundle_contract.py | 8 +++++++- .../workspace_services/health-services/terraform/main.tf | 3 +++ 4 files changed, 15 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fb1a418ca3..f22365925b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,7 @@ BUG FIXES: * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Replace the Health private endpoint when an update adds or removes FHIR or DICOM. Azure wrote private DNS records only for the services that existed when it created the endpoint, so a DICOM service that an update added resolved to a public address. An existing service that got FHIR or DICOM after its install keeps the missing record until the endpoint is created again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index f9e86ddde2..f2ae399026 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -31,6 +31,10 @@ Before version 0.3.9, an update did not replace the endpoint. If you added FHIR or DICOM to an existing service before this version, the new host can resolve to a public address. To repair the service, delete its private endpoint `pe-hs` in Azure, then upgrade the service. Terraform then creates the endpoint again. +The DICOM service has public network access turned off. +Before version 0.3.9, Terraform created the DICOM service with public network access on. +An upgrade to version 0.3.9 turns it off. + ## Authentication Learn more about authentication and application roles in [this doc](https://learn.microsoft.com/en-us/azure/healthcare-apis/authentication-authorization). Make sure to assign your users/apps with the required role and follow the guidelines to retrieve a token. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index a1ac504dee..e6cc49c276 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -1,4 +1,4 @@ -"""Check that the Health private endpoint has a DNS record for each service.""" +"""Check the private network settings of the Health bundle.""" from pathlib import Path import unittest @@ -20,3 +20,9 @@ def test_endpoint_is_replaced_when_the_services_change(self): self.assertRegex(services, r"\n dicom\s*=\s*var\.deploy_dicom\n") endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') self.assertRegex(endpoint, r"replace_triggered_by\s*=\s*\[terraform_data\.private_endpoint_services\]") + + def test_dicom_public_network_access_is_off(self): + # The provider turns it on unless the bundle sets it. + source = (BUNDLE / "main.tf").read_text() + dicom = block(source, 'resource "azurerm_healthcare_dicom_service" "dicom"') + self.assertRegex(dicom, r"\n public_network_access_enabled\s*=\s*false\n") diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index ddd23ff038..eaa6acd15e 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -36,6 +36,9 @@ resource "azurerm_healthcare_dicom_service" "dicom" { location = data.azurerm_resource_group.ws.location tags = local.workspace_service_tags + # The provider default turns on public network access. + public_network_access_enabled = false + identity { type = "SystemAssigned" } From 52e72b261bcc66e684d01064b67322724cd4057b Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 23 Oct 2026 23:10:10 +0100 Subject: [PATCH 07/28] Try the Health probe again after a failed FHIR name lookup (#5147) In a live run, the access probe resolved the FHIR host to its private address. The write probe on the same VM, 60 seconds later, failed at the private-dns stage. No Azure resource changed between the two probes, and the workspace uses Azure DNS directly. The probe changes no data before this stage. So run_probe runs a probe again after 60 seconds if only the private-dns stage failed, 3 attempts in total. A failure at any other stage, which includes a DICOM host that is not private, still fails at once. --- docs/tre-developers/end-to-end-tests.md | 1 + e2e_tests/resources/health_probe.py | 29 +++++++++++++++++++++-- e2e_tests/unit_tests/test_health_probe.py | 27 ++++++++++++++++++++- 3 files changed, 54 insertions(+), 3 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index c8da2a18a0..d556af650d 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -211,6 +211,7 @@ The case creates its own workspace, Guacamole service, Windows VM and Health ser The Health bundle needs the workspace owners group, and only an Automatic workspace gets Entra ID groups. So the case always creates an Automatic workspace. It does not use `TEST_WORKSPACE_ID`, `TEST_WORKSPACE_SERVICE_ID` or `TEST_WORKSPACE_APP_ID`. From the VM, the case makes sure that the FHIR host resolves to a private address and that the FHIR metadata is available. +If the FHIR name lookup fails, the case runs that probe again after 60 seconds, 3 attempts in total. Before this lookup, the probe changes no data. It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. An update adds DICOM. The case reads the Patient again and makes sure that the DICOM host is private and needs a token. diff --git a/e2e_tests/resources/health_probe.py b/e2e_tests/resources/health_probe.py index 7708b89889..a764ab9b53 100644 --- a/e2e_tests/resources/health_probe.py +++ b/e2e_tests/resources/health_probe.py @@ -3,6 +3,7 @@ import asyncio from contextlib import asynccontextmanager import json +import logging from pathlib import Path from uuid import uuid4 @@ -12,6 +13,15 @@ RESULT = "HEALTH_PROBE_RESULT=" FAILED = "HEALTH_PROBE_FAILED=" FHIR_DATA_CONTRIBUTOR = "5a1fc7df-4bf1-4951-a576-89034ee01acd" +DNS_ATTEMPTS = 3 +DNS_RETRY_SECONDS = 60 +LOGGER = logging.getLogger(__name__) + + +class ProbeFailure(AssertionError): + def __init__(self, message, stages): + super().__init__(message) + self.stages = stages def expected_result(phase, probe_id, dicom): @@ -31,8 +41,9 @@ def probe_result(resource, phase, probe_id, dicom=False): lines = view.get("output", "").splitlines() if properties.get("provisioningState") in ("Failed", "Canceled") or state in ("Failed", "Canceled", "TimedOut"): stages = [line[len(FAILED) :] for line in lines if line.startswith(FAILED)] - raise AssertionError( - f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, stage={','.join(stages) or 'unknown'}" + raise ProbeFailure( + f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, stage={','.join(stages) or 'unknown'}", + stages, ) if state != "Succeeded": return False @@ -65,6 +76,20 @@ async def role_access(arm, scope, role, principal_id, subscription_id): async def run_probe(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): + # A lookup of the FHIR name failed once on a new VM. The probe changes no data before this stage. + for attempt in range(1, DNS_ATTEMPTS + 1): + try: + return await run_probe_once( + arm, vm_id, location, fhir_url=fhir_url, probe_id=probe_id, phase=phase, dicom_url=dicom_url + ) + except ProbeFailure as error: + if error.stages != ["private-dns"] or attempt == DNS_ATTEMPTS: + raise + LOGGER.warning("Health probe %s failed at private-dns, attempt %d. Trying again.", phase, attempt) + await asyncio.sleep(DNS_RETRY_SECONDS) + + +async def run_probe_once(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): parameters = {"FhirUrl": fhir_url, "ProbeId": probe_id, "Phase": phase} if dicom_url: parameters["DicomUrl"] = dicom_url diff --git a/e2e_tests/unit_tests/test_health_probe.py b/e2e_tests/unit_tests/test_health_probe.py index acd1cbd038..ae265e68f6 100644 --- a/e2e_tests/unit_tests/test_health_probe.py +++ b/e2e_tests/unit_tests/test_health_probe.py @@ -4,7 +4,7 @@ from pathlib import Path from types import SimpleNamespace import unittest -from unittest.mock import AsyncMock +from unittest.mock import AsyncMock, patch from jsonschema import validate import yaml @@ -119,3 +119,28 @@ async def test_failed_command_is_removed_and_dicom_is_optional(self): arm.delete.assert_awaited_once_with(command, probe.COMPUTE_API) names = [p["name"] for p in arm.request.call_args_list[0].kwargs["body"]["properties"]["parameters"]] self.assertEqual(names, ["FhirUrl", "ProbeId", "Phase"] + (["DicomUrl"] if dicom_url else [])) + + async def test_only_a_dns_failure_is_tried_again(self): + def failed(stage): + return result({}, state="Failed", code=1, output=f"HEALTH_PROBE_FAILED={stage}\n") + + passed = result(probe.expected_result("access", PROBE_ID, False)) + cases = ( + ([failed("private-dns"), passed], None), + ([failed("private-dns")] * probe.DNS_ATTEMPTS, "stage=private-dns"), + ([failed("fhir-metadata")], "stage=fhir-metadata"), + ) + for results, error in cases: + # Each attempt creates a command, reads its result and then removes the command. + responses = [response for r in results for response in ({}, r)] + arm = SimpleNamespace(request=AsyncMock(side_effect=responses), delete=AsyncMock()) + with self.subTest(error=error), patch.object(probe, "DNS_RETRY_SECONDS", 0): + run = probe.run_probe( + arm, VM_ID, "switzerlandnorth", fhir_url=FHIR_URL, probe_id=PROBE_ID, phase="access" + ) + if error: + with self.assertRaisesRegex(AssertionError, error): + await run + else: + await run + self.assertEqual(arm.delete.await_count, len(results)) From 8de1c596ff70dc420e72315ec8de45438aad9381 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 01:57:01 +0100 Subject: [PATCH 08/28] Check that the Health services reject public requests (#5147) The E2E case read publicNetworkAccess on FHIR before the DICOM update. It now sends requests without a token from the test runner to FHIR /metadata and DICOM /health/check. Azure must reject each one with HTTP 403, as the Azure private link guide describes. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. It then reads the workspace and DICOM again. The case does not read publicNetworkAccess on FHIR. In a live run, a request set this value to Disabled twice, and Azure accepted both requests. FHIR still reported Enabled, and public requests to FHIR still got HTTP 403. The private endpoint on the workspace blocks public traffic to all services in the workspace. --- docs/tre-developers/end-to-end-tests.md | 5 ++- .../workspace-services/health_services.md | 5 +++ e2e_tests/resources/health_probe.py | 14 ++++++++ e2e_tests/test_health.py | 32 +++++++++++++------ e2e_tests/unit_tests/test_health_probe.py | 16 ++++++++++ 5 files changed, 62 insertions(+), 10 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index d556af650d..1ede34ec82 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -215,7 +215,10 @@ If the FHIR name lookup fails, the case runs that probe again after 60 seconds, It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. An update adds DICOM. The case reads the Patient again and makes sure that the DICOM host is private and needs a token. -At the end, the case makes sure that the Health workspace, FHIR and DICOM have public network access disabled. +At the end, the case sends FHIR and DICOM requests without a token from the test runner, outside the workspace network. +Azure must reject each request with HTTP 403. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. +The case then reads the Health workspace and DICOM again, and makes sure that each has public network access disabled. +The case does not read this value on FHIR. Azure keeps `publicNetworkAccess` `Enabled` on FHIR, also after a request sets `Disabled`. The Health job uses the SQL time limits with its own `HEALTH_VALIDATION_DEADLINE`. ### Run the branch workflow before merge diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index f2ae399026..de33cb12e9 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -35,6 +35,11 @@ The DICOM service has public network access turned off. Before version 0.3.9, Terraform created the DICOM service with public network access on. An upgrade to version 0.3.9 turns it off. +The private endpoint on the healthcare workspace blocks public traffic to FHIR and DICOM. +A request from outside the workspace network gets HTTP 403. +The FHIR service reports `publicNetworkAccess` `Enabled`, but Azure does not use this value when the workspace has an approved private endpoint. +Azure also keeps this value when a request sets it to `Disabled`. + ## Authentication Learn more about authentication and application roles in [this doc](https://learn.microsoft.com/en-us/azure/healthcare-apis/authentication-authorization). Make sure to assign your users/apps with the required role and follow the guidelines to retrieve a token. diff --git a/e2e_tests/resources/health_probe.py b/e2e_tests/resources/health_probe.py index a764ab9b53..9ac3810107 100644 --- a/e2e_tests/resources/health_probe.py +++ b/e2e_tests/resources/health_probe.py @@ -15,6 +15,10 @@ FHIR_DATA_CONTRIBUTOR = "5a1fc7df-4bf1-4951-a576-89034ee01acd" DNS_ATTEMPTS = 3 DNS_RETRY_SECONDS = 60 +PUBLIC_BLOCKED = 403 +# Azure can take up to 5 minutes to block public traffic after a change. +PUBLIC_WAIT_SECONDS = 5 * 60 +PUBLIC_RETRY_SECONDS = 30 LOGGER = logging.getLogger(__name__) @@ -75,6 +79,16 @@ async def role_access(arm, scope, role, principal_id, subscription_id): yield +async def public_statuses(client, urls): + """Send requests from outside the workspace network until Azure blocks each one, or the wait ends.""" + deadline = asyncio.get_running_loop().time() + PUBLIC_WAIT_SECONDS + while True: + statuses = {name: (await client.get(url)).status_code for name, url in urls.items()} + if set(statuses.values()) == {PUBLIC_BLOCKED} or asyncio.get_running_loop().time() >= deadline: + return statuses + await asyncio.sleep(PUBLIC_RETRY_SECONDS) + + async def run_probe(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): # A lookup of the FHIR name failed once on a new VM. The probe changes no data before this stage. for attempt in range(1, DNS_ATTEMPTS + 1): diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py index 2aa5973f1f..f7242c938e 100644 --- a/e2e_tests/test_health.py +++ b/e2e_tests/test_health.py @@ -6,12 +6,19 @@ import os from uuid import uuid4 +from httpx import AsyncClient import pytest from e2e_tests import config from e2e_tests.conftest import get_workspace_owner_token from e2e_tests.resources import strings -from e2e_tests.resources.health_probe import FHIR_DATA_CONTRIBUTOR, role_access, run_probe +from e2e_tests.resources.health_probe import ( + FHIR_DATA_CONTRIBUTOR, + PUBLIC_BLOCKED, + public_statuses, + role_access, + run_probe, +) from e2e_tests.resources.resource import get_resource, post_resource from e2e_tests.resources.sql_lifecycle import sql_lifecycle from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, validate_vm @@ -99,16 +106,23 @@ async def test_private_fhir_access_and_dicom_update(verify): ) health = (await get_resource(f"/api{health_path}", await new_token(), verify))["workspaceService"] assert health["properties"]["dicom_url"] == dicom_url - dicom = await arm.request("GET", f"{workspace_arm_id}/dicomservices/dicom{suffix}", HEALTH_API) + dicom_id = f"{workspace_arm_id}/dicomservices/dicom{suffix}" + dicom = await arm.request("GET", dicom_id, HEALTH_API) assert dicom["properties"]["provisioningState"] == "Succeeded" await probe(phase="read", dicom_url=dicom_url) LOGGER.info("FHIR data survived the update that added DICOM") # Check the network settings last, so that a failure here keeps the functional evidence above. - workspace_record = await arm.request("GET", workspace_arm_id, HEALTH_API) - network = { - "workspace": workspace_record["properties"].get("publicNetworkAccess"), - "fhir": fhir["properties"].get("publicNetworkAccess"), - "dicom": dicom["properties"].get("publicNetworkAccess"), - } - LOGGER.info("Health public network access: %s", network) + async with AsyncClient(timeout=30) as client: + public = await public_statuses( + client, {"fhir": f"{fhir_url}/metadata", "dicom": f"{dicom_url}/health/check"} + ) + # Read the resources again, because the update replaced the private endpoint. + # FHIR is not in this list: Azure keeps publicNetworkAccess Enabled on FHIR, also after a request + # sets Disabled. The private endpoint on the workspace blocks public traffic to FHIR (checked above). + network = {} + for name, resource_id in (("workspace", workspace_arm_id), ("dicom", dicom_id)): + record = await arm.request("GET", resource_id, HEALTH_API) + network[name] = record["properties"].get("publicNetworkAccess") + LOGGER.info("Health public responses: %s, public network access: %s", public, network) + assert public == dict.fromkeys(public, PUBLIC_BLOCKED), f"Health services answer public requests: {public}" assert network == dict.fromkeys(network, "Disabled"), f"Health services allow public access: {network}" diff --git a/e2e_tests/unit_tests/test_health_probe.py b/e2e_tests/unit_tests/test_health_probe.py index ae265e68f6..d78434290d 100644 --- a/e2e_tests/unit_tests/test_health_probe.py +++ b/e2e_tests/unit_tests/test_health_probe.py @@ -6,6 +6,7 @@ import unittest from unittest.mock import AsyncMock, patch +import httpx from jsonschema import validate import yaml @@ -144,3 +145,18 @@ def failed(stage): else: await run self.assertEqual(arm.delete.await_count, len(results)) + + async def test_public_requests_are_sent_again_until_azure_blocks_them(self): + urls = {"fhir": FHIR_URL + "/metadata", "dicom": "https://dicom.example/health/check"} + cases = ( + ([200, 403, 403, 403], {}, {"fhir": 403, "dicom": 403}), + # If the wait ends first, the caller gets the last statuses and reports them. + ([200, 401], {"PUBLIC_WAIT_SECONDS": 0}, {"fhir": 200, "dicom": 401}), + ) + for codes, settings, expected in cases: + responses = iter(codes) + transport = httpx.MockTransport(lambda request: httpx.Response(next(responses))) + with self.subTest(codes=codes), patch.multiple(probe, PUBLIC_RETRY_SECONDS=0, **settings): + async with httpx.AsyncClient(transport=transport) as client: + statuses = await probe.public_statuses(client, urls) + self.assertEqual(statuses, expected) From 947c0517038377fb11f27d237ce6033be2223bbf Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 07:15:55 +0100 Subject: [PATCH 09/28] Send FHIR data requests again while the probe role applies (#5147) In a live run, the role wait got HTTP 200 and the create passed, but the next search failed. The probe gave only the stage, so the cause is unknown. Until the 10-minute role deadline, the probe now sends the create again after HTTP 403 or 429, and the search again after HTTP 403, 429, 500, 502, 503 or 504. FHIR writes nothing for a refused or throttled create, so a repeated create cannot add a second Patient. A failed probe now also prints the HTTP status of the failed request, and the test error shows it. --- docs/tre-developers/end-to-end-tests.md | 4 +++ e2e_tests/resources/health_probe.ps1 | 30 +++++++++++++++++------ e2e_tests/resources/health_probe.py | 5 +++- e2e_tests/unit_tests/test_health_probe.py | 5 +++- 4 files changed, 35 insertions(+), 9 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 1ede34ec82..de8b22ea02 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -214,6 +214,10 @@ From the VM, the case makes sure that the FHIR host resolves to a private addres If the FHIR name lookup fails, the case runs that probe again after 60 seconds, 3 attempts in total. Before this lookup, the probe changes no data. It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. +The role can take some minutes to apply, so the probe waits up to 10 minutes for a first HTTP 200. +Until the end of this wait, the probe sends the create again after HTTP 403 or 429. +It sends the search again after HTTP 403, 429, 500, 502, 503 or 504. +If a probe fails, the test error gives the stage and the HTTP status of the failed request. An update adds DICOM. The case reads the Patient again and makes sure that the DICOM host is private and needs a token. At the end, the case sends FHIR and DICOM requests without a token from the test runner, outside the workspace network. Azure must reject each request with HTTP 403. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. diff --git a/e2e_tests/resources/health_probe.ps1 b/e2e_tests/resources/health_probe.ps1 index 2b5f35444b..7c049e76de 100644 --- a/e2e_tests/resources/health_probe.ps1 +++ b/e2e_tests/resources/health_probe.ps1 @@ -40,8 +40,22 @@ function Get-FhirHeaders { return @{Authorization = "Bearer $($identity.access_token)"} } -function Get-ProbeCount([hashtable]$Headers) { - $search = Invoke-RestMethod -Headers $Headers -Uri "$FhirUrl/Patient?identifier=urn:tre:e2e|$ProbeId" -TimeoutSec 30 +function Invoke-Fhir([hashtable]$Request, [int[]]$RetryStatus) { + # A 200 in the role wait does not prove that each later request gets the new role. + while ($true) { + try { + return Invoke-RestMethod @Request -Headers (Get-FhirHeaders) -TimeoutSec 30 + } catch { + $response = $_.Exception.Response + if ($null -eq $response -or [int]$response.StatusCode -notin $RetryStatus -or [DateTime]::UtcNow -ge $deadline) { throw } + Start-Sleep -Seconds 20 + } + } +} + +function Get-ProbeCount { + # A search changes no data, so throttling and server errors are also tried again. + $search = Invoke-Fhir -Request @{Uri = "$FhirUrl/Patient?identifier=urn:tre:e2e|$ProbeId"} -RetryStatus 403, 429, 500, 502, 503, 504 if ($search.resourceType -ne 'Bundle') { throw 'Unexpected search result' } return @($search.entry).Where({ $null -ne $_ }).Count } @@ -66,15 +80,16 @@ try { if ([DateTime]::UtcNow -ge $deadline) { throw 'Role did not apply' } Start-Sleep -Seconds 20 } - $headers = Get-FhirHeaders if ($Phase -eq 'write') { $stage = 'fhir-write' $patient = @{resourceType = 'Patient'; identifier = @(@{system = 'urn:tre:e2e'; value = $ProbeId})} | ConvertTo-Json -Depth 4 - $created = Invoke-RestMethod -Method Post -Headers $headers -ContentType 'application/fhir+json' -Uri "$FhirUrl/Patient" -Body $patient -TimeoutSec 30 + # FHIR writes nothing for a refused (403) or throttled (429) create, so only these are tried again. + $create = @{Method = 'Post'; ContentType = 'application/fhir+json'; Uri = "$FhirUrl/Patient"; Body = $patient} + $created = Invoke-Fhir -Request $create -RetryStatus 403, 429 if ($created.resourceType -ne 'Patient') { throw 'Unexpected create result' } } $stage = 'fhir-read' - $result.count = Get-ProbeCount $headers + $result.count = Get-ProbeCount if ($DicomUrl) { $stage = 'dicom-access' Assert-PrivateDns $DicomUrl @@ -83,10 +98,11 @@ try { } Write-Output ('HEALTH_PROBE_RESULT=' + ($result | ConvertTo-Json -Compress)) } catch { - # Do not emit exception details or tokens. + # Do not emit exception details or tokens. The HTTP status of a failed request is safe to emit. + $response = $_.Exception.Response Write-Output "HEALTH_PROBE_FAILED=$stage" + if ($null -ne $response) { Write-Output "HEALTH_PROBE_STATUS=$([int]$response.StatusCode)" } exit 1 } finally { $identity = $null - $headers = $null } diff --git a/e2e_tests/resources/health_probe.py b/e2e_tests/resources/health_probe.py index 9ac3810107..3a23b8e01c 100644 --- a/e2e_tests/resources/health_probe.py +++ b/e2e_tests/resources/health_probe.py @@ -12,6 +12,7 @@ SCRIPT = Path(__file__).with_name("health_probe.ps1") RESULT = "HEALTH_PROBE_RESULT=" FAILED = "HEALTH_PROBE_FAILED=" +STATUS = "HEALTH_PROBE_STATUS=" FHIR_DATA_CONTRIBUTOR = "5a1fc7df-4bf1-4951-a576-89034ee01acd" DNS_ATTEMPTS = 3 DNS_RETRY_SECONDS = 60 @@ -45,8 +46,10 @@ def probe_result(resource, phase, probe_id, dicom=False): lines = view.get("output", "").splitlines() if properties.get("provisioningState") in ("Failed", "Canceled") or state in ("Failed", "Canceled", "TimedOut"): stages = [line[len(FAILED) :] for line in lines if line.startswith(FAILED)] + statuses = [line[len(STATUS) :] for line in lines if line.startswith(STATUS)] raise ProbeFailure( - f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, stage={','.join(stages) or 'unknown'}", + f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, " + f"stage={','.join(stages) or 'unknown'}, http_status={','.join(statuses) or 'none'}", stages, ) if state != "Succeeded": diff --git a/e2e_tests/unit_tests/test_health_probe.py b/e2e_tests/unit_tests/test_health_probe.py index d78434290d..7c97e1867e 100644 --- a/e2e_tests/unit_tests/test_health_probe.py +++ b/e2e_tests/unit_tests/test_health_probe.py @@ -62,7 +62,10 @@ def test_running_command_is_not_a_pass_and_failure_names_the_stage(self): running["properties"]["instanceView"] = {} self.assertFalse(probe.probe_result(running, "access", PROBE_ID)) failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-role\n") - with self.assertRaisesRegex(AssertionError, "stage=fhir-role"): + with self.assertRaisesRegex(AssertionError, "stage=fhir-role, http_status=none"): + probe.probe_result(failed, "write", PROBE_ID) + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-read\nHEALTH_PROBE_STATUS=403\n") + with self.assertRaisesRegex(AssertionError, "stage=fhir-read, http_status=403"): probe.probe_result(failed, "write", PROBE_ID) def test_payloads_match_enriched_api_schemas(self): From c3f98b3ad78df33235da5e385ea2c85fc37f4728 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 01:31:22 +0000 Subject: [PATCH 10/28] Keep the Health private endpoint when its services change (#5147) A live run added DICOM to a Health service with an update. Terraform deleted the private endpoint early in the update and created it again after DICOM, so the workspace had no endpoint for about 13 minutes. During this time, private access to FHIR stopped. A zone group change in the azurerm endpoint resource is not a fix. The provider then sends the whole endpoint again, and Azure refused this with HTTP 409 AnotherOperationInProgress while it linked the new DICOM service to the endpoint. The DNS zone group is now a separate azapi resource. It always has the zone for the workspace and FHIR, and it gets the DICOM zone when the service has DICOM. Azure writes the records of a zone when the group gets that zone. An update that adds DICOM changes the group in place, the endpoint stays, and the provider tries again on HTTP 409. Before this version, the endpoint resource created the group. An action deletes that group once, so that the azapi resource can create a group with the same name. An upgrade then writes the records again, also a record that an earlier update did not get. So the docs no longer tell the reader to delete the endpoint. The bundle uses Terraform 1.9.8. Before version 2.9.0, the azapi provider fails each update with this Terraform version, so the bundle uses azapi 2.9.0. A unit test makes sure of this. The Health case now makes sure that the update kept the endpoint. --- CHANGELOG.md | 2 + docs/tre-developers/end-to-end-tests.md | 3 +- .../workspace-services/health_services.md | 22 ++++--- e2e_tests/test_health.py | 10 +++- .../unit_tests/test_health_bundle_contract.py | 42 +++++++++++-- .../health-services/porter.yaml | 2 +- .../terraform/.terraform.lock.hcl | 23 +++++++ .../health-services/terraform/main.tf | 60 +++++++++++++------ .../health-services/terraform/providers.tf | 7 +++ 9 files changed, 135 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f22365925b..1808da05bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,8 @@ BUG FIXES: * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Replace the Health private endpoint when an update adds or removes FHIR or DICOM. Azure wrote private DNS records only for the services that existed when it created the endpoint, so a DICOM service that an update added resolved to a public address. An existing service that got FHIR or DICOM after its install keeps the missing record until the endpoint is created again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index de8b22ea02..8e7a34d1e1 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -218,7 +218,8 @@ The role can take some minutes to apply, so the probe waits up to 10 minutes for Until the end of this wait, the probe sends the create again after HTTP 403 or 429. It sends the search again after HTTP 403, 429, 500, 502, 503 or 504. If a probe fails, the test error gives the stage and the HTTP status of the failed request. -An update adds DICOM. The case reads the Patient again and makes sure that the DICOM host is private and needs a token. +An update adds DICOM. The case makes sure that the update kept the private endpoint. +It then reads the Patient again and makes sure that the DICOM host is private and needs a token. At the end, the case sends FHIR and DICOM requests without a token from the test runner, outside the workspace network. Azure must reject each request with HTTP 403. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. The case then reads the Health workspace and DICOM again, and makes sure that each has public network access disabled. diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index de33cb12e9..bad6f6e0ae 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -23,17 +23,21 @@ To repair the service, delete its FHIR service in the Azure portal, and then del ## Private network access The service connects the healthcare workspace to the workspace network through one private endpoint. -Azure writes the private DNS records for FHIR and DICOM when it creates this endpoint. -If an update adds or removes FHIR or DICOM, Terraform replaces the endpoint, so that each service gets a private DNS record. -During this replacement, private access to the services stops for some minutes. - -Before version 0.3.9, an update did not replace the endpoint. -If you added FHIR or DICOM to an existing service before this version, the new host can resolve to a public address. -To repair the service, delete its private endpoint `pe-hs` in Azure, then upgrade the service. Terraform then creates the endpoint again. +Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. +The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. +If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. +Azure links a new service to the endpoint some minutes after Terraform creates the service. +During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. + +Before version 0.3.10, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. +If you added DICOM to an existing service before this version, the DICOM host can resolve to a public address. +An upgrade to version 0.3.10 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. +Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. +Do not delete the private endpoint to repair the service. The DICOM service has public network access turned off. -Before version 0.3.9, Terraform created the DICOM service with public network access on. -An upgrade to version 0.3.9 turns it off. +Before version 0.3.10, Terraform created the DICOM service with public network access on. +An upgrade to version 0.3.10 turns it off. The private endpoint on the healthcare workspace blocks public traffic to FHIR and DICOM. A request from outside the workspace network gets HTTP 403. diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py index f7242c938e..8b41f619de 100644 --- a/e2e_tests/test_health.py +++ b/e2e_tests/test_health.py @@ -25,6 +25,7 @@ LOGGER = logging.getLogger(__name__) HEALTH_API = "2024-03-31" +NETWORK_API = "2024-05-01" HEALTH_PAYLOAD = { "templateName": strings.HEALTH_SERVICE, "properties": { @@ -87,6 +88,8 @@ async def test_private_fhir_access_and_dicom_update(verify): fhir_id = f"{workspace_arm_id}/fhirservices/fhir{suffix}" fhir = await arm.request("GET", fhir_id, HEALTH_API) assert fhir["properties"]["provisioningState"] == "Succeeded" + endpoint_id = f"{group}/providers/Microsoft.Network/privateEndpoints/pe-hs{suffix}" + endpoint = await arm.request("GET", endpoint_id, NETWORK_API) probe_id = uuid4().hex probe = partial(run_probe, arm, azure_vm_id, vm["location"], fhir_url=fhir_url, probe_id=probe_id) await probe(phase="access") @@ -109,6 +112,11 @@ async def test_private_fhir_access_and_dicom_update(verify): dicom_id = f"{workspace_arm_id}/dicomservices/dicom{suffix}" dicom = await arm.request("GET", dicom_id, HEALTH_API) assert dicom["properties"]["provisioningState"] == "Succeeded" + # A new endpoint stops private access to FHIR for some minutes. + updated_endpoint = await arm.request("GET", endpoint_id, NETWORK_API) + assert updated_endpoint["properties"]["resourceGuid"] == endpoint["properties"]["resourceGuid"], ( + "The update replaced the Health private endpoint" + ) await probe(phase="read", dicom_url=dicom_url) LOGGER.info("FHIR data survived the update that added DICOM") # Check the network settings last, so that a failure here keeps the functional evidence above. @@ -116,7 +124,7 @@ async def test_private_fhir_access_and_dicom_update(verify): public = await public_statuses( client, {"fhir": f"{fhir_url}/metadata", "dicom": f"{dicom_url}/health/check"} ) - # Read the resources again, because the update replaced the private endpoint. + # Read the resources again after the update. # FHIR is not in this list: Azure keeps publicNetworkAccess Enabled on FHIR, also after a request # sets Disabled. The private endpoint on the workspace blocks public traffic to FHIR (checked above). network = {} diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index e6cc49c276..d7749e9afe 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -1,6 +1,7 @@ """Check the private network settings of the Health bundle.""" from pathlib import Path +import re import unittest ROOT = Path(__file__).resolve().parents[2] @@ -11,15 +12,44 @@ def block(source, header): return source.split(header + " {", 1)[1].split("\n}", 1)[0] +def version(text): + return tuple(int(part) for part in text.split(".")) + + class HealthBundleContractTests(unittest.TestCase): - def test_endpoint_is_replaced_when_the_services_change(self): - # Azure does not always add a DNS record for a service that joins after the endpoint. + def test_dns_zone_group_gets_the_dicom_zone_without_an_endpoint_change(self): + # A zone group change in the endpoint resource also sends the whole endpoint. Azure + # refuses this with HTTP 409 while it links a new service, so the group is separate. source = (BUNDLE / "main.tf").read_text() - services = block(source, 'resource "terraform_data" "private_endpoint_services"') - self.assertRegex(services, r"\n fhir\s*=\s*var\.deploy_fhir\n") - self.assertRegex(services, r"\n dicom\s*=\s*var\.deploy_dicom\n") endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') - self.assertRegex(endpoint, r"replace_triggered_by\s*=\s*\[terraform_data\.private_endpoint_services\]") + self.assertNotIn("private_dns_zone_group {", endpoint) + self.assertIn("ignore_changes = [tags, private_dns_zone_group]", endpoint) + self.assertNotIn("replace_triggered_by", endpoint) + + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + self.assertIn("parent_id = azurerm_private_endpoint.health_services_private_endpoint.id", group) + self.assertIn("var.deploy_dicom ? [{", group) + self.assertIn('error_message_regex = ["AnotherOperationInProgress"]', group) + self.assertIn("azurerm_healthcare_dicom_service.dicom", group.split("depends_on", 1)[1]) + + def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): + source = (BUNDLE / "main.tf").read_text() + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + remove = block(source, 'resource "azapi_resource_action" "remove_endpoint_dns_zone_group"') + self.assertIn('method = "DELETE"', remove) + self.assertIn('/privateDnsZoneGroups/private-dns-zone-group"', remove) + self.assertIn('name = "private-dns-zone-group"', group) + self.assertIn("azapi_resource_action.remove_endpoint_dns_zone_group", group) + + def test_azapi_can_update_resources_with_the_terraform_of_the_bundle(self): + # Before 2.9.0, azapi returns no resource identity after an update. Terraform 1.12 and + # later keep the identity, but earlier versions stop the update with an error. + porter = (BUNDLE.parent / "porter.yaml").read_text() + providers = (BUNDLE / "providers.tf").read_text() + terraform = re.search(r"- terraform:\s*\n\s*clientVersion:\s*([\d.]+)", porter).group(1) + azapi = re.search(r'source\s*=\s*"Azure/azapi"\s*\n\s*version\s*=\s*"=\s*([\d.]+)"', providers).group(1) + if version(terraform) < (1, 12): + self.assertGreaterEqual(version(azapi), (2, 9, 0)) def test_dicom_public_network_access_is_off(self): # The provider turns it on unless the bundle sets it. diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 4dc4af76cd..7ef063ddb2 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.9 +version: 0.3.10 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl index 23c631c6bc..b93ee2ce7f 100644 --- a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl +++ b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl @@ -1,6 +1,29 @@ # This file is maintained automatically by "terraform init". # Manual edits may be lost in future updates. +provider "registry.terraform.io/azure/azapi" { + version = "2.9.0" + constraints = "2.9.0" + hashes = [ + "h1:XtiXcDMV+sEfQFcRot9fwKF/Y0aXeho9A/WVfHdHwCc=", + "h1:ivmN39jnMJRZXlPuX8r5TTXM8lyF8ZocPQQgEUy61eM=", + "h1:lN0C7TI2dqStnZysEBhDq33t9DbpUp/8rOWxZUF/st0=", + "h1:tF2SjvM2Vj+7MTbFXhssLYu1Uc9uK/cVWBYIH8heDqM=", + "zh:0a4eee8c9362db6ca19d371eccf38701c8306be761182da87b624294f7e8f867", + "zh:4df87bee5b8f4cce27461ae26132a599542583cdf035942b42b0259a514ab46e", + "zh:57dc1f4227f1d0eab630fcf868d6c9a1b4dc4c165608ce5a4d7028a482770547", + "zh:5c500e42c419c324ea8f41c40e4cc8af187323cebc9f70543749b6c0e59c0fe9", + "zh:6c7dca31242e27d2f215b1a9370b65579b3a988282a3609564ea96866f86f0ca", + "zh:74dada7351b25d01e61aa70aa8cf1f1fb96d09c514be4b66d12816c5e61f9e01", + "zh:9c67c0727ab8be15879793f929f3d71f0f149ae1daaf577d0bd4d57b2e61c408", + "zh:a0266da16db14b635a61d6766efc28468056bac91ed6662fc9f2a0aad923b063", + "zh:a4a555627f40fa797b34ddede599812cddb1e0e9be105ccfcf1f293451b97cd7", + "zh:a6a1c4a46f4f01e4420a456adefe8bcf3bff3c9d09a8dfeedeea5b94c866dfd8", + "zh:c64fc4067a8276d4405d9359990e0c45e66dba07433a1f3ab1e6c32f6ef3b048", + "zh:ee77a881d071b3a0ad8a64f9c1158e3aee9b37e9063de5dd8fa9d238dbe70ba1", + ] +} + provider "registry.terraform.io/hashicorp/azurerm" { version = "5.11.0" constraints = "5.11.0" diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index eaa6acd15e..a7cf4ae558 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -46,16 +46,6 @@ resource "azurerm_healthcare_dicom_service" "dicom" { lifecycle { ignore_changes = [tags] } } -# Azure writes the private DNS records when it creates the endpoint. It does not -# always add a record for a service that joins the workspace later, so the -# endpoint is replaced when the set of services changes. -resource "terraform_data" "private_endpoint_services" { - input = { - fhir = var.deploy_fhir - dicom = var.deploy_dicom - } -} - resource "azurerm_private_endpoint" "health_services_private_endpoint" { name = "pe-${azurerm_healthcare_workspace.healthcare_workspace.name}" location = data.azurerm_resource_group.ws.location @@ -63,11 +53,6 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { subnet_id = data.azurerm_subnet.services.id tags = local.workspace_service_tags - private_dns_zone_group { - name = "private-dns-zone-group" - private_dns_zone_ids = [data.azurerm_private_dns_zone.health.id, data.azurerm_private_dns_zone.dicom.id] - } - private_service_connection { private_connection_resource_id = azurerm_healthcare_workspace.healthcare_workspace.id name = "psc-${azurerm_healthcare_workspace.healthcare_workspace.name}" @@ -80,8 +65,47 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { azurerm_healthcare_dicom_service.dicom ] - lifecycle { - ignore_changes = [tags] - replace_triggered_by = [terraform_data.private_endpoint_services] + # The DNS zone group is a separate resource. The provider changes a zone group in this + # resource only together with the whole endpoint. + lifecycle { ignore_changes = [tags, private_dns_zone_group] } +} + +# Before version 0.3.10, the endpoint resource created the DNS zone group. This removes +# that group once, so that the resource below can create a group with the same name. +resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { + type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" + resource_id = "${azurerm_private_endpoint.health_services_private_endpoint.id}/privateDnsZoneGroups/private-dns-zone-group" + method = "DELETE" + + retry = { + error_message_regex = ["AnotherOperationInProgress"] + } +} + +# Azure writes the private DNS records of a zone when the group gets that zone. Azure +# links a new service to the endpoint after the service create returns. A change to the +# group during this time gets HTTP 409, so the provider tries again. +resource "azapi_resource" "health_services_dns_zone_group" { + type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" + name = "private-dns-zone-group" + parent_id = azurerm_private_endpoint.health_services_private_endpoint.id + + body = { + properties = { + privateDnsZoneConfigs = concat( + [{ name = "health", properties = { privateDnsZoneId = data.azurerm_private_dns_zone.health.id } }], + var.deploy_dicom ? [{ name = "dicom", properties = { privateDnsZoneId = data.azurerm_private_dns_zone.dicom.id } }] : [] + ) + } } + + retry = { + error_message_regex = ["AnotherOperationInProgress"] + } + + depends_on = [ + azapi_resource_action.remove_endpoint_dns_zone_group, + azurerm_healthcare_fhir_service.fhir, + azurerm_healthcare_dicom_service.dicom + ] } diff --git a/templates/workspace_services/health-services/terraform/providers.tf b/templates/workspace_services/health-services/terraform/providers.tf index b2f9f931fc..68f6934183 100644 --- a/templates/workspace_services/health-services/terraform/providers.tf +++ b/templates/workspace_services/health-services/terraform/providers.tf @@ -6,6 +6,11 @@ terraform { source = "hashicorp/azurerm" version = "= 5.11.0" } + # Versions before 2.9.0 fail each update when Terraform is earlier than 1.12. + azapi = { + source = "Azure/azapi" + version = "= 2.9.0" + } } backend "azurerm" {} @@ -31,6 +36,8 @@ provider "azurerm" { storage_use_azuread = true } +provider "azapi" {} + module "terraform_azurerm_environment_configuration" { source = "git::https://github.com/microsoft/terraform-azurerm-environment-configuration.git?ref=0.7.0" arm_environment = var.arm_environment From 4c1af667e87ae3f809288e73ffb1acac23a3f4bc Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 10:08:30 +0000 Subject: [PATCH 11/28] Create the Health services after the private endpoint (#5147) Terraform created the Health private endpoint after FHIR and DICOM, and deleted it before them. So during an install, the services existed before the endpoint, and during an uninstall, they existed after it. FHIR and DICOM now depend on the endpoint, so Terraform creates them after it and deletes them before it. The DNS zone group still waits for the services, because Azure writes a record only for a service that exists. A unit test makes sure of the order. --- CHANGELOG.md | 1 + .../workspace-services/health_services.md | 1 + e2e_tests/unit_tests/test_health_bundle_contract.py | 8 ++++++++ .../workspace_services/health-services/porter.yaml | 2 +- .../health-services/terraform/main.tf | 10 +++++----- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1808da05bc..f05921009a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,7 @@ BUG FIXES: * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index bad6f6e0ae..4909e26982 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -23,6 +23,7 @@ To repair the service, delete its FHIR service in the Azure portal, and then del ## Private network access The service connects the healthcare workspace to the workspace network through one private endpoint. +Terraform creates the endpoint before FHIR and DICOM, and deletes FHIR and DICOM before the endpoint. Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index d7749e9afe..b3c2eef4c0 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -32,6 +32,14 @@ def test_dns_zone_group_gets_the_dicom_zone_without_an_endpoint_change(self): self.assertIn('error_message_regex = ["AnotherOperationInProgress"]', group) self.assertIn("azurerm_healthcare_dicom_service.dicom", group.split("depends_on", 1)[1]) + def test_services_are_created_after_the_private_endpoint(self): + source = (BUNDLE / "main.tf").read_text() + endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') + self.assertNotIn("depends_on", endpoint) + for header in ('resource "azurerm_healthcare_fhir_service" "fhir"', 'resource "azurerm_healthcare_dicom_service" "dicom"'): + service = block(source, header) + self.assertIn("depends_on = [azurerm_private_endpoint.health_services_private_endpoint]", service) + def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): source = (BUNDLE / "main.tf").read_text() group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 7ef063ddb2..71f34eabd0 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.10 +version: 0.3.11 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index a7cf4ae558..b2ff3a544c 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -25,6 +25,8 @@ resource "azurerm_healthcare_fhir_service" "fhir" { type = "SystemAssigned" } + # Terraform creates the services after the private endpoint and deletes them before it. + depends_on = [azurerm_private_endpoint.health_services_private_endpoint] lifecycle { ignore_changes = [tags] } } @@ -43,6 +45,9 @@ resource "azurerm_healthcare_dicom_service" "dicom" { type = "SystemAssigned" } + # Terraform creates the services after the private endpoint and deletes them before it. + depends_on = [azurerm_private_endpoint.health_services_private_endpoint] + lifecycle { ignore_changes = [tags] } } @@ -60,11 +65,6 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { is_manual_connection = false } - depends_on = [ - azurerm_healthcare_fhir_service.fhir, - azurerm_healthcare_dicom_service.dicom - ] - # The DNS zone group is a separate resource. The provider changes a zone group in this # resource only together with the whole endpoint. lifecycle { ignore_changes = [tags, private_dns_zone_group] } From 7df22dfb3cb15de26768616cf3280db5ba2b6a51 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 10:08:44 +0000 Subject: [PATCH 12/28] Keep the Health data roles when the service changes (#5147) Live runs showed that each Health update replaced the FHIR and DICOM role assignments of the workspace groups. The role definition data source gives an ID without the subscription, but Azure returns the ID with the subscription. So the provider planned a replacement, and the groups lost their data roles until Azure applied the new assignments. The role assignments now use the role names. The provider reads the name back, so the plan after an upgrade shows no change. A unit test makes sure that the file uses no role definition ID. --- CHANGELOG.md | 1 + .../unit_tests/test_health_bundle_contract.py | 7 +++ .../health-services/terraform/roles.tf | 43 ++++++++----------- 3 files changed, 27 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f05921009a..6175acf25c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,7 @@ BUG FIXES: * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index b3c2eef4c0..74b5c1b7fa 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -64,3 +64,10 @@ def test_dicom_public_network_access_is_off(self): source = (BUNDLE / "main.tf").read_text() dicom = block(source, 'resource "azurerm_healthcare_dicom_service" "dicom"') self.assertRegex(dicom, r"\n public_network_access_enabled\s*=\s*false\n") + + def test_role_assignments_use_the_role_names(self): + # Azure returns the role definition ID with the subscription. The role definition data + # source gives the ID without it, so the plan replaced each role assignment. + roles = (BUNDLE / "roles.tf").read_text() + self.assertNotIn("role_definition_id", roles) + self.assertEqual(roles.count("role_definition_name"), 4) diff --git a/templates/workspace_services/health-services/terraform/roles.tf b/templates/workspace_services/health-services/terraform/roles.tf index 8af1f03ed3..f3cef02425 100644 --- a/templates/workspace_services/health-services/terraform/roles.tf +++ b/templates/workspace_services/health-services/terraform/roles.tf @@ -1,37 +1,32 @@ +# Azure returns the role definition ID with the subscription. An ID without it in the +# configuration replaces each role assignment at each update, so these use the role name. + # Role assignments for workspace researchers group resource "azurerm_role_assignment" "researchers_fhir_contributor" { - count = var.deploy_fhir && var.workspace_researchers_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_fhir_service.fhir[0].id - role_definition_id = data.azurerm_role_definition.azure_fhir_contributor.id - principal_id = var.workspace_researchers_group_id + count = var.deploy_fhir && var.workspace_researchers_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_fhir_service.fhir[0].id + role_definition_name = "FHIR Data Contributor" + principal_id = var.workspace_researchers_group_id } resource "azurerm_role_assignment" "researchers_dicom_data_owner" { - count = var.deploy_dicom && var.workspace_researchers_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_dicom_service.dicom[0].id - role_definition_id = data.azurerm_role_definition.azure_dicom_data_owner.id - principal_id = var.workspace_researchers_group_id + count = var.deploy_dicom && var.workspace_researchers_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_dicom_service.dicom[0].id + role_definition_name = "DICOM Data Owner" + principal_id = var.workspace_researchers_group_id } # Role assignments for workspace owners group resource "azurerm_role_assignment" "owners_fhir_contributor" { - count = var.deploy_fhir && var.workspace_owners_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_fhir_service.fhir[0].id - role_definition_id = data.azurerm_role_definition.azure_fhir_contributor.id - principal_id = var.workspace_owners_group_id + count = var.deploy_fhir && var.workspace_owners_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_fhir_service.fhir[0].id + role_definition_name = "FHIR Data Contributor" + principal_id = var.workspace_owners_group_id } resource "azurerm_role_assignment" "owners_dicom_data_owner" { - count = var.deploy_dicom && var.workspace_owners_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_dicom_service.dicom[0].id - role_definition_id = data.azurerm_role_definition.azure_dicom_data_owner.id - principal_id = var.workspace_owners_group_id -} - -data "azurerm_role_definition" "azure_fhir_contributor" { - name = "FHIR Data Contributor" -} - -data "azurerm_role_definition" "azure_dicom_data_owner" { - name = "DICOM Data Owner" + count = var.deploy_dicom && var.workspace_owners_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_dicom_service.dicom[0].id + role_definition_name = "DICOM Data Owner" + principal_id = var.workspace_owners_group_id } From 60a3f8e6234f24ce99f3ba8b7ca902551d8ba1cf Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:43:38 +0000 Subject: [PATCH 13/28] Wrap a long CHANGELOG entry The Markdown lint rejects a line with more than 400 characters. The entry now continues on lines that start with two spaces, as an earlier entry in the file does. The rendered text does not change. --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6175acf25c..88f7d77df2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,8 @@ BUG FIXES: * Replace the Health private endpoint when an update adds or removes FHIR or DICOM. Azure wrote private DNS records only for the services that existed when it created the endpoint, so a DICOM service that an update added resolved to a public address. An existing service that got FHIR or DICOM after its install keeps the missing record until the endpoint is created again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. + The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) From fa09cf34e46617c17210ed793c64fe5a403339e8 Mon Sep 17 00:00:00 2001 From: Jay W Date: Mon, 26 Oct 2026 13:20:50 +0000 Subject: [PATCH 14/28] Format the Health bundle contract test The ruff-format hook of pre-commit (ruff 0.15.17) splits a line of the test that is longer than the line limit. The test does not change. --- e2e_tests/unit_tests/test_health_bundle_contract.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index 74b5c1b7fa..ce2bf1739d 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -36,7 +36,10 @@ def test_services_are_created_after_the_private_endpoint(self): source = (BUNDLE / "main.tf").read_text() endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') self.assertNotIn("depends_on", endpoint) - for header in ('resource "azurerm_healthcare_fhir_service" "fhir"', 'resource "azurerm_healthcare_dicom_service" "dicom"'): + for header in ( + 'resource "azurerm_healthcare_fhir_service" "fhir"', + 'resource "azurerm_healthcare_dicom_service" "dicom"', + ): service = block(source, header) self.assertIn("depends_on = [azurerm_private_endpoint.health_services_private_endpoint]", service) From 3d33076fa062b9c93b7e28fd34cc4fdb24619123 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 21:12:20 +0000 Subject: [PATCH 15/28] Give the DICOM data roles after the endpoint link (#5147) Terraform gave the workspace groups the role "DICOM Data Owner" immediately after the DICOM create. The two role assignments now wait for the DNS zone group of the Health private endpoint. Azure accepts a change to the zone group only after it links the new DICOM service to the endpoint. A contract test makes sure that both DICOM role assignments depend on the zone group. Version: `tre-workspace-service-health` 0.3.12. --- CHANGELOG.md | 1 + e2e_tests/unit_tests/test_health_bundle_contract.py | 7 +++++++ templates/workspace_services/health-services/porter.yaml | 2 +- .../workspace_services/health-services/terraform/roles.tf | 6 ++++++ 4 files changed, 15 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 88f7d77df2..8cd1c555e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,6 +57,7 @@ BUG FIXES: * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.12) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index ce2bf1739d..d6eb18cd70 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -74,3 +74,10 @@ def test_role_assignments_use_the_role_names(self): roles = (BUNDLE / "roles.tf").read_text() self.assertNotIn("role_definition_id", roles) self.assertEqual(roles.count("role_definition_name"), 4) + + def test_dicom_roles_wait_for_the_endpoint_link(self): + # Azure accepts the zone group write only after it links a new DICOM service to the endpoint. + roles = (BUNDLE / "roles.tf").read_text() + for name in ("researchers_dicom_data_owner", "owners_dicom_data_owner"): + role = block(roles, f'resource "azurerm_role_assignment" "{name}"') + self.assertIn("depends_on = [azapi_resource.health_services_dns_zone_group]", role) diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 71f34eabd0..2846e79ac6 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.11 +version: 0.3.12 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/roles.tf b/templates/workspace_services/health-services/terraform/roles.tf index f3cef02425..7c020ca782 100644 --- a/templates/workspace_services/health-services/terraform/roles.tf +++ b/templates/workspace_services/health-services/terraform/roles.tf @@ -14,6 +14,9 @@ resource "azurerm_role_assignment" "researchers_dicom_data_owner" { scope = azurerm_healthcare_dicom_service.dicom[0].id role_definition_name = "DICOM Data Owner" principal_id = var.workspace_researchers_group_id + + # The zone group write succeeds only after Azure links a new DICOM service to the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } # Role assignments for workspace owners group @@ -29,4 +32,7 @@ resource "azurerm_role_assignment" "owners_dicom_data_owner" { scope = azurerm_healthcare_dicom_service.dicom[0].id role_definition_name = "DICOM Data Owner" principal_id = var.workspace_owners_group_id + + # The zone group write succeeds only after Azure links a new DICOM service to the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } From fe7c11b8f4397780d6abe8693e196034ad07b7ec Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 23:08:46 +0000 Subject: [PATCH 16/28] Use one Health version bump Earlier commits bumped the Health bundle four times on an earlier main. Set one bump from the current main, 0.3.9, in the bundle, the CHANGELOG, the document and the Terraform comment. Remove the CHANGELOG entries that later commits replaced. --- CHANGELOG.md | 14 +++++--------- .../workspace-services/health_services.md | 8 ++++---- .../workspace_services/health-services/porter.yaml | 2 +- .../health-services/terraform/main.tf | 2 +- 4 files changed, 11 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8cd1c555e9..590706e155 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,6 @@ Setting to `false` will delete existing airlock storage accounts and must only b ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) -* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. The generic workspace service install case no longer installs Health, because its CI workspace has no Entra ID groups. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) @@ -49,15 +48,12 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) -* Replace the Health private endpoint when an update adds or removes FHIR or DICOM. Azure wrote private DNS records only for the services that existed when it created the endpoint, so a DICOM service that an update added resolved to a public address. An existing service that got FHIR or DICOM after its install keeps the missing record until the endpoint is created again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. - The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.10) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.11) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.12) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 4909e26982..c66163587a 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -30,15 +30,15 @@ If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure w Azure links a new service to the endpoint some minutes after Terraform creates the service. During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. -Before version 0.3.10, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. +Before version 0.3.9, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. If you added DICOM to an existing service before this version, the DICOM host can resolve to a public address. -An upgrade to version 0.3.10 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. +An upgrade to version 0.3.9 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. Do not delete the private endpoint to repair the service. The DICOM service has public network access turned off. -Before version 0.3.10, Terraform created the DICOM service with public network access on. -An upgrade to version 0.3.10 turns it off. +Before version 0.3.9, Terraform created the DICOM service with public network access on. +An upgrade to version 0.3.9 turns it off. The private endpoint on the healthcare workspace blocks public traffic to FHIR and DICOM. A request from outside the workspace network gets HTTP 403. diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 2846e79ac6..4dc4af76cd 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.12 +version: 0.3.9 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index b2ff3a544c..d4dc5b0ea1 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -70,7 +70,7 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { lifecycle { ignore_changes = [tags, private_dns_zone_group] } } -# Before version 0.3.10, the endpoint resource created the DNS zone group. This removes +# Before version 0.3.9, the endpoint resource created the DNS zone group. This removes # that group once, so that the resource below can create a group with the same name. resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" From f4f5575a8887865404e220373e0a85afd145dcbe Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 01:45:49 +0000 Subject: [PATCH 17/28] Use one owner token in the Health test, as the MySQL test does (#5147) #5282 removed the cleanup_token parameter of SQLResources.create, because the owner token gets a current token for each request. The Health test still passed cleanup_token, so on this branch the case stopped with a TypeError at the first create, after the setup created the workspace and the Guacamole service. The earlier live passes used a branch that still had the parameter. --- e2e_tests/test_health.py | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py index 8b41f619de..b9e8900f03 100644 --- a/e2e_tests/test_health.py +++ b/e2e_tests/test_health.py @@ -62,16 +62,14 @@ async def test_private_fhir_access_and_dicom_update(verify): async with sql_lifecycle(verify, deadline_variable="HEALTH_VALIDATION_DEADLINE") as resources: # The bundle refuses to install without the workspace owners group. workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup(groups=True) - new_token = partial(get_workspace_owner_token, workspace_id, verify) - workspace = (await get_resource(f"/api{workspace_path}", await new_token(), verify))["workspace"] + # The owner token gets a current token for each request, so one token lasts the case and its clean-up. + token = await get_workspace_owner_token(workspace_id, verify) + workspace = (await get_resource(f"/api{workspace_path}", token, verify))["workspace"] assert workspace["properties"].get("workspace_owners_group_id"), "The workspace has no owners group" health_path, health_id = await resources.create( - HEALTH_PAYLOAD, f"/api{workspace_path}/workspace-services", await new_token(), cleanup_token=new_token + HEALTH_PAYLOAD, f"/api{workspace_path}/workspace-services", token ) - vm_path, vm_id = await resources.create( - VM_PAYLOAD, f"/api{guacamole_path}/user-resources", await new_token(), cleanup_token=new_token - ) - token = await new_token() + vm_path, vm_id = await resources.create(VM_PAYLOAD, f"/api{guacamole_path}/user-resources", token) health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] vm_properties = (await get_resource(f"/api{vm_path}", token, verify))["userResource"]["properties"] subscription_id = vm_properties.get("workspace_subscription_id") or os.environ["ARM_SUBSCRIPTION_ID"] @@ -96,7 +94,6 @@ async def test_private_fhir_access_and_dicom_update(verify): LOGGER.info("FHIR metadata is private, and data requests need a token and a role") async with role_access(arm, fhir_id, FHIR_DATA_CONTRIBUTOR, vm["identity"]["principalId"], subscription_id): await probe(phase="write") - token = await new_token() health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] async with asyncio.timeout(60 * 60): await post_resource( @@ -107,7 +104,7 @@ async def test_private_fhir_access_and_dicom_update(verify): method="PATCH", etag=health["_etag"], ) - health = (await get_resource(f"/api{health_path}", await new_token(), verify))["workspaceService"] + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] assert health["properties"]["dicom_url"] == dicom_url dicom_id = f"{workspace_arm_id}/dicomservices/dicom{suffix}" dicom = await arm.request("GET", dicom_id, HEALTH_API) From bba7e0aec4c65eed9aeaffc82ced2c36c6c6d9d2 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 01:45:54 +0000 Subject: [PATCH 18/28] Create the Health DNS zone group again when FHIR changes (#5147) The FHIR record is in the same private DNS zone as the record of the healthcare workspace. So an update that adds FHIR does not change the body of the zone group, azapi sends no request, and Azure does not write the FHIR record. The same thing occurred for DICOM before this pull request. A terraform_data resource with the deploy_fhir value now makes Terraform replace the zone group, as the upgrade to 0.3.9 does. A contract test, the Health document and the CHANGELOG describe it. --- CHANGELOG.md | 3 ++- docs/tre-templates/workspace-services/health_services.md | 2 ++ e2e_tests/unit_tests/test_health_bundle_contract.py | 8 ++++++++ .../workspace_services/health-services/terraform/main.tf | 8 ++++++++ 4 files changed, 20 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 590706e155..7f46311a6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,7 +49,8 @@ BUG FIXES: * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. - The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. + The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index c66163587a..7c103c753d 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -27,6 +27,8 @@ Terraform creates the endpoint before FHIR and DICOM, and deletes FHIR and DICOM Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. +The FHIR record is in the same zone as the workspace record. So if an update adds or removes FHIR, Terraform deletes the group and creates it again, and Azure writes the records again. +Private access to the services stops until the new group exists. Azure links a new service to the endpoint some minutes after Terraform creates the service. During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index d6eb18cd70..0c46460d9b 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -32,6 +32,14 @@ def test_dns_zone_group_gets_the_dicom_zone_without_an_endpoint_change(self): self.assertIn('error_message_regex = ["AnotherOperationInProgress"]', group) self.assertIn("azurerm_healthcare_dicom_service.dicom", group.split("depends_on", 1)[1]) + def test_an_update_that_adds_fhir_creates_the_zone_group_again(self): + # The FHIR record is in the zone of the workspace record, so the group body stays the same. + source = (BUNDLE / "main.tf").read_text() + trigger = block(source, 'resource "terraform_data" "fhir_deployed"') + self.assertIn("input = var.deploy_fhir", trigger) + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + self.assertIn("replace_triggered_by = [terraform_data.fhir_deployed]", group) + def test_services_are_created_after_the_private_endpoint(self): source = (BUNDLE / "main.tf").read_text() endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index d4dc5b0ea1..09eea6ce7d 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -82,6 +82,12 @@ resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { } } +# The workspace and FHIR records are in the same zone, so an update that adds FHIR does not +# change the group. A new group makes Azure write the records of all services of the endpoint. +resource "terraform_data" "fhir_deployed" { + input = var.deploy_fhir +} + # Azure writes the private DNS records of a zone when the group gets that zone. Azure # links a new service to the endpoint after the service create returns. A change to the # group during this time gets HTTP 409, so the provider tries again. @@ -108,4 +114,6 @@ resource "azapi_resource" "health_services_dns_zone_group" { azurerm_healthcare_fhir_service.fhir, azurerm_healthcare_dicom_service.dicom ] + + lifecycle { replace_triggered_by = [terraform_data.fhir_deployed] } } From 4ad9b95de87c0af8a0fb7901754ba8cc7e72cea3 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 01:45:59 +0000 Subject: [PATCH 19/28] Merge the PR bot marker sentences and remove a clean-up without effect - The restack kept the marker sentence of main and added the sentence of the earlier branch. One sentence now names all seven markers. - Add the blank line before the Health section of the E2E document. - The finally block of health_probe.ps1 cleared a script variable that nothing set. The token variable is local to Get-FhirHeaders. --- docs/tre-developers/end-to-end-tests.md | 1 + docs/tre-developers/github-pr-bot-commands.md | 3 +-- e2e_tests/resources/health_probe.ps1 | 2 -- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 8e7a34d1e1..25b58f7c46 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -204,6 +204,7 @@ The last start has the same OIDC settings, so the log must show only one stop fo The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. + ### Health selection Select `tre-workspace-service-health` to run `test_health.py::test_private_fhir_access_and_dicom_update`. diff --git a/docs/tre-developers/github-pr-bot-commands.md b/docs/tre-developers/github-pr-bot-commands.md index 61da508abc..0a67f6c912 100644 --- a/docs/tre-developers/github-pr-bot-commands.md +++ b/docs/tre-developers/github-pr-bot-commands.md @@ -22,8 +22,7 @@ The table describes the selected assertions. A passing command does not prove ev | `/test-airlock` | Import requests, access checks, the import review VM flow and export request storage routing | No export review VM or migration proof. | | `/test-backups` | Base workspace deployment with backups enabled and disabled | No backup restore or retention proof. | -The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `gitea_validation` markers have no dedicated slash command. -The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `health_validation` markers have no dedicated slash command. +The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation`, `gitea_validation` and `health_validation` markers have no dedicated slash command. Use the branch workflow or the local test runner for these selections. The `airlock_validation` marker selects two Airlock cases. It does not select the full `airlock` suite. See [bundle selection](end-to-end-tests.md#validate-one-bundle) for exact cases and declared coverage gaps. diff --git a/e2e_tests/resources/health_probe.ps1 b/e2e_tests/resources/health_probe.ps1 index 7c049e76de..828e0d9924 100644 --- a/e2e_tests/resources/health_probe.ps1 +++ b/e2e_tests/resources/health_probe.ps1 @@ -103,6 +103,4 @@ try { Write-Output "HEALTH_PROBE_FAILED=$stage" if ($null -ne $response) { Write-Output "HEALTH_PROBE_STATUS=$([int]$response.StatusCode)" } exit 1 -} finally { - $identity = $null } From 88988b9fa1ccf0faa0021c1bd16041e8169d3737 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 09:05:02 +0000 Subject: [PATCH 20/28] Delete the old Health zone group after the FHIR and DICOM changes The one-time delete of the zone group of the endpoint resource now waits for FHIR and DICOM. Before, it ran at the start of the upgrade, and the records were absent until the DICOM update and the link completed. If a change to a service fails, the old group now stays. The Health document now says that an update that adds or removes FHIR deletes the group before the FHIR change. It also says that private access stays stopped if the FHIR change fails, and how to repair it. The CHANGELOG entry names the stop, and two long sentences are split. --- CHANGELOG.md | 6 +++--- docs/tre-templates/workspace-services/health_services.md | 6 ++++-- e2e_tests/unit_tests/test_health_bundle_contract.py | 4 ++++ .../workspace_services/health-services/terraform/main.tf | 7 +++++++ 4 files changed, 18 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f46311a6d..c10f4fb44a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ Setting to `false` will delete existing airlock storage accounts and must only b ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) -* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, gets HTTP 401 without a token and HTTP 403 without a role, then writes and reads a Patient across an update that adds DICOM. The generic workspace service install case no longer installs Health, because its CI workspace has no Entra ID groups. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, and gets HTTP 401 without a token and HTTP 403 without a role. Then the VM writes and reads a Patient across an update that adds DICOM. The generic workspace service install case no longer installs Health, because its CI workspace has no Entra ID groups. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) * Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. @@ -48,9 +48,9 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) -* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group, so a DICOM service that an update added resolved to a public address. +* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group. So a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. - The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 7c103c753d..661fc2ddd3 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -27,14 +27,16 @@ Terraform creates the endpoint before FHIR and DICOM, and deletes FHIR and DICOM Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. -The FHIR record is in the same zone as the workspace record. So if an update adds or removes FHIR, Terraform deletes the group and creates it again, and Azure writes the records again. -Private access to the services stops until the new group exists. +The FHIR record is in the same zone as the workspace record. So if an update adds or removes FHIR, Terraform deletes the group before the FHIR change and creates it again after the change. Azure then writes the records again. +Private access to the healthcare workspace and DICOM stops until the new group exists. For an update that adds FHIR, this time includes the FHIR create, which took 9 to 12 minutes in tests. +If the FHIR change fails, the new group does not exist, and private access stays stopped. To repair the service, run the update again. Azure links a new service to the endpoint some minutes after Terraform creates the service. During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. Before version 0.3.9, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. If you added DICOM to an existing service before this version, the DICOM host can resolve to a public address. An upgrade to version 0.3.9 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. +Terraform deletes the old group after the changes to FHIR and DICOM, directly before it creates the new group. Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. Do not delete the private endpoint to repair the service. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index 0c46460d9b..d5d0bfcb62 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -59,6 +59,10 @@ def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): self.assertIn('/privateDnsZoneGroups/private-dns-zone-group"', remove) self.assertIn('name = "private-dns-zone-group"', group) self.assertIn("azapi_resource_action.remove_endpoint_dns_zone_group", group) + # The old group stays until FHIR and DICOM have their changes. + depends_on = remove.split("depends_on", 1)[1] + self.assertIn("azurerm_healthcare_fhir_service.fhir", depends_on) + self.assertIn("azurerm_healthcare_dicom_service.dicom", depends_on) def test_azapi_can_update_resources_with_the_terraform_of_the_bundle(self): # Before 2.9.0, azapi returns no resource identity after an update. Terraform 1.12 and diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 09eea6ce7d..519e88408b 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -80,6 +80,13 @@ resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { retry = { error_message_regex = ["AnotherOperationInProgress"] } + + # The delete waits for the changes to FHIR and DICOM, so that the records are absent only + # until the new group exists. If a change to a service fails, the old group stays. + depends_on = [ + azurerm_healthcare_fhir_service.fhir, + azurerm_healthcare_dicom_service.dicom + ] } # The workspace and FHIR records are in the same zone, so an update that adds FHIR does not From 5b3c7b45aac55d310322c55fd7f07e591babc891 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 09:05:17 +0000 Subject: [PATCH 21/28] Name azurerm 4.75.0 as the first version with the FHIR create fix azurerm 4.75.0 removed the wait for 12 Succeeded reads after the FHIR create. Version 4.74.0 still has it. The CHANGELOG entry, the provider comment and the provider test named 5.3.0, which changed only the delete wait. The test now requires 4.75.0 or later. The pin stays at 5.11.0, which the live runs used. --- CHANGELOG.md | 2 +- e2e_tests/unit_tests/test_health_bundle_providers.py | 4 ++-- .../workspace_services/health-services/terraform/providers.tf | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c10f4fb44a..95950c685a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,7 +111,7 @@ BUG FIXES: * Fix Guacamole login failures in large workspaces: raise the user-resources API timeout from 5s to 30s, batch template version lookups into one Cosmos query, and query VM power states concurrently. (`tre-service-guacamole` 0.14.6, `API` 0.27.31) ([#5099](https://github.com/microsoft/AzureTRE/issues/5099)) * Fix API resource update retries so a successful retry returns the patched resource to the original caller. (`API` 0.27.32) ([#5025](https://github.com/microsoft/AzureTRE/pull/5025)) * Fix the health check falsely returning OK when Cosmos is down or inaccessible. ([#4926](https://github.com/microsoft/AzureTRE/issues/4926)) -* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 5.3.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) ## (0.29.1) **BREAKING CHANGES** diff --git a/e2e_tests/unit_tests/test_health_bundle_providers.py b/e2e_tests/unit_tests/test_health_bundle_providers.py index 4b2b4efe3d..d7c4052d7f 100644 --- a/e2e_tests/unit_tests/test_health_bundle_providers.py +++ b/e2e_tests/unit_tests/test_health_bundle_providers.py @@ -14,13 +14,13 @@ def version(text): class HealthBundleProviderTests(unittest.TestCase): def test_azurerm_keeps_a_new_fhir_service_in_the_state(self): - # Before 5.3.0, azurerm waits for Succeeded after the FHIR create and saves the ID after this + # Before 4.75.0, azurerm waits for Succeeded after the FHIR create and saves the ID after this # wait. If Azure reports Accepted, the create fails, and the state does not get the service. providers = (BUNDLE / "providers.tf").read_text() lock = (BUNDLE / ".terraform.lock.hcl").read_text() azurerm = re.search(r'source\s*=\s*"hashicorp/azurerm"\s*\n\s*version\s*=\s*"=\s*([\d.]+)"', providers).group(1) locked = re.search(r'"registry.terraform.io/hashicorp/azurerm" \{\s*version\s*=\s*"([\d.]+)"', lock).group(1) - self.assertGreaterEqual(version(azurerm), (5, 3, 0)) + self.assertGreaterEqual(version(azurerm), (4, 75, 0)) self.assertEqual(locked, azurerm) diff --git a/templates/workspace_services/health-services/terraform/providers.tf b/templates/workspace_services/health-services/terraform/providers.tf index 68f6934183..e74ffb7415 100644 --- a/templates/workspace_services/health-services/terraform/providers.tf +++ b/templates/workspace_services/health-services/terraform/providers.tf @@ -1,7 +1,7 @@ # Azure Provider source and version being used terraform { required_providers { - # Before 5.3.0, a FHIR create that Azure reports as Accepted fails, and the state does not get the service. + # Before 4.75.0, a FHIR create that Azure reports as Accepted fails, and the state does not get the service. azurerm = { source = "hashicorp/azurerm" version = "= 5.11.0" From 31eb9ea06836cc76b6545e1905f38ce8916bbccb Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 12:39:40 +0000 Subject: [PATCH 22/28] Create the Health private endpoint after FHIR and DICOM again (#5147) On 31 October 2026, Azure failed each private endpoint to a healthcare workspace without a FHIR or DICOM service with InternalServerError. Two live installs failed this way. A minimal Terraform configuration got the same error with azurerm 5.11.0 and 4.58.0 in three regions, and the Azure CLI got it too. An endpoint to the same kind of workspace succeeded after the workspace had a DICOM service. The endpoint now depends on FHIR and DICOM again, as on main. The DNS zone group still waits for the services. A unit test makes sure of the order, and the Health document says that each Health service must deploy FHIR, DICOM or both. --- CHANGELOG.md | 1 - .../workspace-services/health_services.md | 2 +- e2e_tests/unit_tests/test_health_bundle_contract.py | 10 ++++++---- .../health-services/terraform/main.tf | 12 ++++++------ 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 95950c685a..f8b5eeaa4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,6 @@ BUG FIXES: The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Create the Health private endpoint before FHIR and DICOM, and delete FHIR and DICOM before the endpoint. Terraform created the endpoint after the services and deleted it before them. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 661fc2ddd3..580f9ab628 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -23,7 +23,7 @@ To repair the service, delete its FHIR service in the Azure portal, and then del ## Private network access The service connects the healthcare workspace to the workspace network through one private endpoint. -Terraform creates the endpoint before FHIR and DICOM, and deletes FHIR and DICOM before the endpoint. +Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. So Terraform creates the endpoint after FHIR and DICOM, and each Health service must deploy FHIR, DICOM or both. Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index d5d0bfcb62..5a06574d03 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -40,16 +40,18 @@ def test_an_update_that_adds_fhir_creates_the_zone_group_again(self): group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') self.assertIn("replace_triggered_by = [terraform_data.fhir_deployed]", group) - def test_services_are_created_after_the_private_endpoint(self): + def test_private_endpoint_is_created_after_the_services(self): + # Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. source = (BUNDLE / "main.tf").read_text() endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') - self.assertNotIn("depends_on", endpoint) + depends_on = endpoint.split("depends_on", 1)[1] + self.assertIn("azurerm_healthcare_fhir_service.fhir", depends_on) + self.assertIn("azurerm_healthcare_dicom_service.dicom", depends_on) for header in ( 'resource "azurerm_healthcare_fhir_service" "fhir"', 'resource "azurerm_healthcare_dicom_service" "dicom"', ): - service = block(source, header) - self.assertIn("depends_on = [azurerm_private_endpoint.health_services_private_endpoint]", service) + self.assertNotIn("depends_on", block(source, header)) def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): source = (BUNDLE / "main.tf").read_text() diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 519e88408b..6f62bfccdc 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -25,9 +25,6 @@ resource "azurerm_healthcare_fhir_service" "fhir" { type = "SystemAssigned" } - # Terraform creates the services after the private endpoint and deletes them before it. - depends_on = [azurerm_private_endpoint.health_services_private_endpoint] - lifecycle { ignore_changes = [tags] } } @@ -45,9 +42,6 @@ resource "azurerm_healthcare_dicom_service" "dicom" { type = "SystemAssigned" } - # Terraform creates the services after the private endpoint and deletes them before it. - depends_on = [azurerm_private_endpoint.health_services_private_endpoint] - lifecycle { ignore_changes = [tags] } } @@ -65,6 +59,12 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { is_manual_connection = false } + # Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. + depends_on = [ + azurerm_healthcare_fhir_service.fhir, + azurerm_healthcare_dicom_service.dicom + ] + # The DNS zone group is a separate resource. The provider changes a zone group in this # resource only together with the whole endpoint. lifecycle { ignore_changes = [tags, private_dns_zone_group] } From bab198b82e465515015a1d26a5cd7c03dfeb3deb Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 12:56:44 +0000 Subject: [PATCH 23/28] Stop a Health service without FHIR and DICOM before a change (#5147) Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. Such an install left the failed endpoint outside the Terraform state, so each later update and delete of the service failed. The Health form had both services off by default. A precondition on the healthcare workspace now stops an install, an update or an upgrade without FHIR and DICOM at the plan, with the message "Deploy FHIR, DICOM or both". A delete still works. The Health form turns on FHIR by default. The bundle default stays false, so an upgrade does not add FHIR to a service that an API request created without the parameter. Two unit tests make sure of the precondition and the form default, and the Health document describes both. --- CHANGELOG.md | 1 + .../workspace-services/health_services.md | 3 +++ e2e_tests/unit_tests/test_health_bundle_contract.py | 12 ++++++++++++ .../health-services/template_schema.json | 2 +- .../health-services/terraform/main.tf | 10 +++++++++- 5 files changed, 26 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8b5eeaa4e..d54f156082 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,7 @@ BUG FIXES: * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Stop a Health install or update without FHIR and DICOM before Terraform changes a resource, and turn on FHIR by default in the Health form. Azure creates no private endpoint to a healthcare workspace without a FHIR or DICOM service. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 580f9ab628..bb3d957a38 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -24,6 +24,9 @@ To repair the service, delete its FHIR service in the Azure portal, and then del The service connects the healthcare workspace to the workspace network through one private endpoint. Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. So Terraform creates the endpoint after FHIR and DICOM, and each Health service must deploy FHIR, DICOM or both. +If an install or an update has neither FHIR nor DICOM, Terraform stops before it changes a resource, with the message "Deploy FHIR, DICOM or both". +This also applies to an upgrade of an earlier Health service without FHIR and DICOM. Turn on FHIR or DICOM in the same request, or delete the service. +The form of the TRE UI turns on FHIR by default. A request through the API without `deploy_fhir` keeps the bundle default `false`. Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index 5a06574d03..f7c18f0f92 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -1,5 +1,6 @@ """Check the private network settings of the Health bundle.""" +import json from pathlib import Path import re import unittest @@ -53,6 +54,17 @@ def test_private_endpoint_is_created_after_the_services(self): ): self.assertNotIn("depends_on", block(source, header)) + def test_a_service_without_fhir_and_dicom_stops_before_a_change(self): + source = (BUNDLE / "main.tf").read_text() + workspace = block(source, 'resource "azurerm_healthcare_workspace" "healthcare_workspace"') + precondition = workspace.split("precondition {", 1)[1] + self.assertIn("condition = var.deploy_fhir || var.deploy_dicom", precondition) + self.assertIn('error_message = "Deploy FHIR, DICOM or both.', precondition) + + def test_the_form_turns_on_fhir_by_default(self): + schema = json.loads((BUNDLE.parent / "template_schema.json").read_text()) + self.assertIs(schema["properties"]["deploy_fhir"]["default"], True) + def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): source = (BUNDLE / "main.tf").read_text() group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') diff --git a/templates/workspace_services/health-services/template_schema.json b/templates/workspace_services/health-services/template_schema.json index a48769dcfa..fe42629efb 100644 --- a/templates/workspace_services/health-services/template_schema.json +++ b/templates/workspace_services/health-services/template_schema.json @@ -32,7 +32,7 @@ "title": "Deploy FHIR", "description": "Deploy FHIR instance", "updateable": true, - "default": false + "default": true }, "workspace_owners_group_id": { "type": "string", diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 6f62bfccdc..316c34fcca 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -4,7 +4,15 @@ resource "azurerm_healthcare_workspace" "healthcare_workspace" { location = data.azurerm_resource_group.ws.location tags = local.workspace_service_tags - lifecycle { ignore_changes = [tags] } + lifecycle { + ignore_changes = [tags] + + # Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. + precondition { + condition = var.deploy_fhir || var.deploy_dicom + error_message = "Deploy FHIR, DICOM or both. Azure creates no private endpoint to a healthcare workspace without a FHIR or DICOM service." + } + } } resource "azurerm_healthcare_fhir_service" "fhir" { From cb48fac5c6d0ff71c63f4cd381cd13265b024d0e Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 22:55:08 +0000 Subject: [PATCH 24/28] Use the bundle defaults in the Health form and FHIR R4 without fhir_kind (#5147) With deploy_fhir on by default, the create form sent no fhir_kind, and the update form showed FHIR on for a service that an API request created without deploy_fhir. The form default is false again, the same as the bundle default. The update form does not send fhir_kind, so the bundle default of fhir_kind is now R4, the form default. The CHANGELOG names the upgrade that the precondition stops. --- CHANGELOG.md | 3 ++- e2e_tests/unit_tests/test_health_bundle_contract.py | 13 +++++++++++-- .../workspace_services/health-services/porter.yaml | 2 +- .../health-services/template_schema.json | 2 +- .../health-services/terraform/main.tf | 2 +- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d54f156082..78a147919e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,7 +54,8 @@ BUG FIXES: * Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Stop a Health install or update without FHIR and DICOM before Terraform changes a resource, and turn on FHIR by default in the Health form. Azure creates no private endpoint to a healthcare workspace without a FHIR or DICOM service. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Stop a Health install, update or upgrade without FHIR and DICOM before Terraform changes a resource. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. + If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. A request that turns on FHIR without `fhir_kind` gets FHIR R4. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index f7c18f0f92..f4e5f8cb95 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -61,9 +61,18 @@ def test_a_service_without_fhir_and_dicom_stops_before_a_change(self): self.assertIn("condition = var.deploy_fhir || var.deploy_dicom", precondition) self.assertIn('error_message = "Deploy FHIR, DICOM or both.', precondition) - def test_the_form_turns_on_fhir_by_default(self): + def test_the_form_and_the_bundle_use_the_same_defaults(self): + # The API stores a request without schema defaults, so the bundle default applies to an absent property. schema = json.loads((BUNDLE.parent / "template_schema.json").read_text()) - self.assertIs(schema["properties"]["deploy_fhir"]["default"], True) + porter = (BUNDLE.parent / "porter.yaml").read_text() + for name in ("deploy_fhir", "deploy_dicom"): + with self.subTest(name=name): + self.assertIs(schema["properties"][name]["default"], False) + self.assertRegex(porter, rf"- name: {name}\n type: boolean\n default: false\n") + # The update form removes fhir_kind, which cannot change, so a request that turns on FHIR can lack it. + fhir_kind = schema["allOf"][0]["then"]["properties"]["fhir_kind"]["default"] + self.assertEqual(fhir_kind, "R4") + self.assertIn(f'- name: fhir_kind\n type: string\n default: "{fhir_kind}"\n', porter) def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): source = (BUNDLE / "main.tf").read_text() diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 4dc4af76cd..a38e5b62c0 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -65,7 +65,7 @@ parameters: default: false - name: fhir_kind type: string - default: "" + default: "R4" outputs: - name: fhir_url diff --git a/templates/workspace_services/health-services/template_schema.json b/templates/workspace_services/health-services/template_schema.json index fe42629efb..a48769dcfa 100644 --- a/templates/workspace_services/health-services/template_schema.json +++ b/templates/workspace_services/health-services/template_schema.json @@ -32,7 +32,7 @@ "title": "Deploy FHIR", "description": "Deploy FHIR instance", "updateable": true, - "default": true + "default": false }, "workspace_owners_group_id": { "type": "string", diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 316c34fcca..c088690ccc 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -10,7 +10,7 @@ resource "azurerm_healthcare_workspace" "healthcare_workspace" { # Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. precondition { condition = var.deploy_fhir || var.deploy_dicom - error_message = "Deploy FHIR, DICOM or both. Azure creates no private endpoint to a healthcare workspace without a FHIR or DICOM service." + error_message = "Deploy FHIR, DICOM or both. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service." } } } From d5b8adb9401d9a79e135130517f9b29ec2c05848 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 31 Oct 2026 22:55:17 +0000 Subject: [PATCH 25/28] Give the Health repairs for a failed endpoint and a failed FHIR change (#5147) An earlier install without FHIR and DICOM can leave a failed private endpoint outside the Terraform state. The document gives the repair, the steps after a stopped upgrade in the TRE UI, the disable that the check also stops, and the repair if a FHIR change fails again. --- docs/tre-developers/end-to-end-tests.md | 2 +- .../workspace-services/health_services.md | 21 +++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 25b58f7c46..dc870bff45 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -223,7 +223,7 @@ An update adds DICOM. The case makes sure that the update kept the private endpo It then reads the Patient again and makes sure that the DICOM host is private and needs a token. At the end, the case sends FHIR and DICOM requests without a token from the test runner, outside the workspace network. Azure must reject each request with HTTP 403. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. -The case then reads the Health workspace and DICOM again, and makes sure that each has public network access disabled. +The case then reads the healthcare workspace and DICOM again, and makes sure that each has public network access disabled. The case does not read this value on FHIR. Azure keeps `publicNetworkAccess` `Enabled` on FHIR, also after a request sets `Disabled`. The Health job uses the SQL time limits with its own `HEALTH_VALIDATION_DEADLINE`. diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index bb3d957a38..46c23db739 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -20,19 +20,28 @@ If Azure reports the FHIR service as `Accepted` after the create, the azurerm pr After this failure, each update and each delete of the Health service fails. To repair the service, delete its FHIR service in the Azure portal, and then delete the Health service again. +Before version 0.3.9, an install without FHIR and DICOM can fail at the private endpoint. +Azure keeps the failed endpoint, but Terraform does not save it. After this failure, each update and each delete of the Health service fails. +To repair the service, delete the private endpoint `pe-hs` of the service in the Azure portal. Its provisioning state is `Failed`. +Then delete the Health service, or upgrade it and turn on FHIR or DICOM with an update. + ## Private network access The service connects the healthcare workspace to the workspace network through one private endpoint. Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. So Terraform creates the endpoint after FHIR and DICOM, and each Health service must deploy FHIR, DICOM or both. -If an install or an update has neither FHIR nor DICOM, Terraform stops before it changes a resource, with the message "Deploy FHIR, DICOM or both". -This also applies to an upgrade of an earlier Health service without FHIR and DICOM. Turn on FHIR or DICOM in the same request, or delete the service. -The form of the TRE UI turns on FHIR by default. A request through the API without `deploy_fhir` keeps the bundle default `false`. +If an install, an update or an upgrade has neither FHIR nor DICOM, Terraform stops before it changes a resource, with the message "Deploy FHIR, DICOM or both". +This applies to an upgrade of an earlier Health service without FHIR and DICOM. The upgrade dialog of the TRE UI cannot change a property. +If the upgrade stops for this reason, update the service and turn on FHIR or DICOM, or delete the service. +A disable of such a service also stops at this check, but the delete then works. +If a request turns on FHIR without `fhir_kind`, the service gets FHIR R4. The update form of the TRE UI does not send `fhir_kind`. Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. -The DNS zone group always has the zone for the workspace and FHIR. It has the DICOM zone only when the service has DICOM. +The DNS zone group always has the zone for the healthcare workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. -The FHIR record is in the same zone as the workspace record. So if an update adds or removes FHIR, Terraform deletes the group before the FHIR change and creates it again after the change. Azure then writes the records again. +The FHIR record is in the same zone as the healthcare workspace record. So if an update adds or removes FHIR, Terraform deletes the group before the FHIR change and creates it again after the change. Azure then writes the records again. Private access to the healthcare workspace and DICOM stops until the new group exists. For an update that adds FHIR, this time includes the FHIR create, which took 9 to 12 minutes in tests. If the FHIR change fails, the new group does not exist, and private access stays stopped. To repair the service, run the update again. +If the update fails again, set `deploy_fhir` back to its earlier value. Terraform then creates the group for the other services. +If Azure has a FHIR service that Terraform did not save, delete that FHIR service in the Azure portal first. Azure links a new service to the endpoint some minutes after Terraform creates the service. During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. @@ -41,7 +50,7 @@ If you added DICOM to an existing service before this version, the DICOM host ca An upgrade to version 0.3.9 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. Terraform deletes the old group after the changes to FHIR and DICOM, directly before it creates the new group. Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. -Do not delete the private endpoint to repair the service. +Do not delete a working private endpoint to repair the service. The DICOM service has public network access turned off. Before version 0.3.9, Terraform created the DICOM service with public network access on. From 8f36a52b9f4a2b51415d8579a06a368edc751679 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 1 Nov 2026 16:46:58 +0000 Subject: [PATCH 26/28] Give the FHIR data roles after the Health private endpoint (#5147) Both FHIR role assignments now depend on the DNS zone group, the same as the DICOM role assignments. So the workspace groups get FHIR data access only after the private endpoint exists, and an uninstall deletes the roles before the endpoint. --- e2e_tests/unit_tests/test_health_bundle_contract.py | 7 +++++++ .../workspace_services/health-services/terraform/roles.tf | 8 ++++++++ 2 files changed, 15 insertions(+) diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py index f4e5f8cb95..dbe2896309 100644 --- a/e2e_tests/unit_tests/test_health_bundle_contract.py +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -110,6 +110,13 @@ def test_role_assignments_use_the_role_names(self): self.assertNotIn("role_definition_id", roles) self.assertEqual(roles.count("role_definition_name"), 4) + def test_fhir_roles_wait_for_the_private_endpoint(self): + # The workspace groups get FHIR data access only through the private endpoint. + roles = (BUNDLE / "roles.tf").read_text() + for name in ("researchers_fhir_contributor", "owners_fhir_contributor"): + role = block(roles, f'resource "azurerm_role_assignment" "{name}"') + self.assertIn("depends_on = [azapi_resource.health_services_dns_zone_group]", role) + def test_dicom_roles_wait_for_the_endpoint_link(self): # Azure accepts the zone group write only after it links a new DICOM service to the endpoint. roles = (BUNDLE / "roles.tf").read_text() diff --git a/templates/workspace_services/health-services/terraform/roles.tf b/templates/workspace_services/health-services/terraform/roles.tf index 7c020ca782..05ea331ce1 100644 --- a/templates/workspace_services/health-services/terraform/roles.tf +++ b/templates/workspace_services/health-services/terraform/roles.tf @@ -7,6 +7,10 @@ resource "azurerm_role_assignment" "researchers_fhir_contributor" { scope = azurerm_healthcare_fhir_service.fhir[0].id role_definition_name = "FHIR Data Contributor" principal_id = var.workspace_researchers_group_id + + # The groups get FHIR data access only after the private endpoint and its DNS records exist. + # On uninstall, Terraform deletes the role before the zone group and the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } resource "azurerm_role_assignment" "researchers_dicom_data_owner" { @@ -25,6 +29,10 @@ resource "azurerm_role_assignment" "owners_fhir_contributor" { scope = azurerm_healthcare_fhir_service.fhir[0].id role_definition_name = "FHIR Data Contributor" principal_id = var.workspace_owners_group_id + + # The groups get FHIR data access only after the private endpoint and its DNS records exist. + # On uninstall, Terraform deletes the role before the zone group and the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } resource "azurerm_role_assignment" "owners_dicom_data_owner" { From df9797d6329e5c96889d48b0e5e759977a44e8d5 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 1 Nov 2026 17:25:17 +0000 Subject: [PATCH 27/28] Set Health 1.0.0 for the stop of a service without FHIR and DICOM (#5147) The new precondition stops the upgrade or disable of an earlier Health service without FHIR and DICOM until an operator changes or deletes it. The template rules ask for a major version and a BREAKING CHANGES entry for a change that needs manual action. The Health document now gives the measured time of an update that adds FHIR. --- CHANGELOG.md | 16 +++++++++------- .../workspace-services/health_services.md | 16 ++++++++-------- .../health-services/porter.yaml | 2 +- .../health-services/terraform/main.tf | 2 +- 4 files changed, 19 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 78a147919e..95b5c79778 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,8 @@ ## (Unreleased) **BREAKING CHANGES** +* `tre-workspace-service-health` 1.0.0 stops an install, update or upgrade of a Health service without FHIR and DICOM before Terraform changes a resource. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. + If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Existing resources with legacy Porter parameter overrides must complete a resource upgrade before custom actions or uninstall. The current template version can be reused. See [upgrading resources](docs/tre-admins/upgrading-resources.md#upgrade-existing-resources-for-document-based-porter-parameters). (`resource_processor` 0.13.10) ([#4904](https://github.com/microsoft/AzureTRE/pull/4904)) * Set `enable_legacy_airlock` explicitly to `true` in your `config.yaml`. It currently defaults to `true` but will default to `false` in a future release; Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048)) @@ -50,12 +52,12 @@ BUG FIXES: * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group. So a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. - The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Stop a Health install, update or upgrade without FHIR and DICOM before Terraform changes a resource. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. - If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. A request that turns on FHIR without `fhir_kind` gets FHIR R4. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Give FHIR R4 to a Health request that turns on FHIR without `fhir_kind`. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Give the workspace groups their FHIR data role after the Health private endpoint exists. Terraform gave the role before the endpoint, and an uninstall did not delete the role before the endpoint. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) @@ -112,7 +114,7 @@ BUG FIXES: * Fix Guacamole login failures in large workspaces: raise the user-resources API timeout from 5s to 30s, batch template version lookups into one Cosmos query, and query VM power states concurrently. (`tre-service-guacamole` 0.14.6, `API` 0.27.31) ([#5099](https://github.com/microsoft/AzureTRE/issues/5099)) * Fix API resource update retries so a successful retry returns the patched resource to the original caller. (`API` 0.27.32) ([#5025](https://github.com/microsoft/AzureTRE/pull/5025)) * Fix the health check falsely returning OK when Cosmos is down or inaccessible. ([#4926](https://github.com/microsoft/AzureTRE/issues/4926)) -* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 0.3.9) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) ## (0.29.1) **BREAKING CHANGES** diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index 46c23db739..bb577076c8 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -13,14 +13,14 @@ In addition, when creating this workspace service you can choose to deploy [FHIR ![Healthcare Service](images/hs_details.png) -## Install failures before version 0.3.9 +## Install failures before version 1.0.0 -Before version 0.3.9, an install can fail after Azure creates the FHIR service. +Before version 1.0.0, an install can fail after Azure creates the FHIR service. If Azure reports the FHIR service as `Accepted` after the create, the azurerm provider stops its wait, and Terraform does not save the FHIR service. After this failure, each update and each delete of the Health service fails. To repair the service, delete its FHIR service in the Azure portal, and then delete the Health service again. -Before version 0.3.9, an install without FHIR and DICOM can fail at the private endpoint. +Before version 1.0.0, an install without FHIR and DICOM can fail at the private endpoint. Azure keeps the failed endpoint, but Terraform does not save it. After this failure, each update and each delete of the Health service fails. To repair the service, delete the private endpoint `pe-hs` of the service in the Azure portal. Its provisioning state is `Failed`. Then delete the Health service, or upgrade it and turn on FHIR or DICOM with an update. @@ -38,23 +38,23 @@ Azure writes the private DNS records of a zone when the DNS zone group of this e The DNS zone group always has the zone for the healthcare workspace and FHIR. It has the DICOM zone only when the service has DICOM. If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. The FHIR record is in the same zone as the healthcare workspace record. So if an update adds or removes FHIR, Terraform deletes the group before the FHIR change and creates it again after the change. Azure then writes the records again. -Private access to the healthcare workspace and DICOM stops until the new group exists. For an update that adds FHIR, this time includes the FHIR create, which took 9 to 12 minutes in tests. +Private access to the healthcare workspace and DICOM stops until the new group exists. For an update that adds FHIR, this time includes the FHIR create. In tests, the whole update that adds FHIR took 12 to 18 minutes. If the FHIR change fails, the new group does not exist, and private access stays stopped. To repair the service, run the update again. If the update fails again, set `deploy_fhir` back to its earlier value. Terraform then creates the group for the other services. If Azure has a FHIR service that Terraform did not save, delete that FHIR service in the Azure portal first. Azure links a new service to the endpoint some minutes after Terraform creates the service. During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. -Before version 0.3.9, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. +Before version 1.0.0, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. If you added DICOM to an existing service before this version, the DICOM host can resolve to a public address. -An upgrade to version 0.3.9 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. +An upgrade to version 1.0.0 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. Terraform deletes the old group after the changes to FHIR and DICOM, directly before it creates the new group. Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. Do not delete a working private endpoint to repair the service. The DICOM service has public network access turned off. -Before version 0.3.9, Terraform created the DICOM service with public network access on. -An upgrade to version 0.3.9 turns it off. +Before version 1.0.0, Terraform created the DICOM service with public network access on. +An upgrade to version 1.0.0 turns it off. The private endpoint on the healthcare workspace blocks public traffic to FHIR and DICOM. A request from outside the workspace network gets HTTP 403. diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index a38e5b62c0..12f393fa60 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.9 +version: 1.0.0 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index c088690ccc..2b281d4464 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -78,7 +78,7 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { lifecycle { ignore_changes = [tags, private_dns_zone_group] } } -# Before version 0.3.9, the endpoint resource created the DNS zone group. This removes +# Before version 1.0.0, the endpoint resource created the DNS zone group. This removes # that group once, so that the resource below can create a group with the same name. resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" From ff18c9dff7dcee18116b1abfaedf692712a4139e Mon Sep 17 00:00:00 2001 From: Jay W Date: Mon, 2 Nov 2026 11:08:18 +0000 Subject: [PATCH 28/28] Link the Health CHANGELOG entries to issue #5295 The breaking change and the 7 bug fix entries now link the defect issue. The entry for the private Health test keeps its link to #5147. --- CHANGELOG.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 95b5c79778..4090d20c07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## (Unreleased) **BREAKING CHANGES** * `tre-workspace-service-health` 1.0.0 stops an install, update or upgrade of a Health service without FHIR and DICOM before Terraform changes a resource. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. - If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) * Existing resources with legacy Porter parameter overrides must complete a resource upgrade before custom actions or uninstall. The current template version can be reused. See [upgrading resources](docs/tre-admins/upgrading-resources.md#upgrade-existing-resources-for-document-based-porter-parameters). (`resource_processor` 0.13.10) ([#4904](https://github.com/microsoft/AzureTRE/pull/4904)) * Set `enable_legacy_airlock` explicitly to `true` in your `config.yaml`. It currently defaults to `true` but will default to `false` in a future release; Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048)) @@ -52,12 +52,12 @@ BUG FIXES: * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group. So a DICOM service that an update added resolved to a public address. The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. - The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Give FHIR R4 to a Health request that turns on FHIR without `fhir_kind`. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) -* Give the workspace groups their FHIR data role after the Health private endpoint exists. Terraform gave the role before the endpoint, and an uninstall did not delete the role before the endpoint. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) + The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give FHIR R4 to a Health request that turns on FHIR without `fhir_kind`. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give the workspace groups their FHIR data role after the Health private endpoint exists. Terraform gave the role before the endpoint, and an uninstall did not delete the role before the endpoint. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) @@ -114,7 +114,7 @@ BUG FIXES: * Fix Guacamole login failures in large workspaces: raise the user-resources API timeout from 5s to 30s, batch template version lookups into one Cosmos query, and query VM power states concurrently. (`tre-service-guacamole` 0.14.6, `API` 0.27.31) ([#5099](https://github.com/microsoft/AzureTRE/issues/5099)) * Fix API resource update retries so a successful retry returns the patched resource to the original caller. (`API` 0.27.32) ([#5025](https://github.com/microsoft/AzureTRE/pull/5025)) * Fix the health check falsely returning OK when Cosmos is down or inaccessible. ([#4926](https://github.com/microsoft/AzureTRE/issues/4926)) -* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 1.0.0) ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) +* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) ## (0.29.1) **BREAKING CHANGES**