diff --git a/.github/workflows/deploy_tre_reusable.yml b/.github/workflows/deploy_tre_reusable.yml index 65ebe15392..5f00fe5509 100644 --- a/.github/workflows/deploy_tre_reusable.yml +++ b/.github/workflows/deploy_tre_reusable.yml @@ -1083,6 +1083,9 @@ jobs: - name: Reserve Gitea cleanup and reporting time if: inputs.e2eBundle == 'tre-workspace-service-gitea' run: echo "GITEA_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" + - name: Reserve Health cleanup and reporting time + if: inputs.e2eBundle == 'tre-workspace-service-health' + run: echo "HEALTH_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" - name: Checkout uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 @@ -1113,6 +1116,7 @@ jobs: SQL_VALIDATION_DEADLINE=${{ env.SQL_VALIDATION_DEADLINE }} MYSQL_VALIDATION_DEADLINE=${{ env.MYSQL_VALIDATION_DEADLINE }} GITEA_VALIDATION_DEADLINE=${{ env.GITEA_VALIDATION_DEADLINE }} + HEALTH_VALIDATION_DEADLINE=${{ env.HEALTH_VALIDATION_DEADLINE }} TEST_BUNDLE_REQUEST_FILE=e2e_tests/bundle-validation-request.json make test-e2e-bundle DEVCONTAINER_TAG: ${{ inputs.DEVCONTAINER_TAG }} CI_CACHE_ACR_NAME: ${{ secrets.CI_CACHE_ACR_NAME}} diff --git a/CHANGELOG.md b/CHANGELOG.md index decc945754..4090d20c07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,8 @@ ## (Unreleased) **BREAKING CHANGES** +* `tre-workspace-service-health` 1.0.0 stops an install, update or upgrade of a Health service without FHIR and DICOM before Terraform changes a resource. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. + If the upgrade of an earlier service without FHIR and DICOM stops, turn on FHIR or DICOM with an update, or delete the service. ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) * Existing resources with legacy Porter parameter overrides must complete a resource upgrade before custom actions or uninstall. The current template version can be reused. See [upgrading resources](docs/tre-admins/upgrading-resources.md#upgrade-existing-resources-for-document-based-porter-parameters). (`resource_processor` 0.13.10) ([#4904](https://github.com/microsoft/AzureTRE/pull/4904)) * Set `enable_legacy_airlock` explicitly to `true` in your `config.yaml`. It currently defaults to `true` but will default to `false` in a future release; Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048)) @@ -9,6 +11,7 @@ Setting to `false` will delete existing airlock storage accounts and must only b ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) +* Add a private Health lifecycle check. A workspace VM reads the FHIR metadata through private DNS, and gets HTTP 401 without a token and HTTP 403 without a role. Then the VM writes and reads a Patient across an update that adds DICOM. The generic workspace service install case no longer installs Health, because its CI workspace has no Entra ID groups. ([#5147](https://github.com/microsoft/AzureTRE/issues/5147)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) * Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. @@ -47,6 +50,14 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) +* Add the DICOM zone to the DNS zone group of the Health private endpoint when an update adds DICOM. Azure wrote private DNS records only for the services that existed when it created the zone group. So a DICOM service that an update added resolved to a public address. + The zone group is now a separate resource, so the endpoint stays, and Terraform tries again while Azure links the new service to the endpoint. An upgrade creates the zone group again and writes a record that an earlier update did not get. + The FHIR record is in the zone of the workspace record, so an update that adds or removes FHIR also creates the zone group again. Private access to the Health services stops until the new group exists. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Turn off public network access on the Health DICOM service. The azurerm provider created DICOM with public network access on, and an upgrade turned it on again after Azure turned it off. An upgrade changes the setting in place. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Keep the Health data roles of the workspace groups when the service changes. The role assignments used a role definition ID without the subscription, but Azure returns the ID with the subscription. So each update deleted the role assignments and created them again. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give the workspace groups their DICOM data role after Azure links the DICOM service to the Health private endpoint. Terraform gave the role immediately after the DICOM create. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give FHIR R4 to a Health request that turns on FHIR without `fhir_kind`. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) +* Give the workspace groups their FHIR data role after the Health private endpoint exists. Terraform gave the role before the endpoint, and an uninstall did not delete the role before the endpoint. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) @@ -103,6 +114,7 @@ BUG FIXES: * Fix Guacamole login failures in large workspaces: raise the user-resources API timeout from 5s to 30s, batch template version lookups into one Cosmos query, and query VM power states concurrently. (`tre-service-guacamole` 0.14.6, `API` 0.27.31) ([#5099](https://github.com/microsoft/AzureTRE/issues/5099)) * Fix API resource update retries so a successful retry returns the patched resource to the original caller. (`API` 0.27.32) ([#5025](https://github.com/microsoft/AzureTRE/pull/5025)) * Fix the health check falsely returning OK when Cosmos is down or inaccessible. ([#4926](https://github.com/microsoft/AzureTRE/issues/4926)) +* Keep a new Health FHIR service in the Terraform state when Azure reports it as `Accepted` after the create. Before version 4.75.0, the azurerm provider waited for `Succeeded` after the create and saved the service only after this wait. So the install failed, and each later update and delete of the service failed. The Health bundle now uses azurerm 5.11.0. (`tre-workspace-service-health` 1.0.0) ([#5295](https://github.com/microsoft/AzureTRE/issues/5295)) ## (0.29.1) **BREAKING CHANGES** diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 306f4ca158..dc870bff45 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -205,6 +205,28 @@ The query returns counts only. The CI identity needs read access to the Log Anal At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. +### Health selection + +Select `tre-workspace-service-health` to run `test_health.py::test_private_fhir_access_and_dicom_update`. +The case creates its own workspace, Guacamole service, Windows VM and Health service with FHIR R4. +The Health bundle needs the workspace owners group, and only an Automatic workspace gets Entra ID groups. +So the case always creates an Automatic workspace. It does not use `TEST_WORKSPACE_ID`, `TEST_WORKSPACE_SERVICE_ID` or `TEST_WORKSPACE_APP_ID`. +From the VM, the case makes sure that the FHIR host resolves to a private address and that the FHIR metadata is available. +If the FHIR name lookup fails, the case runs that probe again after 60 seconds, 3 attempts in total. Before this lookup, the probe changes no data. +It also makes sure that FHIR rejects a data request without a token (HTTP 401) and from an identity without a role (HTTP 403). +The case then gives the VM identity a temporary FHIR Data Contributor role on the FHIR service and writes a Patient. +The role can take some minutes to apply, so the probe waits up to 10 minutes for a first HTTP 200. +Until the end of this wait, the probe sends the create again after HTTP 403 or 429. +It sends the search again after HTTP 403, 429, 500, 502, 503 or 504. +If a probe fails, the test error gives the stage and the HTTP status of the failed request. +An update adds DICOM. The case makes sure that the update kept the private endpoint. +It then reads the Patient again and makes sure that the DICOM host is private and needs a token. +At the end, the case sends FHIR and DICOM requests without a token from the test runner, outside the workspace network. +Azure must reject each request with HTTP 403. The case tries again for up to 5 minutes, because Azure can take this time to block public traffic. +The case then reads the healthcare workspace and DICOM again, and makes sure that each has public network access disabled. +The case does not read this value on FHIR. Azure keeps `publicNetworkAccess` `Enabled` on FHIR, also after a request sets `Disabled`. +The Health job uses the SQL time limits with its own `HEALTH_VALIDATION_DEADLINE`. + ### Run the branch workflow before merge 1. Push the reviewed changes to a branch in `microsoft/AzureTRE`. diff --git a/docs/tre-developers/github-pr-bot-commands.md b/docs/tre-developers/github-pr-bot-commands.md index da6f973b54..0a67f6c912 100644 --- a/docs/tre-developers/github-pr-bot-commands.md +++ b/docs/tre-developers/github-pr-bot-commands.md @@ -22,7 +22,7 @@ The table describes the selected assertions. A passing command does not prove ev | `/test-airlock` | Import requests, access checks, the import review VM flow and export request storage routing | No export review VM or migration proof. | | `/test-backups` | Base workspace deployment with backups enabled and disabled | No backup restore or retention proof. | -The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `gitea_validation` markers have no dedicated slash command. +The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation`, `gitea_validation` and `health_validation` markers have no dedicated slash command. Use the branch workflow or the local test runner for these selections. The `airlock_validation` marker selects two Airlock cases. It does not select the full `airlock` suite. See [bundle selection](end-to-end-tests.md#validate-one-bundle) for exact cases and declared coverage gaps. diff --git a/docs/tre-templates/workspace-services/health_services.md b/docs/tre-templates/workspace-services/health_services.md index e93c63999b..bb577076c8 100644 --- a/docs/tre-templates/workspace-services/health_services.md +++ b/docs/tre-templates/workspace-services/health_services.md @@ -13,6 +13,54 @@ In addition, when creating this workspace service you can choose to deploy [FHIR ![Healthcare Service](images/hs_details.png) +## Install failures before version 1.0.0 + +Before version 1.0.0, an install can fail after Azure creates the FHIR service. +If Azure reports the FHIR service as `Accepted` after the create, the azurerm provider stops its wait, and Terraform does not save the FHIR service. +After this failure, each update and each delete of the Health service fails. +To repair the service, delete its FHIR service in the Azure portal, and then delete the Health service again. + +Before version 1.0.0, an install without FHIR and DICOM can fail at the private endpoint. +Azure keeps the failed endpoint, but Terraform does not save it. After this failure, each update and each delete of the Health service fails. +To repair the service, delete the private endpoint `pe-hs` of the service in the Azure portal. Its provisioning state is `Failed`. +Then delete the Health service, or upgrade it and turn on FHIR or DICOM with an update. + +## Private network access + +The service connects the healthcare workspace to the workspace network through one private endpoint. +Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. So Terraform creates the endpoint after FHIR and DICOM, and each Health service must deploy FHIR, DICOM or both. +If an install, an update or an upgrade has neither FHIR nor DICOM, Terraform stops before it changes a resource, with the message "Deploy FHIR, DICOM or both". +This applies to an upgrade of an earlier Health service without FHIR and DICOM. The upgrade dialog of the TRE UI cannot change a property. +If the upgrade stops for this reason, update the service and turn on FHIR or DICOM, or delete the service. +A disable of such a service also stops at this check, but the delete then works. +If a request turns on FHIR without `fhir_kind`, the service gets FHIR R4. The update form of the TRE UI does not send `fhir_kind`. +Azure writes the private DNS records of a zone when the DNS zone group of this endpoint gets that zone. +The DNS zone group always has the zone for the healthcare workspace and FHIR. It has the DICOM zone only when the service has DICOM. +If an update adds DICOM, Terraform adds the DICOM zone to the group, and Azure writes the DICOM record. The endpoint stays. +The FHIR record is in the same zone as the healthcare workspace record. So if an update adds or removes FHIR, Terraform deletes the group before the FHIR change and creates it again after the change. Azure then writes the records again. +Private access to the healthcare workspace and DICOM stops until the new group exists. For an update that adds FHIR, this time includes the FHIR create. In tests, the whole update that adds FHIR took 12 to 18 minutes. +If the FHIR change fails, the new group does not exist, and private access stays stopped. To repair the service, run the update again. +If the update fails again, set `deploy_fhir` back to its earlier value. Terraform then creates the group for the other services. +If Azure has a FHIR service that Terraform did not save, delete that FHIR service in the Azure portal first. +Azure links a new service to the endpoint some minutes after Terraform creates the service. +During this time, Azure refuses a change to the DNS zone group with HTTP 409, and Terraform tries again. + +Before version 1.0.0, the endpoint resource created the DNS zone group, and the group did not change when an update added a service. +If you added DICOM to an existing service before this version, the DICOM host can resolve to a public address. +An upgrade to version 1.0.0 deletes the old DNS zone group and creates a new one, so Azure writes the missing record. +Terraform deletes the old group after the changes to FHIR and DICOM, directly before it creates the new group. +Until the upgrade creates the new group, the private DNS records do not exist, and private access to the services stops. +Do not delete a working private endpoint to repair the service. + +The DICOM service has public network access turned off. +Before version 1.0.0, Terraform created the DICOM service with public network access on. +An upgrade to version 1.0.0 turns it off. + +The private endpoint on the healthcare workspace blocks public traffic to FHIR and DICOM. +A request from outside the workspace network gets HTTP 403. +The FHIR service reports `publicNetworkAccess` `Enabled`, but Azure does not use this value when the workspace has an approved private endpoint. +Azure also keeps this value when a request sets it to `Disabled`. + ## Authentication Learn more about authentication and application roles in [this doc](https://learn.microsoft.com/en-us/azure/healthcare-apis/authentication-authorization). Make sure to assign your users/apps with the required role and follow the guidelines to retrieve a token. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index 49194451cd..237d8b1157 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -372,19 +372,26 @@ "manifest": "templates/workspace_services/health-services/porter.yaml", "family": "workspace_service", "prerequisites": [ - "tre-workspace-base" + "tre-workspace-base", + "tre-service-guacamole", + "tre-service-guacamole-windowsvm" ], "tests": [ - "test_workspace_services.py::test_install_workspace_service[tre-workspace-service-health]" + "test_health.py::test_private_fhir_access_and_dicom_update" ], "related_tests": [], "coverage": [ - "install", + "install with FHIR R4", + "private FHIR metadata", + "anonymous and unassigned FHIR requests denied", + "private FHIR write/read with a temporary data role", + "update that adds DICOM with retained FHIR data", + "public network access disabled", "uninstall" ], "limitations": [ - "Functional, upgrade and custom-action coverage must be assessed separately.", - "Default inputs do not enable FHIR or DICOM. Endpoint validation requires an explicit configuration." + "AzureCloud only. Requires permission to run commands on a test-created workspace VM and create a temporary service-scoped role assignment.", + "The case does not use DICOM data, FHIR STU3 or researcher identities. The update uses the current bundle version. Previous-version migration remains separate (#5128)." ], "unavailable_reason": "" }, diff --git a/e2e_tests/pytest.ini b/e2e_tests/pytest.ini index ea391b6c83..f037ef6fc2 100644 --- a/e2e_tests/pytest.ini +++ b/e2e_tests/pytest.ini @@ -19,6 +19,7 @@ markers = sql_validation: Private SQL query and configuration upgrade lifecycle mysql_validation: opt-in private MySQL query and SKU change lifecycle from a Nexus-backed Linux VM gitea_validation: opt-in private Gitea repository write/read across an update and an app restart + health_validation: opt-in private FHIR access control, write/read and DICOM update lifecycle asyncio_mode = auto asyncio_default_fixture_loop_scope = session diff --git a/e2e_tests/resources/health_probe.ps1 b/e2e_tests/resources/health_probe.ps1 new file mode 100644 index 0000000000..828e0d9924 --- /dev/null +++ b/e2e_tests/resources/health_probe.ps1 @@ -0,0 +1,106 @@ +param( + [Parameter(Mandatory)][ValidatePattern('^https://[a-z0-9-]+\.fhir\.azurehealthcareapis\.com$')][string]$FhirUrl, + [ValidatePattern('^(https://[a-z0-9-]+\.dicom\.azurehealthcareapis\.com)?$')][string]$DicomUrl = '', + [Parameter(Mandatory)][ValidatePattern('^[a-f0-9]{32}$')][string]$ProbeId, + [Parameter(Mandatory)][ValidateSet('access', 'write', 'read')][string]$Phase +) +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 +$stage = 'private-dns' + +function Assert-PrivateDns([string]$Url) { + $addresses = [System.Net.Dns]::GetHostAddresses(([uri]$Url).Host) + if ($addresses.Count -eq 0) { throw 'No address' } + foreach ($address in $addresses) { + $bytes = $address.GetAddressBytes() + $private = $bytes.Length -eq 4 -and ( + $bytes[0] -eq 10 -or + ($bytes[0] -eq 172 -and $bytes[1] -ge 16 -and $bytes[1] -le 31) -or + ($bytes[0] -eq 192 -and $bytes[1] -eq 168) + ) + if (-not $private) { throw 'DNS is not private' } + } +} + +function Get-Status([string]$Uri, [hashtable]$Headers = @{}) { + try { + return [int](Invoke-WebRequest -UseBasicParsing -Uri $Uri -Headers $Headers -TimeoutSec 30).StatusCode + } catch { + # Windows PowerShell and PowerShell 7 use different exception types, but both have a Response. + $response = $_.Exception.Response + if ($null -eq $response) { throw } + return [int]$response.StatusCode + } +} + +function Get-FhirHeaders { + $resource = [uri]::EscapeDataString($FhirUrl) + $identity = Invoke-RestMethod -Headers @{Metadata = 'true'} -Uri "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=$resource" -TimeoutSec 15 + return @{Authorization = "Bearer $($identity.access_token)"} +} + +function Invoke-Fhir([hashtable]$Request, [int[]]$RetryStatus) { + # A 200 in the role wait does not prove that each later request gets the new role. + while ($true) { + try { + return Invoke-RestMethod @Request -Headers (Get-FhirHeaders) -TimeoutSec 30 + } catch { + $response = $_.Exception.Response + if ($null -eq $response -or [int]$response.StatusCode -notin $RetryStatus -or [DateTime]::UtcNow -ge $deadline) { throw } + Start-Sleep -Seconds 20 + } + } +} + +function Get-ProbeCount { + # A search changes no data, so throttling and server errors are also tried again. + $search = Invoke-Fhir -Request @{Uri = "$FhirUrl/Patient?identifier=urn:tre:e2e|$ProbeId"} -RetryStatus 403, 429, 500, 502, 503, 504 + if ($search.resourceType -ne 'Bundle') { throw 'Unexpected search result' } + return @($search.entry).Where({ $null -ne $_ }).Count +} + +try { + Assert-PrivateDns $FhirUrl + $result = [ordered]@{phase = $Phase; probe = $ProbeId; private_dns = $true} + if ($Phase -eq 'access') { + $stage = 'fhir-metadata' + $metadata = Invoke-RestMethod -Uri "$FhirUrl/metadata" -TimeoutSec 30 + if ($metadata.resourceType -ne 'CapabilityStatement') { throw 'Unexpected metadata' } + $result.fhir_version = [string]$metadata.fhirVersion + $stage = 'fhir-anonymous' + $result.anonymous = Get-Status "$FhirUrl/Patient?_count=1" + $stage = 'fhir-unassigned' + $result.unassigned = Get-Status "$FhirUrl/Patient?_count=1" (Get-FhirHeaders) + } else { + $stage = 'fhir-role' + # A new role assignment can take some minutes to apply. + $deadline = [DateTime]::UtcNow.AddMinutes(10) + while ((Get-Status "$FhirUrl/Patient?_count=1" (Get-FhirHeaders)) -ne 200) { + if ([DateTime]::UtcNow -ge $deadline) { throw 'Role did not apply' } + Start-Sleep -Seconds 20 + } + if ($Phase -eq 'write') { + $stage = 'fhir-write' + $patient = @{resourceType = 'Patient'; identifier = @(@{system = 'urn:tre:e2e'; value = $ProbeId})} | ConvertTo-Json -Depth 4 + # FHIR writes nothing for a refused (403) or throttled (429) create, so only these are tried again. + $create = @{Method = 'Post'; ContentType = 'application/fhir+json'; Uri = "$FhirUrl/Patient"; Body = $patient} + $created = Invoke-Fhir -Request $create -RetryStatus 403, 429 + if ($created.resourceType -ne 'Patient') { throw 'Unexpected create result' } + } + $stage = 'fhir-read' + $result.count = Get-ProbeCount + if ($DicomUrl) { + $stage = 'dicom-access' + Assert-PrivateDns $DicomUrl + $result.dicom_anonymous = Get-Status "$DicomUrl/v2/studies" + } + } + Write-Output ('HEALTH_PROBE_RESULT=' + ($result | ConvertTo-Json -Compress)) +} catch { + # Do not emit exception details or tokens. The HTTP status of a failed request is safe to emit. + $response = $_.Exception.Response + Write-Output "HEALTH_PROBE_FAILED=$stage" + if ($null -ne $response) { Write-Output "HEALTH_PROBE_STATUS=$([int]$response.StatusCode)" } + exit 1 +} diff --git a/e2e_tests/resources/health_probe.py b/e2e_tests/resources/health_probe.py new file mode 100644 index 0000000000..3a23b8e01c --- /dev/null +++ b/e2e_tests/resources/health_probe.py @@ -0,0 +1,135 @@ +"""Run a private FHIR probe from a test-created workspace Windows VM.""" + +import asyncio +from contextlib import asynccontextmanager +import json +import logging +from pathlib import Path +from uuid import uuid4 + +from e2e_tests.resources.sql_probe import COMPUTE_API, ROLE_API, delete_after + +SCRIPT = Path(__file__).with_name("health_probe.ps1") +RESULT = "HEALTH_PROBE_RESULT=" +FAILED = "HEALTH_PROBE_FAILED=" +STATUS = "HEALTH_PROBE_STATUS=" +FHIR_DATA_CONTRIBUTOR = "5a1fc7df-4bf1-4951-a576-89034ee01acd" +DNS_ATTEMPTS = 3 +DNS_RETRY_SECONDS = 60 +PUBLIC_BLOCKED = 403 +# Azure can take up to 5 minutes to block public traffic after a change. +PUBLIC_WAIT_SECONDS = 5 * 60 +PUBLIC_RETRY_SECONDS = 30 +LOGGER = logging.getLogger(__name__) + + +class ProbeFailure(AssertionError): + def __init__(self, message, stages): + super().__init__(message) + self.stages = stages + + +def expected_result(phase, probe_id, dicom): + expected = {"phase": phase, "probe": probe_id, "private_dns": True} + if phase == "access": + return {**expected, "fhir_version": "4.0.1", "anonymous": 401, "unassigned": 403} + expected["count"] = 1 + if dicom: + expected["dicom_anonymous"] = 401 + return expected + + +def probe_result(resource, phase, probe_id, dicom=False): + properties = resource.get("properties", {}) + view = properties.get("instanceView", {}) + state = view.get("executionState", "") + lines = view.get("output", "").splitlines() + if properties.get("provisioningState") in ("Failed", "Canceled") or state in ("Failed", "Canceled", "TimedOut"): + stages = [line[len(FAILED) :] for line in lines if line.startswith(FAILED)] + statuses = [line[len(STATUS) :] for line in lines if line.startswith(STATUS)] + raise ProbeFailure( + f"Health probe failed: state={state}, exitCode={view.get('exitCode')}, " + f"stage={','.join(stages) or 'unknown'}, http_status={','.join(statuses) or 'none'}", + stages, + ) + if state != "Succeeded": + return False + if view.get("exitCode") != 0: + raise AssertionError("Health probe returned a non-zero exit code") + results = [line[len(RESULT) :] for line in lines if line.startswith(RESULT)] + if len(results) != 1: + raise AssertionError("Health probe returned no single result") + actual = json.loads(results[0]) + if actual != expected_result(phase, probe_id, dicom): + raise AssertionError(f"Health probe result does not match: {actual}") + return True + + +@asynccontextmanager +async def role_access(arm, scope, role, principal_id, subscription_id): + """Grant only this VM a data role on one service, then remove the assignment.""" + assignment = f"{scope}/providers/Microsoft.Authorization/roleAssignments/{uuid4()}" + body = { + "properties": { + "roleDefinitionId": f"/subscriptions/{subscription_id}/providers/Microsoft.Authorization/roleDefinitions/{role}", + "principalId": principal_id, + "principalType": "ServicePrincipal", + } + } + # The PUT may have succeeded even if its response was lost. + async with delete_after(arm, assignment, ROLE_API): + await arm.request("PUT", assignment, ROLE_API, body=body) + yield + + +async def public_statuses(client, urls): + """Send requests from outside the workspace network until Azure blocks each one, or the wait ends.""" + deadline = asyncio.get_running_loop().time() + PUBLIC_WAIT_SECONDS + while True: + statuses = {name: (await client.get(url)).status_code for name, url in urls.items()} + if set(statuses.values()) == {PUBLIC_BLOCKED} or asyncio.get_running_loop().time() >= deadline: + return statuses + await asyncio.sleep(PUBLIC_RETRY_SECONDS) + + +async def run_probe(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): + # A lookup of the FHIR name failed once on a new VM. The probe changes no data before this stage. + for attempt in range(1, DNS_ATTEMPTS + 1): + try: + return await run_probe_once( + arm, vm_id, location, fhir_url=fhir_url, probe_id=probe_id, phase=phase, dicom_url=dicom_url + ) + except ProbeFailure as error: + if error.stages != ["private-dns"] or attempt == DNS_ATTEMPTS: + raise + LOGGER.warning("Health probe %s failed at private-dns, attempt %d. Trying again.", phase, attempt) + await asyncio.sleep(DNS_RETRY_SECONDS) + + +async def run_probe_once(arm, vm_id, location, *, fhir_url, probe_id, phase, dicom_url=""): + parameters = {"FhirUrl": fhir_url, "ProbeId": probe_id, "Phase": phase} + if dicom_url: + parameters["DicomUrl"] = dicom_url + command = f"{vm_id}/runCommands/health-probe-{uuid4().hex}" + # Removing our command also terminates it if its deadline expired. + async with delete_after(arm, command, COMPUTE_API): + async with asyncio.timeout(20 * 60): + await arm.request( + "PUT", + command, + COMPUTE_API, + body={ + "location": location, + "properties": { + "source": {"script": SCRIPT.read_text()}, + "parameters": [{"name": k, "value": v} for k, v in parameters.items()], + "timeoutInSeconds": 900, + "treatFailureAsDeploymentFailure": True, + }, + }, + ) + while True: + result = await arm.request("GET", command, COMPUTE_API, expand=True) + if probe_result(result, phase, probe_id, bool(dicom_url)): + return + await asyncio.sleep(10) diff --git a/e2e_tests/resources/sql_lifecycle.py b/e2e_tests/resources/sql_lifecycle.py index 8b19ae9986..0f36a2b91d 100644 --- a/e2e_tests/resources/sql_lifecycle.py +++ b/e2e_tests/resources/sql_lifecycle.py @@ -65,22 +65,27 @@ async def enter(self, name, context): await context.__aenter__() self.own(name, context.__aexit__, None, None, None) - async def setup(self): + async def setup(self, groups=False): # Keep these resources inside the SQL budget rather than session fixtures. + # Only an Automatic workspace gets Entra ID groups, so a case that needs + # them creates its own workspace and does not use the configured ones. + manual = bool(config.TEST_WORKSPACE_APP_ID) and not groups + existing_workspace = "" if groups else config.TEST_WORKSPACE_ID + existing_service = "" if groups else config.TEST_WORKSPACE_SERVICE_ID workspace_path, workspace_id = await create_or_get_test_workspace( - auth_type="Manual" if config.TEST_WORKSPACE_APP_ID else "Automatic", + auth_type="Manual" if manual else "Automatic", verify=self.verify, - pre_created_workspace_id=config.TEST_WORKSPACE_ID, - client_id=config.TEST_WORKSPACE_APP_ID, - client_secret=config.TEST_WORKSPACE_APP_SECRET, + pre_created_workspace_id=existing_workspace, + client_id=config.TEST_WORKSPACE_APP_ID if manual else "", + client_secret=config.TEST_WORKSPACE_APP_SECRET if manual else "", ) - if not config.TEST_WORKSPACE_ID: + if not existing_workspace: self.own(workspace_path, clean_up_test_workspace, "", workspace_path, self.verify) token = await get_workspace_owner_token(workspace_id, self.verify) guacamole_path, guacamole_id = await create_or_get_test_workpace_service( - workspace_path, token, config.TEST_WORKSPACE_SERVICE_ID, self.verify + workspace_path, token, existing_service, self.verify ) - if not config.TEST_WORKSPACE_SERVICE_ID: + if not existing_service: self.own(guacamole_path, clean_up_test_workspace_service, "", guacamole_path, workspace_id, self.verify) return workspace_path, workspace_id, guacamole_path, guacamole_id diff --git a/e2e_tests/test_health.py b/e2e_tests/test_health.py new file mode 100644 index 0000000000..b9e8900f03 --- /dev/null +++ b/e2e_tests/test_health.py @@ -0,0 +1,133 @@ +"""Exercise private FHIR access, access control and a DICOM update.""" + +import asyncio +from functools import partial +import logging +import os +from uuid import uuid4 + +from httpx import AsyncClient +import pytest + +from e2e_tests import config +from e2e_tests.conftest import get_workspace_owner_token +from e2e_tests.resources import strings +from e2e_tests.resources.health_probe import ( + FHIR_DATA_CONTRIBUTOR, + PUBLIC_BLOCKED, + public_statuses, + role_access, + run_probe, +) +from e2e_tests.resources.resource import get_resource, post_resource +from e2e_tests.resources.sql_lifecycle import sql_lifecycle +from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, validate_vm + +LOGGER = logging.getLogger(__name__) +HEALTH_API = "2024-03-31" +NETWORK_API = "2024-05-01" +HEALTH_PAYLOAD = { + "templateName": strings.HEALTH_SERVICE, + "properties": { + "display_name": "Health lifecycle test", + "description": "Private FHIR and DICOM validation", + "deploy_fhir": True, + "fhir_kind": "R4", + "deploy_dicom": False, + }, +} +VM_PAYLOAD = { + "templateName": strings.GUACAMOLE_WINDOWS_USER_RESOURCE, + "properties": { + "display_name": "Health test client", + "description": "Temporary private FHIR validation client", + "os_image": "Windows 11", + "vm_size": "2 CPU | 8GB RAM", + "admin_username": "researcher", + "shared_storage_access": False, + **{ + f"install_{tool}": False + for tool in ("azure_cli", "vscode", "storage_explorer", "git", "python_tools", "r_tools") + }, + }, +} + + +@pytest.mark.asyncio(loop_scope="session") +@pytest.mark.health_validation +@pytest.mark.timeout(285 * 60) +async def test_private_fhir_access_and_dicom_update(verify): + if os.environ.get("AZURE_ENVIRONMENT", "AzureCloud") != "AzureCloud": + raise ValueError("The health probe currently supports AzureCloud only") + async with sql_lifecycle(verify, deadline_variable="HEALTH_VALIDATION_DEADLINE") as resources: + # The bundle refuses to install without the workspace owners group. + workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup(groups=True) + # The owner token gets a current token for each request, so one token lasts the case and its clean-up. + token = await get_workspace_owner_token(workspace_id, verify) + workspace = (await get_resource(f"/api{workspace_path}", token, verify))["workspace"] + assert workspace["properties"].get("workspace_owners_group_id"), "The workspace has no owners group" + health_path, health_id = await resources.create( + HEALTH_PAYLOAD, f"/api{workspace_path}/workspace-services", token + ) + vm_path, vm_id = await resources.create(VM_PAYLOAD, f"/api{guacamole_path}/user-resources", token) + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] + vm_properties = (await get_resource(f"/api{vm_path}", token, verify))["userResource"]["properties"] + subscription_id = vm_properties.get("workspace_subscription_id") or os.environ["ARM_SUBSCRIPTION_ID"] + suffix = f"{workspace_id[-4:]}svc{health_id[-4:]}" + fhir_url = f"https://hs{suffix}-fhir{suffix}.fhir.azurehealthcareapis.com" + dicom_url = f"https://hs{suffix}-dicom{suffix}.dicom.azurehealthcareapis.com" + assert health["properties"]["fhir_url"] == fhir_url + assert not health["properties"].get("dicom_url") + azure_vm_id = vm_properties["azure_resource_id"] + async with arm_client() as arm: + vm = await arm.request("GET", azure_vm_id, COMPUTE_API) + group = validate_vm(vm, azure_vm_id, subscription_id, config.TRE_ID, workspace_id, guacamole_id, vm_id) + workspace_arm_id = f"{group}/providers/Microsoft.HealthcareApis/workspaces/hs{suffix}" + fhir_id = f"{workspace_arm_id}/fhirservices/fhir{suffix}" + fhir = await arm.request("GET", fhir_id, HEALTH_API) + assert fhir["properties"]["provisioningState"] == "Succeeded" + endpoint_id = f"{group}/providers/Microsoft.Network/privateEndpoints/pe-hs{suffix}" + endpoint = await arm.request("GET", endpoint_id, NETWORK_API) + probe_id = uuid4().hex + probe = partial(run_probe, arm, azure_vm_id, vm["location"], fhir_url=fhir_url, probe_id=probe_id) + await probe(phase="access") + LOGGER.info("FHIR metadata is private, and data requests need a token and a role") + async with role_access(arm, fhir_id, FHIR_DATA_CONTRIBUTOR, vm["identity"]["principalId"], subscription_id): + await probe(phase="write") + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] + async with asyncio.timeout(60 * 60): + await post_resource( + {"properties": {"deploy_dicom": True}}, + f"/api{health_path}", + token, + verify, + method="PATCH", + etag=health["_etag"], + ) + health = (await get_resource(f"/api{health_path}", token, verify))["workspaceService"] + assert health["properties"]["dicom_url"] == dicom_url + dicom_id = f"{workspace_arm_id}/dicomservices/dicom{suffix}" + dicom = await arm.request("GET", dicom_id, HEALTH_API) + assert dicom["properties"]["provisioningState"] == "Succeeded" + # A new endpoint stops private access to FHIR for some minutes. + updated_endpoint = await arm.request("GET", endpoint_id, NETWORK_API) + assert updated_endpoint["properties"]["resourceGuid"] == endpoint["properties"]["resourceGuid"], ( + "The update replaced the Health private endpoint" + ) + await probe(phase="read", dicom_url=dicom_url) + LOGGER.info("FHIR data survived the update that added DICOM") + # Check the network settings last, so that a failure here keeps the functional evidence above. + async with AsyncClient(timeout=30) as client: + public = await public_statuses( + client, {"fhir": f"{fhir_url}/metadata", "dicom": f"{dicom_url}/health/check"} + ) + # Read the resources again after the update. + # FHIR is not in this list: Azure keeps publicNetworkAccess Enabled on FHIR, also after a request + # sets Disabled. The private endpoint on the workspace blocks public traffic to FHIR (checked above). + network = {} + for name, resource_id in (("workspace", workspace_arm_id), ("dicom", dicom_id)): + record = await arm.request("GET", resource_id, HEALTH_API) + network[name] = record["properties"].get("publicNetworkAccess") + LOGGER.info("Health public responses: %s, public network access: %s", public, network) + assert public == dict.fromkeys(public, PUBLIC_BLOCKED), f"Health services answer public requests: {public}" + assert network == dict.fromkeys(network, "Disabled"), f"Health services allow public access: {network}" diff --git a/e2e_tests/test_workspace_services.py b/e2e_tests/test_workspace_services.py index 3ce22c5614..bcb7efcd07 100644 --- a/e2e_tests/test_workspace_services.py +++ b/e2e_tests/test_workspace_services.py @@ -11,7 +11,6 @@ strings.AZUREML_SERVICE, strings.GITEA_SERVICE, strings.MYSQL_SERVICE, - strings.HEALTH_SERVICE, strings.AZURESQL_SERVICE, ] diff --git a/e2e_tests/unit_tests/test_health_bundle_contract.py b/e2e_tests/unit_tests/test_health_bundle_contract.py new file mode 100644 index 0000000000..dbe2896309 --- /dev/null +++ b/e2e_tests/unit_tests/test_health_bundle_contract.py @@ -0,0 +1,125 @@ +"""Check the private network settings of the Health bundle.""" + +import json +from pathlib import Path +import re +import unittest + +ROOT = Path(__file__).resolve().parents[2] +BUNDLE = ROOT / "templates/workspace_services/health-services/terraform" + + +def block(source, header): + return source.split(header + " {", 1)[1].split("\n}", 1)[0] + + +def version(text): + return tuple(int(part) for part in text.split(".")) + + +class HealthBundleContractTests(unittest.TestCase): + def test_dns_zone_group_gets_the_dicom_zone_without_an_endpoint_change(self): + # A zone group change in the endpoint resource also sends the whole endpoint. Azure + # refuses this with HTTP 409 while it links a new service, so the group is separate. + source = (BUNDLE / "main.tf").read_text() + endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') + self.assertNotIn("private_dns_zone_group {", endpoint) + self.assertIn("ignore_changes = [tags, private_dns_zone_group]", endpoint) + self.assertNotIn("replace_triggered_by", endpoint) + + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + self.assertIn("parent_id = azurerm_private_endpoint.health_services_private_endpoint.id", group) + self.assertIn("var.deploy_dicom ? [{", group) + self.assertIn('error_message_regex = ["AnotherOperationInProgress"]', group) + self.assertIn("azurerm_healthcare_dicom_service.dicom", group.split("depends_on", 1)[1]) + + def test_an_update_that_adds_fhir_creates_the_zone_group_again(self): + # The FHIR record is in the zone of the workspace record, so the group body stays the same. + source = (BUNDLE / "main.tf").read_text() + trigger = block(source, 'resource "terraform_data" "fhir_deployed"') + self.assertIn("input = var.deploy_fhir", trigger) + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + self.assertIn("replace_triggered_by = [terraform_data.fhir_deployed]", group) + + def test_private_endpoint_is_created_after_the_services(self): + # Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. + source = (BUNDLE / "main.tf").read_text() + endpoint = block(source, 'resource "azurerm_private_endpoint" "health_services_private_endpoint"') + depends_on = endpoint.split("depends_on", 1)[1] + self.assertIn("azurerm_healthcare_fhir_service.fhir", depends_on) + self.assertIn("azurerm_healthcare_dicom_service.dicom", depends_on) + for header in ( + 'resource "azurerm_healthcare_fhir_service" "fhir"', + 'resource "azurerm_healthcare_dicom_service" "dicom"', + ): + self.assertNotIn("depends_on", block(source, header)) + + def test_a_service_without_fhir_and_dicom_stops_before_a_change(self): + source = (BUNDLE / "main.tf").read_text() + workspace = block(source, 'resource "azurerm_healthcare_workspace" "healthcare_workspace"') + precondition = workspace.split("precondition {", 1)[1] + self.assertIn("condition = var.deploy_fhir || var.deploy_dicom", precondition) + self.assertIn('error_message = "Deploy FHIR, DICOM or both.', precondition) + + def test_the_form_and_the_bundle_use_the_same_defaults(self): + # The API stores a request without schema defaults, so the bundle default applies to an absent property. + schema = json.loads((BUNDLE.parent / "template_schema.json").read_text()) + porter = (BUNDLE.parent / "porter.yaml").read_text() + for name in ("deploy_fhir", "deploy_dicom"): + with self.subTest(name=name): + self.assertIs(schema["properties"][name]["default"], False) + self.assertRegex(porter, rf"- name: {name}\n type: boolean\n default: false\n") + # The update form removes fhir_kind, which cannot change, so a request that turns on FHIR can lack it. + fhir_kind = schema["allOf"][0]["then"]["properties"]["fhir_kind"]["default"] + self.assertEqual(fhir_kind, "R4") + self.assertIn(f'- name: fhir_kind\n type: string\n default: "{fhir_kind}"\n', porter) + + def test_upgrade_removes_the_zone_group_of_the_endpoint_resource(self): + source = (BUNDLE / "main.tf").read_text() + group = block(source, 'resource "azapi_resource" "health_services_dns_zone_group"') + remove = block(source, 'resource "azapi_resource_action" "remove_endpoint_dns_zone_group"') + self.assertIn('method = "DELETE"', remove) + self.assertIn('/privateDnsZoneGroups/private-dns-zone-group"', remove) + self.assertIn('name = "private-dns-zone-group"', group) + self.assertIn("azapi_resource_action.remove_endpoint_dns_zone_group", group) + # The old group stays until FHIR and DICOM have their changes. + depends_on = remove.split("depends_on", 1)[1] + self.assertIn("azurerm_healthcare_fhir_service.fhir", depends_on) + self.assertIn("azurerm_healthcare_dicom_service.dicom", depends_on) + + def test_azapi_can_update_resources_with_the_terraform_of_the_bundle(self): + # Before 2.9.0, azapi returns no resource identity after an update. Terraform 1.12 and + # later keep the identity, but earlier versions stop the update with an error. + porter = (BUNDLE.parent / "porter.yaml").read_text() + providers = (BUNDLE / "providers.tf").read_text() + terraform = re.search(r"- terraform:\s*\n\s*clientVersion:\s*([\d.]+)", porter).group(1) + azapi = re.search(r'source\s*=\s*"Azure/azapi"\s*\n\s*version\s*=\s*"=\s*([\d.]+)"', providers).group(1) + if version(terraform) < (1, 12): + self.assertGreaterEqual(version(azapi), (2, 9, 0)) + + def test_dicom_public_network_access_is_off(self): + # The provider turns it on unless the bundle sets it. + source = (BUNDLE / "main.tf").read_text() + dicom = block(source, 'resource "azurerm_healthcare_dicom_service" "dicom"') + self.assertRegex(dicom, r"\n public_network_access_enabled\s*=\s*false\n") + + def test_role_assignments_use_the_role_names(self): + # Azure returns the role definition ID with the subscription. The role definition data + # source gives the ID without it, so the plan replaced each role assignment. + roles = (BUNDLE / "roles.tf").read_text() + self.assertNotIn("role_definition_id", roles) + self.assertEqual(roles.count("role_definition_name"), 4) + + def test_fhir_roles_wait_for_the_private_endpoint(self): + # The workspace groups get FHIR data access only through the private endpoint. + roles = (BUNDLE / "roles.tf").read_text() + for name in ("researchers_fhir_contributor", "owners_fhir_contributor"): + role = block(roles, f'resource "azurerm_role_assignment" "{name}"') + self.assertIn("depends_on = [azapi_resource.health_services_dns_zone_group]", role) + + def test_dicom_roles_wait_for_the_endpoint_link(self): + # Azure accepts the zone group write only after it links a new DICOM service to the endpoint. + roles = (BUNDLE / "roles.tf").read_text() + for name in ("researchers_dicom_data_owner", "owners_dicom_data_owner"): + role = block(roles, f'resource "azurerm_role_assignment" "{name}"') + self.assertIn("depends_on = [azapi_resource.health_services_dns_zone_group]", role) diff --git a/e2e_tests/unit_tests/test_health_bundle_providers.py b/e2e_tests/unit_tests/test_health_bundle_providers.py new file mode 100644 index 0000000000..d7c4052d7f --- /dev/null +++ b/e2e_tests/unit_tests/test_health_bundle_providers.py @@ -0,0 +1,28 @@ +"""Check the provider versions of the Health bundle.""" + +from pathlib import Path +import re +import unittest + +ROOT = Path(__file__).resolve().parents[2] +BUNDLE = ROOT / "templates/workspace_services/health-services/terraform" + + +def version(text): + return tuple(int(part) for part in text.split(".")) + + +class HealthBundleProviderTests(unittest.TestCase): + def test_azurerm_keeps_a_new_fhir_service_in_the_state(self): + # Before 4.75.0, azurerm waits for Succeeded after the FHIR create and saves the ID after this + # wait. If Azure reports Accepted, the create fails, and the state does not get the service. + providers = (BUNDLE / "providers.tf").read_text() + lock = (BUNDLE / ".terraform.lock.hcl").read_text() + azurerm = re.search(r'source\s*=\s*"hashicorp/azurerm"\s*\n\s*version\s*=\s*"=\s*([\d.]+)"', providers).group(1) + locked = re.search(r'"registry.terraform.io/hashicorp/azurerm" \{\s*version\s*=\s*"([\d.]+)"', lock).group(1) + self.assertGreaterEqual(version(azurerm), (4, 75, 0)) + self.assertEqual(locked, azurerm) + + +if __name__ == "__main__": + unittest.main() diff --git a/e2e_tests/unit_tests/test_health_probe.py b/e2e_tests/unit_tests/test_health_probe.py new file mode 100644 index 0000000000..7c97e1867e --- /dev/null +++ b/e2e_tests/unit_tests/test_health_probe.py @@ -0,0 +1,165 @@ +"""Check the health probe's evidence, role scope and cleanup boundaries.""" + +import json +from pathlib import Path +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock, patch + +import httpx +from jsonschema import validate +import yaml + +from api_app.services.schema_service import enrich_user_resource_template, enrich_workspace_service_template +from e2e_tests import test_health as health +from e2e_tests.resources import health_probe as probe + + +ROOT = Path(__file__).resolve().parents[2] +GROUP = "/subscriptions/sub/resourceGroups/rg-tretest-ws-abcd" +VM_ID = f"{GROUP}/providers/Microsoft.Compute/virtualMachines/windowsvm1234" +FHIR_ID = f"{GROUP}/providers/Microsoft.HealthcareApis/workspaces/hsabcdsvc1234/fhirservices/fhirabcdsvc1234" +FHIR_URL = "https://hsabcdsvc1234-fhirabcdsvc1234.fhir.azurehealthcareapis.com" +PROBE_ID = "0123456789abcdef0123456789abcdef" + + +def result(evidence, state="Succeeded", code=0, output=None): + return { + "properties": { + "provisioningState": "Succeeded", + "instanceView": { + "executionState": state, + "exitCode": code, + "output": output if output is not None else "HEALTH_PROBE_RESULT=" + json.dumps(evidence), + }, + } + } + + +class ProbeEvidenceTests(unittest.TestCase): + def test_access_phase_requires_private_metadata_and_denied_requests(self): + evidence = probe.expected_result("access", PROBE_ID, False) + self.assertEqual(evidence["anonymous"], 401) + self.assertEqual(evidence["unassigned"], 403) + self.assertTrue(probe.probe_result(result(evidence), "access", PROBE_ID)) + for change in ({"anonymous": 200}, {"unassigned": 200}, {"private_dns": False}, {"fhir_version": "3.0.2"}): + with self.subTest(change=change), self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, **change}), "access", PROBE_ID) + + def test_data_phases_require_the_written_row(self): + for phase, dicom in (("write", False), ("read", True)): + evidence = probe.expected_result(phase, PROBE_ID, dicom) + with self.subTest(phase=phase): + self.assertTrue(probe.probe_result(result(evidence), phase, PROBE_ID, dicom)) + with self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, "count": 0}), phase, PROBE_ID, dicom) + evidence = probe.expected_result("read", PROBE_ID, False) + with self.assertRaises(AssertionError): + probe.probe_result(result(evidence), "read", PROBE_ID, True) + + def test_running_command_is_not_a_pass_and_failure_names_the_stage(self): + running = result({}) + running["properties"]["instanceView"] = {} + self.assertFalse(probe.probe_result(running, "access", PROBE_ID)) + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-role\n") + with self.assertRaisesRegex(AssertionError, "stage=fhir-role, http_status=none"): + probe.probe_result(failed, "write", PROBE_ID) + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-read\nHEALTH_PROBE_STATUS=403\n") + with self.assertRaisesRegex(AssertionError, "stage=fhir-read, http_status=403"): + probe.probe_result(failed, "write", PROBE_ID) + + def test_payloads_match_enriched_api_schemas(self): + cases = ( + (health.HEALTH_PAYLOAD, "health-services", enrich_workspace_service_template), + (health.VM_PAYLOAD, "guacamole/user_resources/guacamole-azure-windowsvm", enrich_user_resource_template), + ) + for payload, folder, enrich in cases: + with self.subTest(bundle=payload["templateName"]): + schema = json.loads( + (ROOT / "templates/workspace_services" / folder / "template_schema.json").read_text() + ) + validate(payload["properties"], enrich(SimpleNamespace(model_dump=lambda **_: schema))) + self.assertFalse(any(v for k, v in health.VM_PAYLOAD["properties"].items() if k.startswith("install_"))) + + def test_deadline_precedes_container(self): + jobs = yaml.safe_load((ROOT / ".github/workflows/deploy_tre_reusable.yml").read_text())["jobs"] + job = jobs["e2e_tests_bundle"] + reserve = next(s for s in job["steps"] if "HEALTH_VALIDATION_DEADLINE" in s.get("run", "")) + run = next(s for s in job["steps"] if s["name"] == "Run E2E Tests") + self.assertLess(job["steps"].index(reserve), job["steps"].index(run)) + self.assertEqual(reserve["if"], "inputs.e2eBundle == 'tre-workspace-service-health'") + self.assertIn("HEALTH_VALIDATION_DEADLINE=${{ env.HEALTH_VALIDATION_DEADLINE }}", run["with"]["COMMAND"]) + + +class ProbeRecoveryTests(unittest.IsolatedAsyncioTestCase): + async def test_role_is_scoped_to_one_service_and_removed_after_failure(self): + arm = SimpleNamespace(request=AsyncMock(return_value={}), delete=AsyncMock()) + with self.assertRaisesRegex(RuntimeError, "write failed"): + async with probe.role_access(arm, FHIR_ID, probe.FHIR_DATA_CONTRIBUTOR, "principal", "sub"): + raise RuntimeError("write failed") + assignment = arm.request.call_args.args[1] + self.assertTrue(assignment.startswith(FHIR_ID + "/providers/Microsoft.Authorization/roleAssignments/")) + body = arm.request.call_args.kwargs["body"]["properties"] + self.assertEqual(body["principalId"], "principal") + self.assertTrue(body["roleDefinitionId"].endswith("/roleDefinitions/" + probe.FHIR_DATA_CONTRIBUTOR)) + arm.delete.assert_awaited_once_with(assignment, probe.ROLE_API) + + async def test_failed_command_is_removed_and_dicom_is_optional(self): + failed = result({}, state="Failed", code=1, output="HEALTH_PROBE_FAILED=fhir-metadata\n") + for dicom_url in ("", "https://hsabcdsvc1234-dicomabcdsvc1234.dicom.azurehealthcareapis.com"): + arm = SimpleNamespace(request=AsyncMock(side_effect=[{}, failed]), delete=AsyncMock()) + with self.subTest(dicom_url=dicom_url), self.assertRaisesRegex(AssertionError, "fhir-metadata"): + await probe.run_probe( + arm, + VM_ID, + "switzerlandnorth", + fhir_url=FHIR_URL, + probe_id=PROBE_ID, + phase="read", + dicom_url=dicom_url, + ) + command = arm.request.call_args_list[0].args[1] + self.assertTrue(command.startswith(VM_ID + "/runCommands/health-probe-")) + arm.delete.assert_awaited_once_with(command, probe.COMPUTE_API) + names = [p["name"] for p in arm.request.call_args_list[0].kwargs["body"]["properties"]["parameters"]] + self.assertEqual(names, ["FhirUrl", "ProbeId", "Phase"] + (["DicomUrl"] if dicom_url else [])) + + async def test_only_a_dns_failure_is_tried_again(self): + def failed(stage): + return result({}, state="Failed", code=1, output=f"HEALTH_PROBE_FAILED={stage}\n") + + passed = result(probe.expected_result("access", PROBE_ID, False)) + cases = ( + ([failed("private-dns"), passed], None), + ([failed("private-dns")] * probe.DNS_ATTEMPTS, "stage=private-dns"), + ([failed("fhir-metadata")], "stage=fhir-metadata"), + ) + for results, error in cases: + # Each attempt creates a command, reads its result and then removes the command. + responses = [response for r in results for response in ({}, r)] + arm = SimpleNamespace(request=AsyncMock(side_effect=responses), delete=AsyncMock()) + with self.subTest(error=error), patch.object(probe, "DNS_RETRY_SECONDS", 0): + run = probe.run_probe( + arm, VM_ID, "switzerlandnorth", fhir_url=FHIR_URL, probe_id=PROBE_ID, phase="access" + ) + if error: + with self.assertRaisesRegex(AssertionError, error): + await run + else: + await run + self.assertEqual(arm.delete.await_count, len(results)) + + async def test_public_requests_are_sent_again_until_azure_blocks_them(self): + urls = {"fhir": FHIR_URL + "/metadata", "dicom": "https://dicom.example/health/check"} + cases = ( + ([200, 403, 403, 403], {}, {"fhir": 403, "dicom": 403}), + # If the wait ends first, the caller gets the last statuses and reports them. + ([200, 401], {"PUBLIC_WAIT_SECONDS": 0}, {"fhir": 200, "dicom": 401}), + ) + for codes, settings, expected in cases: + responses = iter(codes) + transport = httpx.MockTransport(lambda request: httpx.Response(next(responses))) + with self.subTest(codes=codes), patch.multiple(probe, PUBLIC_RETRY_SECONDS=0, **settings): + async with httpx.AsyncClient(transport=transport) as client: + statuses = await probe.public_statuses(client, urls) + self.assertEqual(statuses, expected) diff --git a/e2e_tests/unit_tests/test_sql_deadlines.py b/e2e_tests/unit_tests/test_sql_deadlines.py index 4bcdd4b23e..aa2e1f2635 100644 --- a/e2e_tests/unit_tests/test_sql_deadlines.py +++ b/e2e_tests/unit_tests/test_sql_deadlines.py @@ -121,6 +121,28 @@ async def test_existing_workspace_and_service_are_not_owned(self): await owned.setup() self.assertEqual(owned.finalisers, []) + async def test_group_setup_creates_and_owns_an_automatic_workspace(self): + create_workspace = AsyncMock(return_value=("/workspaces/ws", "ws")) + create_service = AsyncMock(return_value=("/service", "service")) + with ( + self.budgets(), + patch.object(lifecycle.config, "TEST_WORKSPACE_APP_ID", "app"), + patch.object(lifecycle.config, "TEST_WORKSPACE_APP_SECRET", "secret"), + patch.object(lifecycle.config, "TEST_WORKSPACE_ID", "existing"), + patch.object(lifecycle.config, "TEST_WORKSPACE_SERVICE_ID", "existing-service"), + patch.object(lifecycle, "create_or_get_test_workspace", create_workspace), + patch.object(lifecycle, "get_workspace_owner_token", AsyncMock(return_value="token")), + patch.object(lifecycle, "create_or_get_test_workpace_service", create_service), + ): + async with lifecycle.sql_lifecycle(True) as owned: + await owned.setup(groups=True) + self.assertEqual([name for name, _ in owned.finalisers], ["/workspaces/ws", "/service"]) + owned.finalisers.clear() + create_workspace.assert_awaited_once_with( + auth_type="Automatic", verify=True, pre_created_workspace_id="", client_id="", client_secret="" + ) + create_service.assert_awaited_once_with("/workspaces/ws", "token", "", True) + async def test_job_start_deadline_reduces_work_time(self): with patch.dict(os.environ, SQL_VALIDATION_DEADLINE=str(time.time() + 125 * 60)): work, finish = lifecycle.lifecycle_deadlines() diff --git a/templates/workspace_services/health-services/porter.yaml b/templates/workspace_services/health-services/porter.yaml index 3724596fc8..12f393fa60 100644 --- a/templates/workspace_services/health-services/porter.yaml +++ b/templates/workspace_services/health-services/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-health -version: 0.3.8 +version: 1.0.0 description: "An Azure Data Health Services workspace service" registry: azuretre dockerfile: Dockerfile.tmpl @@ -65,7 +65,7 @@ parameters: default: false - name: fhir_kind type: string - default: "" + default: "R4" outputs: - name: fhir_url diff --git a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl index 80913233fb..b93ee2ce7f 100644 --- a/templates/workspace_services/health-services/terraform/.terraform.lock.hcl +++ b/templates/workspace_services/health-services/terraform/.terraform.lock.hcl @@ -1,44 +1,47 @@ # This file is maintained automatically by "terraform init". # Manual edits may be lost in future updates. -provider "registry.terraform.io/hashicorp/azurerm" { - version = "4.58.0" - constraints = "4.58.0" +provider "registry.terraform.io/azure/azapi" { + version = "2.9.0" + constraints = "2.9.0" hashes = [ - "h1:6lb8gT8vG3vgraxqjePXVqOUUxP6OVOPzz1CxI2Kbj8=", - "h1:MESGDsXHZy6+1pCObte7/WSXaIzH9FUFy/Vp8MPVxiI=", - "h1:bPPD3lFdhTksJqlZlo+piO7OhuX5v9PgCSvWyQRlgPg=", - "zh:041c2a778ab4dd5a9af174b1d6f75409e5aabfc359cb386dfea3fb09e3f32709", - "zh:0a302531a61e7383acf99a6202d7984b2ea559306f45021381665c827a830d46", - "zh:0c69f132c7609683d907e87b89210a298d84c5b0121b62278949931bc54ca952", - "zh:0cadf48e9d2d9daed43212a3c9d886d7faaf68787b6e955456cbe4f43e4a17ec", - "zh:35ef4293d7731f6ff1f8bcba2c4529f987b7fac243c1ac1c154bbc02c9703c25", - "zh:3cb2679e1d56865e0ee0cf4c5d1404dbad0db42d11425e7bf0580a026cc64287", - "zh:4e56411f5119042d4962acff5c6d64224a49a69154ba80e6df63fa57b1e6d284", - "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:ca4626411a111720c220f9849c7d2e1fcd5d380f56459e096d835a9dbf9e6e13", - "zh:d31c4e65dcb096974479b2d548fffb86fc9a5262aff1b01fe62ef442ce536c6b", - "zh:d9631602999c1853e53ee2c5aef7476e23c7787beddc3599c10dbaa4891ba166", - "zh:f31ba7c9341037ceb7d49467946c01b2b0930404ed1d5643c1451f734a613a03", + "h1:XtiXcDMV+sEfQFcRot9fwKF/Y0aXeho9A/WVfHdHwCc=", + "h1:ivmN39jnMJRZXlPuX8r5TTXM8lyF8ZocPQQgEUy61eM=", + "h1:lN0C7TI2dqStnZysEBhDq33t9DbpUp/8rOWxZUF/st0=", + "h1:tF2SjvM2Vj+7MTbFXhssLYu1Uc9uK/cVWBYIH8heDqM=", + "zh:0a4eee8c9362db6ca19d371eccf38701c8306be761182da87b624294f7e8f867", + "zh:4df87bee5b8f4cce27461ae26132a599542583cdf035942b42b0259a514ab46e", + "zh:57dc1f4227f1d0eab630fcf868d6c9a1b4dc4c165608ce5a4d7028a482770547", + "zh:5c500e42c419c324ea8f41c40e4cc8af187323cebc9f70543749b6c0e59c0fe9", + "zh:6c7dca31242e27d2f215b1a9370b65579b3a988282a3609564ea96866f86f0ca", + "zh:74dada7351b25d01e61aa70aa8cf1f1fb96d09c514be4b66d12816c5e61f9e01", + "zh:9c67c0727ab8be15879793f929f3d71f0f149ae1daaf577d0bd4d57b2e61c408", + "zh:a0266da16db14b635a61d6766efc28468056bac91ed6662fc9f2a0aad923b063", + "zh:a4a555627f40fa797b34ddede599812cddb1e0e9be105ccfcf1f293451b97cd7", + "zh:a6a1c4a46f4f01e4420a456adefe8bcf3bff3c9d09a8dfeedeea5b94c866dfd8", + "zh:c64fc4067a8276d4405d9359990e0c45e66dba07433a1f3ab1e6c32f6ef3b048", + "zh:ee77a881d071b3a0ad8a64f9c1158e3aee9b37e9063de5dd8fa9d238dbe70ba1", ] } -provider "registry.terraform.io/hashicorp/external" { - version = "2.3.5" - constraints = "2.3.5" +provider "registry.terraform.io/hashicorp/azurerm" { + version = "5.11.0" + constraints = "5.11.0" hashes = [ - "h1:smKSos4zs57pJjQrNuvGBpSWth2el9SgePPbPHo0aps=", - "zh:6e89509d056091266532fa64de8c06950010498adf9070bf6ff85bc485a82562", + "h1:6OVoAYFOZ5OPkpJT5lS4Pt14o5a0LRjJgDPGBt/1VfE=", + "h1:QldFxdezZrp3E1bKnWS7xcU63Zmol1K7siWsXS2S3pg=", + "h1:pZJ3HOERxQUnw4kbPZ0KHRihAIrxiE2+duDOnxFrJgg=", + "zh:2345a6104bb485141afb8f0047a48a5f27c3bb4e6eeb360c7955f381227d88dd", + "zh:34e86c5ceeba746fed119775cd37d7314ea8b7c2196ed8b8da8d8d71e10b96df", + "zh:358f30581df0670719fc8511066ef9d05eb2612934c147bfe1b1b1d67712d2f0", + "zh:3edb6681e17535f16ad1f2b7588325d86d7af57df9eed82fa4a607fc03e8afc7", + "zh:6a5b9d6dd86a759f8bfd81e15e7e688531c85ee9660bd5923f99e993a1f15c62", "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:86868aec05b58dc0aa1904646a2c26b9367d69b890c9ad70c33c0d3aa7b1485a", - "zh:a2ce38fda83a62fa5fb5a70e6ca8453b168575feb3459fa39803f6f40bd42154", - "zh:a6c72798f4a9a36d1d1433c0372006cc9b904e8cfd60a2ae03ac5b7d2abd2398", - "zh:a8a3141d2fc71c86bf7f3c13b0b3be8a1b0f0144a47572a15af4dfafc051e28a", - "zh:aa20a1242eb97445ad26ebcfb9babf2cd675bdb81cac5f989268ebefa4ef278c", - "zh:b58a22445fb8804e933dcf835ab06c29a0f33148dce61316814783ee7f4e4332", - "zh:cb5626a661ee761e0576defb2a2d75230a3244799d380864f3089c66e99d0dcc", - "zh:d1acb00d20445f682c4e705c965e5220530209c95609194c2dc39324f3d4fcce", - "zh:d91a254ba77b69a29d8eae8ed0e9367cbf0ea6ac1a85b58e190f8cb096a40871", - "zh:f6592327673c9f85cdb6f20336faef240abae7621b834f189c4a62276ea5db41", + "zh:a470dc95f15dcded2e06dd45c44c3b50a22a45b87c9ba7bd19dc6b25022660b7", + "zh:b179f04830c1d8d7ec15ae4be4434042d87679bb23b7d60d7ebd751420bf646c", + "zh:b183835acbcf39047d195f933caebd1902065d2437df17e964ceb4a358441f41", + "zh:b3d3555ac1f16a59baaab894fe9e839baedc809aef05a6d8e6c0cdaba9c92749", + "zh:cc29f59113d468757e9f1a3db774c73d8827f8359816216b7b82ddef5ff02d66", + "zh:e37d3649320c50110675156174f476b3b12ca5c1f267f1cadb56ad26e7f3553e", ] } diff --git a/templates/workspace_services/health-services/terraform/main.tf b/templates/workspace_services/health-services/terraform/main.tf index 0189b5e456..2b281d4464 100644 --- a/templates/workspace_services/health-services/terraform/main.tf +++ b/templates/workspace_services/health-services/terraform/main.tf @@ -4,7 +4,15 @@ resource "azurerm_healthcare_workspace" "healthcare_workspace" { location = data.azurerm_resource_group.ws.location tags = local.workspace_service_tags - lifecycle { ignore_changes = [tags] } + lifecycle { + ignore_changes = [tags] + + # Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service. + precondition { + condition = var.deploy_fhir || var.deploy_dicom + error_message = "Deploy FHIR, DICOM or both. Azure fails a private endpoint to a healthcare workspace without a FHIR or DICOM service." + } + } } resource "azurerm_healthcare_fhir_service" "fhir" { @@ -25,7 +33,6 @@ resource "azurerm_healthcare_fhir_service" "fhir" { type = "SystemAssigned" } - lifecycle { ignore_changes = [tags] } } @@ -36,6 +43,9 @@ resource "azurerm_healthcare_dicom_service" "dicom" { location = data.azurerm_resource_group.ws.location tags = local.workspace_service_tags + # The provider default turns on public network access. + public_network_access_enabled = false + identity { type = "SystemAssigned" } @@ -50,11 +60,6 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { subnet_id = data.azurerm_subnet.services.id tags = local.workspace_service_tags - private_dns_zone_group { - name = "private-dns-zone-group" - private_dns_zone_ids = [data.azurerm_private_dns_zone.health.id, data.azurerm_private_dns_zone.dicom.id] - } - private_service_connection { private_connection_resource_id = azurerm_healthcare_workspace.healthcare_workspace.id name = "psc-${azurerm_healthcare_workspace.healthcare_workspace.name}" @@ -62,10 +67,68 @@ resource "azurerm_private_endpoint" "health_services_private_endpoint" { is_manual_connection = false } + # Azure fails an endpoint to a healthcare workspace without a FHIR or DICOM service. depends_on = [ azurerm_healthcare_fhir_service.fhir, azurerm_healthcare_dicom_service.dicom ] - lifecycle { ignore_changes = [tags] } + # The DNS zone group is a separate resource. The provider changes a zone group in this + # resource only together with the whole endpoint. + lifecycle { ignore_changes = [tags, private_dns_zone_group] } +} + +# Before version 1.0.0, the endpoint resource created the DNS zone group. This removes +# that group once, so that the resource below can create a group with the same name. +resource "azapi_resource_action" "remove_endpoint_dns_zone_group" { + type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" + resource_id = "${azurerm_private_endpoint.health_services_private_endpoint.id}/privateDnsZoneGroups/private-dns-zone-group" + method = "DELETE" + + retry = { + error_message_regex = ["AnotherOperationInProgress"] + } + + # The delete waits for the changes to FHIR and DICOM, so that the records are absent only + # until the new group exists. If a change to a service fails, the old group stays. + depends_on = [ + azurerm_healthcare_fhir_service.fhir, + azurerm_healthcare_dicom_service.dicom + ] +} + +# The workspace and FHIR records are in the same zone, so an update that adds FHIR does not +# change the group. A new group makes Azure write the records of all services of the endpoint. +resource "terraform_data" "fhir_deployed" { + input = var.deploy_fhir +} + +# Azure writes the private DNS records of a zone when the group gets that zone. Azure +# links a new service to the endpoint after the service create returns. A change to the +# group during this time gets HTTP 409, so the provider tries again. +resource "azapi_resource" "health_services_dns_zone_group" { + type = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-05-01" + name = "private-dns-zone-group" + parent_id = azurerm_private_endpoint.health_services_private_endpoint.id + + body = { + properties = { + privateDnsZoneConfigs = concat( + [{ name = "health", properties = { privateDnsZoneId = data.azurerm_private_dns_zone.health.id } }], + var.deploy_dicom ? [{ name = "dicom", properties = { privateDnsZoneId = data.azurerm_private_dns_zone.dicom.id } }] : [] + ) + } + } + + retry = { + error_message_regex = ["AnotherOperationInProgress"] + } + + depends_on = [ + azapi_resource_action.remove_endpoint_dns_zone_group, + azurerm_healthcare_fhir_service.fhir, + azurerm_healthcare_dicom_service.dicom + ] + + lifecycle { replace_triggered_by = [terraform_data.fhir_deployed] } } diff --git a/templates/workspace_services/health-services/terraform/providers.tf b/templates/workspace_services/health-services/terraform/providers.tf index fc3dcaeab4..e74ffb7415 100644 --- a/templates/workspace_services/health-services/terraform/providers.tf +++ b/templates/workspace_services/health-services/terraform/providers.tf @@ -1,9 +1,15 @@ # Azure Provider source and version being used terraform { required_providers { + # Before 4.75.0, a FHIR create that Azure reports as Accepted fails, and the state does not get the service. azurerm = { source = "hashicorp/azurerm" - version = "= 4.58.0" + version = "= 5.11.0" + } + # Versions before 2.9.0 fail each update when Terraform is earlier than 1.12. + azapi = { + source = "Azure/azapi" + version = "= 2.9.0" } } @@ -11,6 +17,8 @@ terraform { } provider "azurerm" { + # Version 5 registers no resource providers by default. Keep the registrations of version 4. + resource_provider_registrations = "legacy" features { key_vault { # Don't purge on destroy (this would fail due to purge protection being enabled on keyvault) @@ -28,6 +36,8 @@ provider "azurerm" { storage_use_azuread = true } +provider "azapi" {} + module "terraform_azurerm_environment_configuration" { source = "git::https://github.com/microsoft/terraform-azurerm-environment-configuration.git?ref=0.7.0" arm_environment = var.arm_environment diff --git a/templates/workspace_services/health-services/terraform/roles.tf b/templates/workspace_services/health-services/terraform/roles.tf index 8af1f03ed3..05ea331ce1 100644 --- a/templates/workspace_services/health-services/terraform/roles.tf +++ b/templates/workspace_services/health-services/terraform/roles.tf @@ -1,37 +1,46 @@ +# Azure returns the role definition ID with the subscription. An ID without it in the +# configuration replaces each role assignment at each update, so these use the role name. + # Role assignments for workspace researchers group resource "azurerm_role_assignment" "researchers_fhir_contributor" { - count = var.deploy_fhir && var.workspace_researchers_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_fhir_service.fhir[0].id - role_definition_id = data.azurerm_role_definition.azure_fhir_contributor.id - principal_id = var.workspace_researchers_group_id + count = var.deploy_fhir && var.workspace_researchers_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_fhir_service.fhir[0].id + role_definition_name = "FHIR Data Contributor" + principal_id = var.workspace_researchers_group_id + + # The groups get FHIR data access only after the private endpoint and its DNS records exist. + # On uninstall, Terraform deletes the role before the zone group and the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } resource "azurerm_role_assignment" "researchers_dicom_data_owner" { - count = var.deploy_dicom && var.workspace_researchers_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_dicom_service.dicom[0].id - role_definition_id = data.azurerm_role_definition.azure_dicom_data_owner.id - principal_id = var.workspace_researchers_group_id + count = var.deploy_dicom && var.workspace_researchers_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_dicom_service.dicom[0].id + role_definition_name = "DICOM Data Owner" + principal_id = var.workspace_researchers_group_id + + # The zone group write succeeds only after Azure links a new DICOM service to the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } # Role assignments for workspace owners group resource "azurerm_role_assignment" "owners_fhir_contributor" { - count = var.deploy_fhir && var.workspace_owners_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_fhir_service.fhir[0].id - role_definition_id = data.azurerm_role_definition.azure_fhir_contributor.id - principal_id = var.workspace_owners_group_id -} + count = var.deploy_fhir && var.workspace_owners_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_fhir_service.fhir[0].id + role_definition_name = "FHIR Data Contributor" + principal_id = var.workspace_owners_group_id -resource "azurerm_role_assignment" "owners_dicom_data_owner" { - count = var.deploy_dicom && var.workspace_owners_group_id != "" ? 1 : 0 - scope = azurerm_healthcare_dicom_service.dicom[0].id - role_definition_id = data.azurerm_role_definition.azure_dicom_data_owner.id - principal_id = var.workspace_owners_group_id + # The groups get FHIR data access only after the private endpoint and its DNS records exist. + # On uninstall, Terraform deletes the role before the zone group and the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] } -data "azurerm_role_definition" "azure_fhir_contributor" { - name = "FHIR Data Contributor" -} +resource "azurerm_role_assignment" "owners_dicom_data_owner" { + count = var.deploy_dicom && var.workspace_owners_group_id != "" ? 1 : 0 + scope = azurerm_healthcare_dicom_service.dicom[0].id + role_definition_name = "DICOM Data Owner" + principal_id = var.workspace_owners_group_id -data "azurerm_role_definition" "azure_dicom_data_owner" { - name = "DICOM Data Owner" + # The zone group write succeeds only after Azure links a new DICOM service to the endpoint. + depends_on = [azapi_resource.health_services_dns_zone_group] }