From 48e6981fe0247e328c0bc28c2a50af63fccf6921 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 02:09:24 +0100 Subject: [PATCH 01/23] Add a private Gitea lifecycle check (#5141) The Gitea selection ran only the generic install case. It did not use the app, its file share or its database. test_gitea.py creates a workspace, a Guacamole service, a Windows VM and a Gitea service. The VM gets a temporary Key Vault Secrets User role on the admin password secret. Through Run Command, the VM makes sure that the Gitea host is private and that Gitea rejects a request without credentials and with a wrong password. It then commits a file to a private repository. After an update and an app restart, the VM reads the file again and deletes the repository. At the end, the case reads the web app, MySQL and storage network settings through Azure. The generic install case stays as a related test. --- .github/workflows/deploy_tre_reusable.yml | 5 + CHANGELOG.md | 1 + docs/tre-developers/end-to-end-tests.md | 12 ++ docs/tre-developers/github-pr-bot-commands.md | 2 +- e2e_tests/bundle_coverage.json | 16 +- e2e_tests/pytest.ini | 1 + e2e_tests/resources/gitea_probe.ps1 | 122 ++++++++++++++++ e2e_tests/resources/gitea_probe.py | 75 ++++++++++ e2e_tests/test_gitea.py | 125 ++++++++++++++++ e2e_tests/unit_tests/test_gitea_probe.py | 137 ++++++++++++++++++ 10 files changed, 492 insertions(+), 4 deletions(-) create mode 100644 e2e_tests/resources/gitea_probe.ps1 create mode 100644 e2e_tests/resources/gitea_probe.py create mode 100644 e2e_tests/test_gitea.py create mode 100644 e2e_tests/unit_tests/test_gitea_probe.py diff --git a/.github/workflows/deploy_tre_reusable.yml b/.github/workflows/deploy_tre_reusable.yml index ffc65f879..5db123799 100644 --- a/.github/workflows/deploy_tre_reusable.yml +++ b/.github/workflows/deploy_tre_reusable.yml @@ -1077,6 +1077,10 @@ jobs: if: inputs.e2eBundle == 'tre-workspace-service-mysql' run: echo "MYSQL_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" + - name: Reserve Gitea cleanup and reporting time + if: inputs.e2eBundle == 'tre-workspace-service-gitea' + run: echo "GITEA_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" + - name: Checkout uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: @@ -1105,6 +1109,7 @@ jobs: TEST_RUN_CERTIFICATE_TESTS_ON_WEEKENDS=${{ inputs.runCertificateTestsOnWeekends }} SQL_VALIDATION_DEADLINE=${{ env.SQL_VALIDATION_DEADLINE }} MYSQL_VALIDATION_DEADLINE=${{ env.MYSQL_VALIDATION_DEADLINE }} + GITEA_VALIDATION_DEADLINE=${{ env.GITEA_VALIDATION_DEADLINE }} TEST_BUNDLE_REQUEST_FILE=e2e_tests/bundle-validation-request.json make test-e2e-bundle DEVCONTAINER_TAG: ${{ inputs.DEVCONTAINER_TAG }} CI_CACHE_ACR_NAME: ${{ secrets.CI_CACHE_ACR_NAME}} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e263cf71..8112dfd01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) +* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update and an app restart. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255)) * Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050)) * Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111)) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index e4f727a0c..22fe8be70 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -186,6 +186,18 @@ The VM reads the administrator password from the workspace Key Vault through a t The MySQL job uses the SQL time limits above with its own `MYSQL_VALIDATION_DEADLINE`. If the case creates Nexus, it removes Nexus after the VM. These checks do not prove previous-version upgrades or researcher access. +### Gitea selection + +Select `tre-workspace-service-gitea` to run `test_gitea.py::test_private_gitea_repository_survives_update`. +The case creates its own workspace, Guacamole service, Windows VM and Gitea service. +It gives the VM identity a temporary Key Vault Secrets User role on the secret that holds the Gitea admin password. +From the VM, the case makes sure that the Gitea host resolves to a private address and that the Gitea API is available. +It also makes sure that Gitea rejects a repository create without credentials (HTTP 401) and a request with a wrong password (HTTP 401). +The VM then creates a private repository with the admin account and commits a file. An anonymous read of the repository gets HTTP 404. +An update changes the display name, and the case restarts the web app. The VM then reads the file again and deletes the repository. +At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. +The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. + ### Run the branch workflow before merge 1. Push the reviewed changes to a branch in `microsoft/AzureTRE`. diff --git a/docs/tre-developers/github-pr-bot-commands.md b/docs/tre-developers/github-pr-bot-commands.md index 9722956fe..da6f973b5 100644 --- a/docs/tre-developers/github-pr-bot-commands.md +++ b/docs/tre-developers/github-pr-bot-commands.md @@ -22,7 +22,7 @@ The table describes the selected assertions. A passing command does not prove ev | `/test-airlock` | Import requests, access checks, the import review VM flow and export request storage routing | No export review VM or migration proof. | | `/test-backups` | Base workspace deployment with backups enabled and disabled | No backup restore or retention proof. | -The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation` and `mysql_validation` markers have no dedicated slash command. +The `workspace_services`, `linux_vm`, `airlock_validation`, `sql_validation`, `mysql_validation` and `gitea_validation` markers have no dedicated slash command. Use the branch workflow or the local test runner for these selections. The `airlock_validation` marker selects two Airlock cases. It does not select the full `airlock` suite. See [bundle selection](end-to-end-tests.md#validate-one-bundle) for exact cases and declared coverage gaps. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index 927319df1..16907481a 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -229,18 +229,28 @@ "manifest": "templates/workspace_services/gitea/porter.yaml", "family": "workspace_service", "prerequisites": [ - "tre-workspace-base" + "tre-workspace-base", + "tre-service-guacamole", + "tre-service-guacamole-windowsvm" ], "tests": [ + "test_gitea.py::test_private_gitea_repository_survives_update" + ], + "related_tests": [ "test_workspace_services.py::test_install_workspace_service[tre-workspace-service-gitea]" ], - "related_tests": [], "coverage": [ "install", + "private Gitea API through private DNS", + "anonymous and wrong-password requests denied", + "private repository write/read with the admin account", + "update and app restart with retained repository data", + "public network access disabled", "uninstall" ], "limitations": [ - "Functional, upgrade and custom-action coverage must be assessed separately." + "AzureCloud only. Requires permission to run commands on a test-created workspace VM and create a temporary secret-scoped role assignment.", + "The case uses the admin account, not OpenID sign-in or researcher identities. The update uses the current bundle version. Previous-version migration remains separate (#5128)." ], "unavailable_reason": "" }, diff --git a/e2e_tests/pytest.ini b/e2e_tests/pytest.ini index 6613a900a..ea391b6c8 100644 --- a/e2e_tests/pytest.ini +++ b/e2e_tests/pytest.ini @@ -18,6 +18,7 @@ markers = openai_validation: opt-in private OpenAI deployment with model and quota preflight sql_validation: Private SQL query and configuration upgrade lifecycle mysql_validation: opt-in private MySQL query and SKU change lifecycle from a Nexus-backed Linux VM + gitea_validation: opt-in private Gitea repository write/read across an update and an app restart asyncio_mode = auto asyncio_default_fixture_loop_scope = session diff --git a/e2e_tests/resources/gitea_probe.ps1 b/e2e_tests/resources/gitea_probe.ps1 new file mode 100644 index 000000000..6fba744ba --- /dev/null +++ b/e2e_tests/resources/gitea_probe.ps1 @@ -0,0 +1,122 @@ +param( + [Parameter(Mandatory)][ValidatePattern('^https://gitea-[a-z0-9-]+\.azurewebsites\.net$')][string]$GiteaUrl, + [Parameter(Mandatory)][ValidatePattern('^[a-z0-9-]+$')][string]$VaultName, + [Parameter(Mandatory)][ValidatePattern('^[a-z0-9-]+$')][string]$SecretName, + [Parameter(Mandatory)][ValidatePattern('^[a-f0-9]{32}$')][string]$ProbeId, + [Parameter(Mandatory)][ValidateSet('write', 'read')][string]$Phase +) +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' +# The Gitea web app accepts only TLS 1.3. +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13 +$stage = 'private-dns' +$api = "$GiteaUrl/api/v1" +$admin = 'giteaadmin' +$repository = "$api/repos/$admin/e2e-$ProbeId" + +function Assert-PrivateDns([string]$Url) { + $addresses = [System.Net.Dns]::GetHostAddresses(([uri]$Url).Host) + if ($addresses.Count -eq 0) { throw 'No address' } + foreach ($address in $addresses) { + $bytes = $address.GetAddressBytes() + $private = $bytes.Length -eq 4 -and ( + $bytes[0] -eq 10 -or + ($bytes[0] -eq 172 -and $bytes[1] -ge 16 -and $bytes[1] -le 31) -or + ($bytes[0] -eq 192 -and $bytes[1] -eq 168) + ) + if (-not $private) { throw 'DNS is not private' } + } +} + +function Get-Status([string]$Method, [string]$Uri, [hashtable]$Headers = @{}, [string]$Body = '') { + $request = @{UseBasicParsing = $true; Method = $Method; Uri = $Uri; Headers = $Headers; TimeoutSec = 30} + if ($Body) { + $request.Body = $Body + $request.ContentType = 'application/json' + } + try { + return [int](Invoke-WebRequest @request).StatusCode + } catch { + # Windows PowerShell and PowerShell 7 use different exception types, but both have a Response. + $response = $_.Exception.Response + if ($null -eq $response) { throw } + return [int]$response.StatusCode + } +} + +function Wait-Ok([string]$Uri, [hashtable]$Headers = @{}) { + # The app can take some minutes to start, and its start script creates the admin user after that. + $deadline = [DateTime]::UtcNow.AddMinutes(15) + while ($true) { + try { + if ((Get-Status 'GET' $Uri $Headers) -eq 200) { return } + } catch { + # A connection error during a restart is the same as not ready. + } + if ([DateTime]::UtcNow -ge $deadline) { throw 'Not ready' } + Start-Sleep -Seconds 20 + } +} + +function Get-Password { + # A new role assignment can take some minutes to apply. + $deadline = [DateTime]::UtcNow.AddMinutes(10) + while ($true) { + try { + $identity = Invoke-RestMethod -Headers @{Metadata = 'true'} -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net' -TimeoutSec 15 + $secret = Invoke-RestMethod -Headers @{Authorization = "Bearer $($identity.access_token)"} -Uri "https://$VaultName.vault.azure.net/secrets/${SecretName}?api-version=7.4" -TimeoutSec 15 + return $secret.value + } catch { + if ([DateTime]::UtcNow -ge $deadline) { throw 'Secret access did not become ready' } + Start-Sleep -Seconds 20 + } + } +} + +function Get-AdminHeaders([string]$Password) { + $pair = [Text.Encoding]::UTF8.GetBytes("${admin}:$Password") + return @{Authorization = 'Basic ' + [Convert]::ToBase64String($pair)} +} + +try { + Assert-PrivateDns $GiteaUrl + $result = [ordered]@{phase = $Phase; probe = $ProbeId; private_dns = $true} + $stage = 'gitea-ready' + Wait-Ok "$api/version" + $stage = 'secret-access' + $headers = Get-AdminHeaders (Get-Password) + $stage = 'gitea-admin' + Wait-Ok "$api/user" $headers + if ($Phase -eq 'write') { + $stage = 'gitea-anonymous' + $result.anonymous_create = Get-Status 'POST' "$api/user/repos" @{} (@{name = "e2e-$ProbeId"} | ConvertTo-Json) + $stage = 'gitea-wrong-password' + $result.wrong_password = Get-Status 'GET' "$api/user" (Get-AdminHeaders ([guid]::NewGuid().ToString())) + $stage = 'gitea-write' + $body = @{name = "e2e-$ProbeId"; private = $true; auto_init = $true} | ConvertTo-Json + $created = Invoke-RestMethod -Method Post -Headers $headers -ContentType 'application/json' -Uri "$api/user/repos" -Body $body -TimeoutSec 30 + if ($created.name -ne "e2e-$ProbeId") { throw 'Unexpected repository' } + $content = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($ProbeId)) + $file = @{content = $content; message = 'E2E probe'} | ConvertTo-Json + $null = Invoke-RestMethod -Method Post -Headers $headers -ContentType 'application/json' -Uri "$repository/contents/probe.txt" -Body $file -TimeoutSec 30 + $stage = 'gitea-private' + $result.anonymous_read = Get-Status 'GET' $repository + } + $stage = 'gitea-read' + $text = Invoke-RestMethod -Headers $headers -Uri "$repository/raw/probe.txt" -TimeoutSec 30 + $result.content = ([string]$text).Trim() -eq $ProbeId + if ($Phase -eq 'read') { + $stage = 'gitea-delete' + $null = Invoke-RestMethod -Method Delete -Headers $headers -Uri $repository -TimeoutSec 30 + $result.deleted = (Get-Status 'GET' $repository $headers) -eq 404 + } + Write-Output ('GITEA_PROBE_RESULT=' + ($result | ConvertTo-Json -Compress)) +} catch { + # Do not emit exception details, passwords or tokens. + Write-Output "GITEA_PROBE_FAILED=$stage" + exit 1 +} finally { + $identity = $null + $secret = $null + $headers = $null +} diff --git a/e2e_tests/resources/gitea_probe.py b/e2e_tests/resources/gitea_probe.py new file mode 100644 index 000000000..d7dea66bb --- /dev/null +++ b/e2e_tests/resources/gitea_probe.py @@ -0,0 +1,75 @@ +"""Run a private Gitea probe from a test-created workspace Windows VM.""" + +import asyncio +import json +from pathlib import Path +from uuid import uuid4 + +from e2e_tests.resources.sql_probe import COMPUTE_API, delete_after + +SCRIPT = Path(__file__).with_name("gitea_probe.ps1") +RESULT = "GITEA_PROBE_RESULT=" +FAILED = "GITEA_PROBE_FAILED=" + + +def expected_result(phase, probe_id): + expected = {"phase": phase, "probe": probe_id, "private_dns": True, "content": True} + if phase == "write": + return {**expected, "anonymous_create": 401, "wrong_password": 401, "anonymous_read": 404} + return {**expected, "deleted": True} + + +def probe_result(resource, phase, probe_id): + properties = resource.get("properties", {}) + view = properties.get("instanceView", {}) + state = view.get("executionState", "") + lines = view.get("output", "").splitlines() + if properties.get("provisioningState") in ("Failed", "Canceled") or state in ("Failed", "Canceled", "TimedOut"): + stages = [line[len(FAILED) :] for line in lines if line.startswith(FAILED)] + raise AssertionError( + f"Gitea probe failed: state={state}, exitCode={view.get('exitCode')}, stage={','.join(stages) or 'unknown'}" + ) + if state != "Succeeded": + return False + if view.get("exitCode") != 0: + raise AssertionError("Gitea probe returned a non-zero exit code") + results = [line[len(RESULT) :] for line in lines if line.startswith(RESULT)] + if len(results) != 1: + raise AssertionError("Gitea probe returned no single result") + actual = json.loads(results[0]) + if actual != expected_result(phase, probe_id): + raise AssertionError(f"Gitea probe result does not match: {actual}") + return True + + +async def run_probe(arm, vm_id, location, *, gitea_url, vault_name, secret_name, probe_id, phase): + parameters = { + "GiteaUrl": gitea_url, + "VaultName": vault_name, + "SecretName": secret_name, + "ProbeId": probe_id, + "Phase": phase, + } + command = f"{vm_id}/runCommands/gitea-probe-{uuid4().hex}" + # Removing our command also terminates it if its deadline expired. + async with delete_after(arm, command, COMPUTE_API): + async with asyncio.timeout(45 * 60): + await arm.request( + "PUT", + command, + COMPUTE_API, + body={ + "location": location, + "properties": { + "source": {"script": SCRIPT.read_text()}, + "parameters": [{"name": k, "value": v} for k, v in parameters.items()], + "timeoutInSeconds": 2400, + "treatFailureAsDeploymentFailure": True, + }, + }, + ) + while True: + result = await arm.request("GET", command, COMPUTE_API, expand=True) + if probe_result(result, phase, probe_id): + return + await asyncio.sleep(10) diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py new file mode 100644 index 000000000..e733aead7 --- /dev/null +++ b/e2e_tests/test_gitea.py @@ -0,0 +1,125 @@ +"""Exercise a private Gitea repository across an update and an app restart.""" + +import asyncio +from functools import partial +import logging +import os +from uuid import uuid4 + +import pytest + +from e2e_tests import config +from e2e_tests.conftest import get_workspace_owner_token +from e2e_tests.resources import strings +from e2e_tests.resources.gitea_probe import run_probe +from e2e_tests.resources.resource import get_resource, post_resource +from e2e_tests.resources.sql_lifecycle import sql_lifecycle +from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, secret_access, validate_vm + +LOGGER = logging.getLogger(__name__) +WEB_API = "2022-09-01" +MYSQL_API = "2023-12-30" +STORAGE_API = "2023-05-01" +GITEA_PAYLOAD = { + "templateName": strings.GITEA_SERVICE, + "properties": { + "display_name": "Gitea lifecycle test", + "description": "Private Gitea repository validation", + "sql_sku": "B | 4GB 2vCores", + }, +} +VM_PAYLOAD = { + "templateName": strings.GUACAMOLE_WINDOWS_USER_RESOURCE, + "properties": { + "display_name": "Gitea test client", + "description": "Temporary private Gitea validation client", + "os_image": "Windows 11", + "vm_size": "2 CPU | 8GB RAM", + "admin_username": "researcher", + "shared_storage_access": False, + **{ + f"install_{tool}": False + for tool in ("azure_cli", "vscode", "storage_explorer", "git", "python_tools", "r_tools") + }, + }, +} + + +@pytest.mark.asyncio(loop_scope="session") +@pytest.mark.gitea_validation +@pytest.mark.timeout(285 * 60) +async def test_private_gitea_repository_survives_update(verify): + if os.environ.get("AZURE_ENVIRONMENT", "AzureCloud") != "AzureCloud": + raise ValueError("The Gitea probe currently supports AzureCloud only") + async with sql_lifecycle(verify, deadline_variable="GITEA_VALIDATION_DEADLINE") as resources: + workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup() + new_token = partial(get_workspace_owner_token, workspace_id, verify) + gitea_path, gitea_id = await resources.create( + GITEA_PAYLOAD, f"/api{workspace_path}/workspace-services", await new_token(), cleanup_token=new_token + ) + vm_path, vm_id = await resources.create( + VM_PAYLOAD, f"/api{guacamole_path}/user-resources", await new_token(), cleanup_token=new_token + ) + token = await new_token() + gitea = (await get_resource(f"/api{gitea_path}", token, verify))["workspaceService"] + vm_properties = (await get_resource(f"/api{vm_path}", token, verify))["userResource"]["properties"] + subscription_id = vm_properties.get("workspace_subscription_id") or os.environ["ARM_SUBSCRIPTION_ID"] + suffix = f"{config.TRE_ID}-ws-{workspace_id[-4:]}-svc-{gitea_id[-4:]}" + site = f"gitea-{suffix}" + gitea_url = f"https://{site}.azurewebsites.net" + assert gitea["properties"]["connection_uri"] == f"{gitea_url}/" + vault = f"kv-{f'{config.TRE_ID}-ws-{workspace_id[-4:]}'[-20:]}".lower() + azure_vm_id = vm_properties["azure_resource_id"] + async with arm_client() as arm: + vm = await arm.request("GET", azure_vm_id, COMPUTE_API) + group = validate_vm(vm, azure_vm_id, subscription_id, config.TRE_ID, workspace_id, guacamole_id, vm_id) + site_id = f"{group}/providers/Microsoft.Web/sites/{site}" + storage_id = f"{group}/providers/Microsoft.Storage/storageAccounts/stgsvc{gitea_id[-4:]}" + # Read the share through Resource Manager, because the storage firewall blocks the file endpoint. + share = await arm.request("GET", f"{storage_id}/fileServices/default/shares/gitea-data", STORAGE_API) + assert share["name"] == "gitea-data" + probe_id = uuid4().hex + secret_name = f"{site}-administrator-password" + vault_id = f"{group}/providers/Microsoft.KeyVault/vaults/{vault}" + async with secret_access(arm, vault_id, secret_name, vm["identity"]["principalId"], subscription_id): + probe = partial( + run_probe, + arm, + azure_vm_id, + vm["location"], + gitea_url=gitea_url, + vault_name=vault, + secret_name=secret_name, + probe_id=probe_id, + ) + await probe(phase="write") + LOGGER.info("Gitea is private, rejects anonymous and wrong-password requests, and stores a commit") + token = await new_token() + gitea = (await get_resource(f"/api{gitea_path}", token, verify))["workspaceService"] + async with asyncio.timeout(60 * 60): + await post_resource( + {"properties": {"display_name": "Gitea lifecycle test (updated)"}}, + f"/api{gitea_path}", + token, + verify, + method="PATCH", + etag=gitea["_etag"], + ) + # A restart makes the app read the repository from the file share and MySQL again. + await arm.request("POST", f"{site_id}/restart", WEB_API) + await probe(phase="read") + LOGGER.info("The Gitea repository survived the update and the restart") + # Check the network settings last, so that a failure here keeps the functional evidence above. + app = await arm.request("GET", site_id, WEB_API) + server = await arm.request( + "GET", f"{group}/providers/Microsoft.DBforMySQL/flexibleServers/mysql-{suffix}", MYSQL_API + ) + storage = await arm.request("GET", storage_id, STORAGE_API) + network = { + "app": app["properties"].get("publicNetworkAccess"), + "mysql": server["properties"]["network"].get("publicNetworkAccess"), + "storage": storage["properties"]["networkAcls"].get("defaultAction"), + } + LOGGER.info("Gitea public network access: %s", network) + expected = {"app": "Disabled", "mysql": "Disabled", "storage": "Deny"} + assert network == expected, f"Gitea services allow public access: {network}" diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py new file mode 100644 index 000000000..d3a099827 --- /dev/null +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -0,0 +1,137 @@ +"""Check the Gitea probe's evidence, payloads and cleanup boundaries.""" + +import json +from pathlib import Path +import re +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock +from uuid import uuid4 + +from jsonschema import validate +import yaml + +from api_app.services.schema_service import enrich_user_resource_template, enrich_workspace_service_template +from e2e_tests import test_gitea as gitea +from e2e_tests.resources import gitea_probe as probe + + +ROOT = Path(__file__).resolve().parents[2] +GROUP = "/subscriptions/sub/resourceGroups/rg-tretest-ws-abcd" +VM_ID = f"{GROUP}/providers/Microsoft.Compute/virtualMachines/windowsvm1234" +GITEA_URL = "https://gitea-tretest-ws-abcd-svc-1234.azurewebsites.net" +PROBE_ID = "0123456789abcdef0123456789abcdef" +PROBE = { + "gitea_url": GITEA_URL, + "vault_name": "kv-tretest-ws-abcd", + "secret_name": "gitea-tretest-ws-abcd-svc-1234-administrator-password", + "probe_id": PROBE_ID, +} + + +def result(evidence, state="Succeeded", code=0, output=None): + return { + "properties": { + "provisioningState": "Succeeded", + "instanceView": { + "executionState": state, + "exitCode": code, + "output": output if output is not None else "GITEA_PROBE_RESULT=" + json.dumps(evidence), + }, + } + } + + +class ProbeEvidenceTests(unittest.TestCase): + def test_write_phase_requires_denied_requests_and_the_commit(self): + evidence = probe.expected_result("write", PROBE_ID) + self.assertEqual(evidence["anonymous_create"], 401) + self.assertEqual(evidence["wrong_password"], 401) + self.assertEqual(evidence["anonymous_read"], 404) + self.assertTrue(probe.probe_result(result(evidence), "write", PROBE_ID)) + for change in ( + {"anonymous_create": 201}, + {"wrong_password": 200}, + {"anonymous_read": 200}, + {"content": False}, + {"private_dns": False}, + ): + with self.subTest(change=change), self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, **change}), "write", PROBE_ID) + + def test_read_phase_requires_the_file_and_the_delete(self): + evidence = probe.expected_result("read", PROBE_ID) + self.assertTrue(probe.probe_result(result(evidence), "read", PROBE_ID)) + for change in ({"content": False}, {"deleted": False}): + with self.subTest(change=change), self.assertRaises(AssertionError): + probe.probe_result(result({**evidence, **change}), "read", PROBE_ID) + with self.assertRaises(AssertionError): + probe.probe_result(result(probe.expected_result("write", PROBE_ID)), "read", PROBE_ID) + + def test_running_command_is_not_a_pass_and_failure_names_the_stage(self): + running = result({}) + running["properties"]["instanceView"] = {} + self.assertFalse(probe.probe_result(running, "write", PROBE_ID)) + failed = result({}, state="Failed", code=1, output="GITEA_PROBE_FAILED=gitea-admin\n") + with self.assertRaisesRegex(AssertionError, "stage=gitea-admin"): + probe.probe_result(failed, "write", PROBE_ID) + + def test_payloads_match_enriched_api_schemas(self): + cases = ( + (gitea.GITEA_PAYLOAD, "gitea", enrich_workspace_service_template), + (gitea.VM_PAYLOAD, "guacamole/user_resources/guacamole-azure-windowsvm", enrich_user_resource_template), + ) + for payload, folder, enrich in cases: + with self.subTest(bundle=payload["templateName"]): + schema = json.loads( + (ROOT / "templates/workspace_services" / folder / "template_schema.json").read_text() + ) + validate(payload["properties"], enrich(SimpleNamespace(model_dump=lambda **_: schema))) + self.assertFalse(any(v for k, v in gitea.VM_PAYLOAD["properties"].items() if k.startswith("install_"))) + + def test_deadline_precedes_container(self): + jobs = yaml.safe_load((ROOT / ".github/workflows/deploy_tre_reusable.yml").read_text())["jobs"] + job = jobs["e2e_tests_bundle"] + reserve = next(s for s in job["steps"] if "GITEA_VALIDATION_DEADLINE" in s.get("run", "")) + run = next(s for s in job["steps"] if s["name"] == "Run E2E Tests") + self.assertLess(job["steps"].index(reserve), job["steps"].index(run)) + self.assertEqual(reserve["if"], "inputs.e2eBundle == 'tre-workspace-service-gitea'") + self.assertIn("GITEA_VALIDATION_DEADLINE=${{ env.GITEA_VALIDATION_DEADLINE }}", run["with"]["COMMAND"]) + + def test_script_accepts_the_names_that_the_test_builds(self): + # The script rejects a parameter that does not match its pattern. + found = re.findall(r"ValidatePattern\('([^']+)'\)\]\[string\]\$(\w+)", probe.SCRIPT.read_text()) + patterns = {name: pattern for pattern, name in found} + values = {**PROBE, "probe_id": uuid4().hex} + for name, key in (("GiteaUrl", "gitea_url"), ("VaultName", "vault_name"), ("SecretName", "secret_name")): + with self.subTest(name=name): + self.assertRegex(values[key], patterns[name]) + self.assertRegex(values["probe_id"], patterns["ProbeId"]) + + def test_script_offers_the_tls_version_that_the_web_app_requires(self): + # A TLS handshake failure leaves no request in the web app logs. + webapp = (ROOT / "templates/workspace_services/gitea/terraform/gitea-webapp.tf").read_text() + version = re.search(r'minimum_tls_version\s*=\s*"1\.(\d)"', webapp).group(1) + self.assertIn(f"[Net.SecurityProtocolType]::Tls1{version}", probe.SCRIPT.read_text()) + + +class ProbeRecoveryTests(unittest.IsolatedAsyncioTestCase): + async def test_failed_command_is_removed(self): + failed = result({}, state="Failed", code=1, output="GITEA_PROBE_FAILED=secret-access\n") + arm = SimpleNamespace(request=AsyncMock(side_effect=[{}, failed]), delete=AsyncMock()) + with self.assertRaisesRegex(AssertionError, "secret-access"): + await probe.run_probe(arm, VM_ID, "switzerlandnorth", **PROBE, phase="write") + command = arm.request.call_args_list[0].args[1] + self.assertTrue(command.startswith(VM_ID + "/runCommands/gitea-probe-")) + arm.delete.assert_awaited_once_with(command, probe.COMPUTE_API) + parameters = arm.request.call_args_list[0].kwargs["body"]["properties"]["parameters"] + self.assertEqual( + {p["name"]: p["value"] for p in parameters}, + { + "GiteaUrl": GITEA_URL, + "VaultName": PROBE["vault_name"], + "SecretName": PROBE["secret_name"], + "ProbeId": PROBE_ID, + "Phase": "write", + }, + ) From eabf48532b7b5c025b78db10a0a0200c7db7c956 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 02:11:11 +0100 Subject: [PATCH 02/23] Create the Gitea file share through Resource Manager (#5141) The Gitea storage account has a private endpoint for its file service. Terraform managed the gitea-data share through the file endpoint of the account, and the share did not depend on the private endpoint. The TRE firewall does not allow the resource processor to reach that endpoint through its public address. A live run of the Gitea test showed the full sequence: 1. Install: Terraform asked the file endpoint for the share at 11:11:03. The private endpoint and its DNS record came at 11:11:20 and 11:11:47. The firewall denied the request, and the install failed with "checking for existing File Share ... EOF". 2. The clean-up of the test ran the bundle again. This time the request went through the private endpoint, so the deployment passed. 3. Delete: Terraform deleted the private endpoint and the share at the same time. The share requests then went to the firewall again, and the delete failed. Each later update or delete failed in the same way. The share now uses storage_account_id, so Terraform manages it through Resource Manager and does not use the file endpoint. In azurerm 4.27.0 this change is an in-place update: the next read moves the share to its Resource Manager ID, so an upgrade keeps the share and its data. The refresh before that update still uses the file endpoint, through the existing private endpoint. The Gitea MySQL server also had public network access enabled, as the MySQL bundle had. It now sets public_network_access to Disabled. In azurerm 4.27.0 this is an in-place update. The Gitea document has the repair steps for a service whose private endpoint is already gone. A contract test makes sure of both settings. --- CHANGELOG.md | 1 + .../tre-templates/workspace-services/gitea.md | 9 +++++++ .../unit_tests/test_gitea_bundle_contract.py | 24 +++++++++++++++++++ .../workspace_services/gitea/porter.yaml | 2 +- .../gitea/terraform/mysql.tf | 4 +++- .../gitea/terraform/storage.tf | 8 ++++--- 6 files changed, 43 insertions(+), 5 deletions(-) create mode 100644 e2e_tests/unit_tests/test_gitea_bundle_contract.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 8112dfd01..2a4474456 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) +* Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index fc3d9f2c8..617b9d7ca 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -28,3 +28,12 @@ The Gitea worskpace service opens outbound access to: ## Upgrading to version 1.0.0 Migrating existing Gitea services to the major version 1.0.0 is not currently supported. This is due to the breaking change in the Terraform to migrate from the deprecated mysql_server to the new mysql_flexible_server. + +## Repairing a service that cannot reach its file share + +Before version 1.3.8, Terraform used the file endpoint of the Gitea storage account to manage the `gitea-data` share. If a delete removed the private endpoint of the account and then failed, each later update or delete of the service fails with `executing request: Get "https://stgsvc.file.core.windows.net/gitea-data?restype=share": EOF`. + +If you want to delete such a service: + +1. Delete the storage account `stgsvc` in the workspace resource group. This deletes the Gitea repositories. +2. Delete the workspace service again. Terraform removes the share from its state, because the account does not exist. diff --git a/e2e_tests/unit_tests/test_gitea_bundle_contract.py b/e2e_tests/unit_tests/test_gitea_bundle_contract.py new file mode 100644 index 000000000..5d530fbdd --- /dev/null +++ b/e2e_tests/unit_tests/test_gitea_bundle_contract.py @@ -0,0 +1,24 @@ +"""Check the private storage and database settings of the Gitea workspace service.""" + +from pathlib import Path +import unittest + +ROOT = Path(__file__).resolve().parents[2] +BUNDLE = ROOT / "templates/workspace_services/gitea/terraform" + + +def block(source, header): + return source.split(header + " {", 1)[1].split("\n}", 1)[0] + + +class GiteaBundleContractTests(unittest.TestCase): + def test_share_uses_resource_manager(self): + # With storage_account_name, the provider uses the file endpoint. The TRE firewall blocks it + # before the private endpoint exists and after Terraform deletes the private endpoint. + share = block((BUNDLE / "storage.tf").read_text(), 'resource "azurerm_storage_share" "gitea"') + self.assertRegex(share, r"\n storage_account_id\s*=\s*azurerm_storage_account\.gitea\.id\n") + self.assertNotIn("storage_account_name", share) + + def test_server_disables_public_network_access(self): + server = block((BUNDLE / "mysql.tf").read_text(), 'resource "azurerm_mysql_flexible_server" "gitea"') + self.assertRegex(server, r'\n public_network_access\s*=\s*"Disabled"\n') diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index 7548ffd34..379186665 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.7 +version: 1.3.8 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/terraform/mysql.tf b/templates/workspace_services/gitea/terraform/mysql.tf index bd823448c..bc5b1fd8d 100644 --- a/templates/workspace_services/gitea/terraform/mysql.tf +++ b/templates/workspace_services/gitea/terraform/mysql.tf @@ -16,7 +16,9 @@ resource "azurerm_mysql_flexible_server" "gitea" { version = "8.0.21" backup_retention_days = 7 geo_redundant_backup_enabled = false - tags = local.workspace_service_tags + # Clients connect through the private endpoint only. + public_network_access = "Disabled" + tags = local.workspace_service_tags lifecycle { ignore_changes = [tags, zone] } } diff --git a/templates/workspace_services/gitea/terraform/storage.tf b/templates/workspace_services/gitea/terraform/storage.tf index 0e74f1688..fbd6eecf1 100644 --- a/templates/workspace_services/gitea/terraform/storage.tf +++ b/templates/workspace_services/gitea/terraform/storage.tf @@ -61,8 +61,10 @@ resource "azurerm_private_endpoint" "stgfilepe" { } +# Resource Manager manages the share, so Terraform does not use the file endpoint of the account. +# The TRE firewall blocks that endpoint before the private endpoint exists and after Terraform deletes it. resource "azurerm_storage_share" "gitea" { - name = "gitea-data" - storage_account_name = azurerm_storage_account.gitea.name - quota = var.gitea_storage_limit + name = "gitea-data" + storage_account_id = azurerm_storage_account.gitea.id + quota = var.gitea_storage_limit } From 7e953b148adff6ef5ba770c9212b3c5ddc7ca4a7 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 18:47:20 +0100 Subject: [PATCH 03/23] Keep the Gitea workspace repositories on the file share The Gitea workspace service mounts its file share at /data/gitea/, but Gitea kept the Git repositories in its image default /data/git/repositories. That folder is on the container disk, so each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. A live E2E run showed this. The probe committed a file, the test updated the service and restarted the app, and Gitea then logged "RepoRef Invalid repo /data/git/repositories/...: no such file or directory". Set GITEA__repository__ROOT to a folder on the share. The Gitea image writes GITEA__ settings to app.ini at each start, so an upgrade also moves existing services. The upgrade restarts the app, so the document tells operators to mirror their repositories first. --- CHANGELOG.md | 1 + docs/tre-templates/workspace-services/gitea.md | 14 +++++++++++++- e2e_tests/unit_tests/test_gitea_probe.py | 9 ++++++++- templates/workspace_services/gitea/porter.yaml | 2 +- .../gitea/terraform/gitea-webapp.tf | 1 + 5 files changed, 24 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a4474456..705374579 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,7 @@ BUG FIXES: * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 617b9d7ca..1ab9473ee 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -29,11 +29,23 @@ The Gitea worskpace service opens outbound access to: Migrating existing Gitea services to the major version 1.0.0 is not currently supported. This is due to the breaking change in the Terraform to migrate from the deprecated mysql_server to the new mysql_flexible_server. +## Upgrading to version 1.3.9 + +Before version 1.3.9, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 1.3.9, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. + +To keep the repositories, do these steps from a workspace VM: + +1. Clone each repository that you want to keep with `git clone --mirror `. +2. Upgrade the workspace service. +3. In Gitea, open Site Administration, then Dashboard. Run "Delete all repositories missing their Git files". +4. Create each repository again in Gitea. +5. Push each mirror into its new repository with `git push --mirror `. + ## Repairing a service that cannot reach its file share Before version 1.3.8, Terraform used the file endpoint of the Gitea storage account to manage the `gitea-data` share. If a delete removed the private endpoint of the account and then failed, each later update or delete of the service fails with `executing request: Get "https://stgsvc.file.core.windows.net/gitea-data?restype=share": EOF`. If you want to delete such a service: -1. Delete the storage account `stgsvc` in the workspace resource group. This deletes the Gitea repositories. +1. Delete the storage account `stgsvc` in the workspace resource group. This deletes the Gitea data on the share. 2. Delete the workspace service again. Terraform removes the share from its state, because the account does not exist. diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py index d3a099827..d234bf291 100644 --- a/e2e_tests/unit_tests/test_gitea_probe.py +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -1,7 +1,7 @@ """Check the Gitea probe's evidence, payloads and cleanup boundaries.""" import json -from pathlib import Path +from pathlib import Path, PurePosixPath import re from types import SimpleNamespace import unittest @@ -114,6 +114,13 @@ def test_script_offers_the_tls_version_that_the_web_app_requires(self): version = re.search(r'minimum_tls_version\s*=\s*"1\.(\d)"', webapp).group(1) self.assertIn(f"[Net.SecurityProtocolType]::Tls1{version}", probe.SCRIPT.read_text()) + def test_repositories_are_on_the_file_share(self): + # The read phase runs after a restart, and a restart deletes the data outside the share. + webapp = (ROOT / "templates/workspace_services/gitea/terraform/gitea-webapp.tf").read_text() + mount = re.search(r'mount_path\s*=\s*"([^"]+)"', webapp).group(1) + repositories = re.search(r'GITEA__repository__ROOT\s*=\s*"([^"]+)"', webapp).group(1) + self.assertIn(PurePosixPath(mount), PurePosixPath(repositories).parents) + class ProbeRecoveryTests(unittest.IsolatedAsyncioTestCase): async def test_failed_command_is_removed(self): diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index 379186665..37cac47e2 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.8 +version: 1.3.9 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/terraform/gitea-webapp.tf b/templates/workspace_services/gitea/terraform/gitea-webapp.tf index 6673ddc5a..30c0ed125 100644 --- a/templates/workspace_services/gitea/terraform/gitea-webapp.tf +++ b/templates/workspace_services/gitea/terraform/gitea-webapp.tf @@ -47,6 +47,7 @@ resource "azurerm_linux_web_app" "gitea" { GITEA__server__ROOT_URL = "https://${local.webapp_name}.azurewebsites.net/" GITEA__server__LFS_START_SERVER = "true" GITEA__server__OFFLINE_MODE = true + GITEA__repository__ROOT = "/data/gitea/repositories" # The image default /data/git is on the container disk, and a restart deletes it. GITEA__lfs__PATH = "/data/gitea/lfs" GITEA__lfs__STORAGE_TYPE = "local" GITEA__log_0x2E_console__COLORIZE = "false" # Azure monitor doens't show colors, so this is easier to read. From 960f8bb56a42fcc4a82fafeb123758e478245c01 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:43:38 +0000 Subject: [PATCH 04/23] Wrap a long CHANGELOG entry The Markdown lint rejects a line with more than 400 characters. The entry now continues on lines that start with two spaces, as an earlier entry in the file does. The rendered text does not change. --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 705374579..0eb87560b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,7 +44,8 @@ BUG FIXES: * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) -* Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. + Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) From f1f342defc426351f23817e8fdb1445bd88e6101 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 20:15:21 +0100 Subject: [PATCH 05/23] Stop writing Gitea secrets to the console log The web app of the Gitea workspace service sends its console output to Log Analytics. Its start script printed the admin user and OIDC commands, with the admin password and the client secret of the workspace application. Gitea also prints its full command line when a command fails. The admin user create fails at each restart, because the user exists, so each restart printed the password again. The admin user create in both Gitea bundles also made an access token, and Gitea printed the token. The admin commands now run in configure_gitea_admin.sh as the gitea user. The script hides the secret values in all Gitea output, creates the admin user only if it does not exist, and makes no access token. It updates an existing OIDC source, so that a new client secret gets to Gitea at the next start. Before, Gitea kept the first secret. The web app reads the client secret through a Key Vault reference, so the value is not in the app settings or in the Terraform state. A local test ran the start script three times in the Gitea 1.17.3 image with dummy values. The output had no secret value, the second run updated the OIDC secret, and a failed user create showed --password=***. The shared-service image only drops --access-token and quotes its values. --- CHANGELOG.md | 1 + .../tre-templates/workspace-services/gitea.md | 15 +++++ .../unit_tests/test_gitea_bundle_contract.py | 62 ++++++++++++++++++- .../shared_services/gitea/docker/Dockerfile | 3 +- .../shared_services/gitea/docker/version.txt | 2 +- templates/shared_services/gitea/porter.yaml | 2 +- .../gitea/docker/configure_gitea.sh | 9 +-- .../gitea/docker/configure_gitea_admin.sh | 36 +++++++++++ .../workspace_services/gitea/porter.yaml | 2 +- .../gitea/terraform/data.tf | 5 -- .../gitea/terraform/gitea-webapp.tf | 2 +- .../workspace_services/gitea/version.txt | 2 +- 12 files changed, 122 insertions(+), 19 deletions(-) create mode 100755 templates/workspace_services/gitea/docker/configure_gitea_admin.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 0eb87560b..971416cad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,7 @@ BUG FIXES: * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 1ab9473ee..231feb179 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -41,6 +41,21 @@ To keep the repositories, do these steps from a workspace VM: 4. Create each repository again in Gitea. 5. Push each mirror into its new repository with `git push --mirror `. +## Upgrading to version 1.3.10 + +Before version 1.3.10, the Gitea workspace service wrote secrets to the console log of its web app. The web app sends this log to Log Analytics, so all people who can read the Log Analytics workspace of the TRE can read these secrets: + +- The client secret of the workspace application. +- The password of the Gitea admin user. +- An access token of the Gitea admin user. + +If a Gitea workspace service ran an earlier version, do these steps after the upgrade: + +1. Rotate the client secret of the workspace application. Make sure that the workspace Key Vault secret `workspace-client-secret` has the new value. +2. Restart the Gitea web app. At each start, Gitea gives its OIDC source the client secret from the Key Vault. +3. Sign in to Gitea as the admin user. In the user settings, open Applications and delete the access tokens. +4. If the people who can read Log Analytics must not get admin access to Gitea, change the password of the admin user in Gitea. + ## Repairing a service that cannot reach its file share Before version 1.3.8, Terraform used the file endpoint of the Gitea storage account to manage the `gitea-data` share. If a delete removed the private endpoint of the account and then failed, each later update or delete of the service fails with `executing request: Get "https://stgsvc.file.core.windows.net/gitea-data?restype=share": EOF`. diff --git a/e2e_tests/unit_tests/test_gitea_bundle_contract.py b/e2e_tests/unit_tests/test_gitea_bundle_contract.py index 5d530fbdd..a2535abbc 100644 --- a/e2e_tests/unit_tests/test_gitea_bundle_contract.py +++ b/e2e_tests/unit_tests/test_gitea_bundle_contract.py @@ -1,10 +1,18 @@ -"""Check the private storage and database settings of the Gitea workspace service.""" +"""Check the private storage, database and secret settings of the Gitea services.""" +import os from pathlib import Path +import re +import subprocess +import tempfile import unittest ROOT = Path(__file__).resolve().parents[2] BUNDLE = ROOT / "templates/workspace_services/gitea/terraform" +START_SCRIPT = ROOT / "templates/workspace_services/gitea/docker/configure_gitea.sh" +ADMIN_SCRIPT = ROOT / "templates/workspace_services/gitea/docker/configure_gitea_admin.sh" +SHARED_DOCKERFILE = ROOT / "templates/shared_services/gitea/docker/Dockerfile" +SECRETS = ("GITEA_PASSWD", "GITEA_OPENID_CLIENT_SECRET") def block(source, header): @@ -22,3 +30,55 @@ def test_share_uses_resource_manager(self): def test_server_disables_public_network_access(self): server = block((BUNDLE / "mysql.tf").read_text(), 'resource "azurerm_mysql_flexible_server" "gitea"') self.assertRegex(server, r'\n public_network_access\s*=\s*"Disabled"\n') + + def test_start_script_prints_no_secret(self): + # The web app sends the console output to Log Analytics. + for script in (START_SCRIPT, ADMIN_SCRIPT): + for line in script.read_text().splitlines(): + if line.lstrip().startswith(("echo", "printf")) or "set -x" in line: + for secret in SECRETS: + with self.subTest(script=script.name, line=line, secret=secret): + self.assertNotIn(secret, line) + + def test_admin_script_hides_secrets_in_gitea_output(self): + # Gitea prints its full command line when a command fails. + secrets = { + "GITEA_PASSWD": "pass$word[1]", + "GITEA_OPENID_CLIENT_SECRET": "client*secret", + "GITEA__database__PASSWD": "db.secret", + } + with tempfile.TemporaryDirectory() as folder: + stub = Path(folder) / "gitea" + stub.write_text('#!/bin/bash\necho "[F] Failed to run app with [gitea $*]"\nexit 1\n') + stub.chmod(0o755) + environment = { + **secrets, + "PATH": f"{folder}{os.pathsep}{os.environ['PATH']}", + "GITEA_USERNAME": "giteaadmin", + "GITEA_EMAIL": "giteaadmin@example.com", + "GITEA_OPENID_CLIENT_ID": "client-id", + "GITEA_OPENID_AUTHORITY": "https://login.example.com/tenant/v2.0", + } + result = subprocess.run( + ["bash", str(ADMIN_SCRIPT)], env=environment, capture_output=True, text=True, check=False + ) + output = result.stdout + result.stderr + for name, value in secrets.items(): + with self.subTest(secret=name): + self.assertNotIn(value, output) + self.assertIn("user create --admin --username=giteaadmin --password=***", output) + self.assertIn("add-oauth --name oidc --provider openidConnect --key client-id --secret ***", output) + + def test_admin_create_makes_no_access_token(self): + # Gitea prints a new access token to the console output. + for source in (START_SCRIPT, SHARED_DOCKERFILE): + commands = [line for line in source.read_text().splitlines() if not line.lstrip().startswith("#")] + with self.subTest(source=source.name): + self.assertNotIn("--access-token", "\n".join(commands)) + + def test_openid_secret_is_a_key_vault_reference(self): + # A plain value is visible in the app settings and in the Terraform state. + webapp = (BUNDLE / "gitea-webapp.tf").read_text() + value = re.search(r"\n GITEA_OPENID_CLIENT_SECRET\s*=\s*(.+)\n", webapp).group(1) + self.assertRegex(value, r'^"@Microsoft\.KeyVault\(SecretUri=.+secrets/workspace-client-secret\)"$') + self.assertNotIn('data "azurerm_key_vault_secret" "client_secret"', (BUNDLE / "data.tf").read_text()) diff --git a/templates/shared_services/gitea/docker/Dockerfile b/templates/shared_services/gitea/docker/Dockerfile index 16b4fd3bf..b9e002fe1 100644 --- a/templates/shared_services/gitea/docker/Dockerfile +++ b/templates/shared_services/gitea/docker/Dockerfile @@ -5,4 +5,5 @@ FROM gitea/gitea:${GITEA_TAG} # need to pass args to stage ARG CERTIFICATE_URL RUN wget -nv -O /usr/local/share/ca-certificates/mysql.crt.pem ${CERTIFICATE_URL} && update-ca-certificates -ENTRYPOINT ["/bin/bash", "-c", "sleep 120 && gitea admin user create --admin --access-token --username=${GITEA_USERNAME} --password=${GITEA_PASSWD} --email=${GITEA_EMAIL} --must-change-password=false & /usr/bin/entrypoint"] +# Do not add --access-token. Gitea prints the new token to the console log. +ENTRYPOINT ["/bin/bash", "-c", "sleep 120 && gitea admin user create --admin --username=\"${GITEA_USERNAME}\" --password=\"${GITEA_PASSWD}\" --email=\"${GITEA_EMAIL}\" --must-change-password=false & /usr/bin/entrypoint"] diff --git a/templates/shared_services/gitea/docker/version.txt b/templates/shared_services/gitea/docker/version.txt index df0ed3321..8a3be2e00 100644 --- a/templates/shared_services/gitea/docker/version.txt +++ b/templates/shared_services/gitea/docker/version.txt @@ -1 +1 @@ -__version__ = "0.3.12" +__version__ = "0.3.13" diff --git a/templates/shared_services/gitea/porter.yaml b/templates/shared_services/gitea/porter.yaml index 00d301cb7..3caacf023 100644 --- a/templates/shared_services/gitea/porter.yaml +++ b/templates/shared_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-shared-service-gitea -version: 1.2.5 +version: 1.2.6 description: "A Gitea shared service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/docker/configure_gitea.sh b/templates/workspace_services/gitea/docker/configure_gitea.sh index 9941fcef9..d5a3af772 100755 --- a/templates/workspace_services/gitea/docker/configure_gitea.sh +++ b/templates/workspace_services/gitea/docker/configure_gitea.sh @@ -11,10 +11,5 @@ if [ -z "$GITEA_USERNAME" ]; then sleep 60 fi -echo "Adding admin user" -echo "gitea admin user create --admin --access-token --username='${GITEA_USERNAME}' --password='${GITEA_PASSWD}' --email='${GITEA_EMAIL}' --must-change-password=false" -su gitea -c "gitea admin user create --admin --access-token --username='${GITEA_USERNAME}' --password='${GITEA_PASSWD}' --email='${GITEA_EMAIL}' --must-change-password=false" - -echo "Configuring OIDC" -echo "gitea admin auth add-oauth --name oidc --provider openidConnect --key '${GITEA_OPENID_CLIENT_ID}' --secret '${GITEA_OPENID_CLIENT_SECRET}' --auto-discover-url '${GITEA_OPENID_AUTHORITY}/.well-known/openid-configuration' --group-claim-name 'roles' --admin-group 'WorkspaceOwner'" -su gitea -c "gitea admin auth add-oauth --name oidc --provider openidConnect --key '${GITEA_OPENID_CLIENT_ID}' --secret '${GITEA_OPENID_CLIENT_SECRET}' --auto-discover-url '${GITEA_OPENID_AUTHORITY}/.well-known/openid-configuration' --group-claim-name 'roles' --admin-group 'WorkspaceOwner'" +# The admin commands contain secrets. The script hides them in the Gitea output. +su gitea -s /bin/bash -c "bash /configure_gitea_admin.sh" diff --git a/templates/workspace_services/gitea/docker/configure_gitea_admin.sh b/templates/workspace_services/gitea/docker/configure_gitea_admin.sh new file mode 100755 index 000000000..5e9d4d10e --- /dev/null +++ b/templates/workspace_services/gitea/docker/configure_gitea_admin.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# configure_gitea.sh runs this script as the gitea user. The web app sends its output to Log Analytics. + +# Gitea prints the full command line when a command fails, so hide the secrets in its output. +gitea_cli() { + local output status secret + output=$(gitea "$@" 2>&1) + status=$? + for secret in "$GITEA_PASSWD" "$GITEA_OPENID_CLIENT_SECRET" "${GITEA__database__PASSWD:-}"; do + if [ -n "$secret" ]; then + output=${output//"$secret"/"***"} + fi + done + if [ -n "$output" ]; then + printf '%s\n' "$output" + fi + return "$status" +} + +echo "Adding admin user" +if gitea_cli admin user list --admin | awk 'NR > 1 { print $2 }' | grep -qxF "$GITEA_USERNAME"; then + echo "The admin user exists" +else + gitea_cli admin user create --admin --username="$GITEA_USERNAME" --password="$GITEA_PASSWD" --email="$GITEA_EMAIL" --must-change-password=false +fi + +echo "Configuring OIDC" +oidc=(--name oidc --provider openidConnect --key "$GITEA_OPENID_CLIENT_ID" --secret "$GITEA_OPENID_CLIENT_SECRET" + --auto-discover-url "$GITEA_OPENID_AUTHORITY/.well-known/openid-configuration" --group-claim-name roles --admin-group WorkspaceOwner) +source_id=$(gitea_cli admin auth list | awk '$2 == "oidc" { print $1 }') +# Update an existing source, so that it gets the current values, for example a new client secret. +if [ -z "$source_id" ]; then + gitea_cli admin auth add-oauth "${oidc[@]}" +else + gitea_cli admin auth update-oauth --id "$source_id" "${oidc[@]}" +fi diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index 37cac47e2..735e18fdb 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.9 +version: 1.3.10 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/terraform/data.tf b/templates/workspace_services/gitea/terraform/data.tf index eb24688f2..6f2fb84f2 100644 --- a/templates/workspace_services/gitea/terraform/data.tf +++ b/templates/workspace_services/gitea/terraform/data.tf @@ -63,11 +63,6 @@ data "azurerm_key_vault_secret" "client_id" { key_vault_id = data.azurerm_key_vault.ws.id } -data "azurerm_key_vault_secret" "client_secret" { - name = "workspace-client-secret" - key_vault_id = data.azurerm_key_vault.ws.id -} - data "azurerm_monitor_diagnostic_categories" "gitea" { resource_id = azurerm_linux_web_app.gitea.id depends_on = [ diff --git a/templates/workspace_services/gitea/terraform/gitea-webapp.tf b/templates/workspace_services/gitea/terraform/gitea-webapp.tf index 30c0ed125..db23f9461 100644 --- a/templates/workspace_services/gitea/terraform/gitea-webapp.tf +++ b/templates/workspace_services/gitea/terraform/gitea-webapp.tf @@ -42,7 +42,7 @@ resource "azurerm_linux_web_app" "gitea" { GITEA_PASSWD = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault_secret.gitea_password.id})" GITEA_EMAIL = "giteaadmin@azuretre.com" GITEA_OPENID_CLIENT_ID = data.azurerm_key_vault_secret.client_id.value - GITEA_OPENID_CLIENT_SECRET = data.azurerm_key_vault_secret.client_secret.value + GITEA_OPENID_CLIENT_SECRET = "@Microsoft.KeyVault(SecretUri=${data.azurerm_key_vault.ws.vault_uri}secrets/workspace-client-secret)" GITEA_OPENID_AUTHORITY = local.gitea_openid_auth GITEA__server__ROOT_URL = "https://${local.webapp_name}.azurewebsites.net/" GITEA__server__LFS_START_SERVER = "true" diff --git a/templates/workspace_services/gitea/version.txt b/templates/workspace_services/gitea/version.txt index 22049ab2c..63af88769 100644 --- a/templates/workspace_services/gitea/version.txt +++ b/templates/workspace_services/gitea/version.txt @@ -1 +1 @@ -__version__ = "0.6.2" +__version__ = "0.6.3" From 6345ac57da6138a09f0432ba89e9047f970c573e Mon Sep 17 00:00:00 2001 From: Jay W Date: Sat, 24 Oct 2026 22:50:35 +0100 Subject: [PATCH 06/23] Make sure that the Gitea console log shows no secret The Gitea E2E case now ends with two more checks before its network checks: - The web app resolves its Key Vault references for the admin password and the OIDC client secret. Azure returns the status of a reference, not its value. - A Log Analytics query counts the start script lines and the lines that show a password, a client secret or a new access token in the web app console log. The log must have start lines and no such line. The query returns counts only, and it waits up to 15 minutes for the lines to arrive. On the Gitea site of an earlier run without the fix, the query found 2 start lines and 10 lines with a secret. --- CHANGELOG.md | 2 +- docs/tre-developers/end-to-end-tests.md | 3 ++ e2e_tests/resources/gitea_secrets.py | 50 ++++++++++++++++++++++++ e2e_tests/test_gitea.py | 11 ++++++ e2e_tests/unit_tests/test_gitea_probe.py | 45 ++++++++++++++++++++- 5 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 e2e_tests/resources/gitea_secrets.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 971416cad..ebb6c6a23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) -* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update and an app restart. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update and an app restart. The check also makes sure that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255)) * Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050)) * Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111)) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 22fe8be70..6c09e1e2e 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -195,6 +195,9 @@ From the VM, the case makes sure that the Gitea host resolves to a private addre It also makes sure that Gitea rejects a repository create without credentials (HTTP 401) and a request with a wrong password (HTTP 401). The VM then creates a private repository with the admin account and commits a file. An anonymous read of the repository gets HTTP 404. An update changes the display name, and the case restarts the web app. The VM then reads the file again and deletes the repository. +Then the case makes sure that the web app resolves its two Key Vault references, for the admin password and the OIDC client secret. +It also counts lines in the web app console log in the TRE Log Analytics workspace. The log must have the start script lines, and no line may show a password, a client secret or a new access token. +The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. diff --git a/e2e_tests/resources/gitea_secrets.py b/e2e_tests/resources/gitea_secrets.py new file mode 100644 index 000000000..d181fec24 --- /dev/null +++ b/e2e_tests/resources/gitea_secrets.py @@ -0,0 +1,50 @@ +"""Make sure that the Gitea web app keeps its secrets out of its console log.""" + +import asyncio +import re + +LOGS = "https://api.loganalytics.io" +LOG_API = "2022-10-01" +WEB_API = "2022-09-01" +SECRET_SETTINGS = ("GITEA_PASSWD", "GITEA_OPENID_CLIENT_SECRET") +SITE_ID = re.compile(r"^/subscriptions/[0-9a-f-]+/resourceGroups/[\w.-]+/providers/Microsoft\.Web/sites/[\w-]+$") +# The start script hides the secret values as ***. A line that shows a value, or a new access token, is a leak. +LEAK_QUERY = """AppServiceConsoleLogs +| where _ResourceId =~ '{site_id}' +| summarize started = countif(ResultDescription contains 'Adding admin user'), + leaks = countif((ResultDescription contains '--password=' and ResultDescription !contains '--password=***') + or (ResultDescription contains '--secret ' and ResultDescription !contains '--secret ***') + or ResultDescription contains 'Access token was successfully created')""" + + +async def key_vault_references(arm, site_id): + """Return the status of each secret app setting. Azure returns the status only, not the value.""" + statuses = {} + for name in SECRET_SETTINGS: + reference = await arm.request("GET", f"{site_id}/config/configreferences/appsettings/{name}", WEB_API) + statuses[name] = reference["properties"].get("status") + return statuses + + +async def console_log_counts(arm, workspace_id, site_id, minutes=15): + """Count the start lines and the secret lines in the console log. The query returns counts only.""" + if not SITE_ID.match(site_id): + raise ValueError("Expected a web app resource ID") + customer_id = (await arm.request("GET", workspace_id, LOG_API))["properties"]["customerId"] + query = LEAK_QUERY.format(site_id=site_id.lower()) + async with asyncio.timeout(minutes * 60): + while True: + token = await asyncio.to_thread(arm.credential.get_token, f"{LOGS}/.default") + response = await arm.client.post( + f"{LOGS}/v1/workspaces/{customer_id}/query", + headers={"Authorization": f"Bearer {token.token}"}, + json={"query": query, "timespan": "P1D"}, + ) + if response.status_code != 200: + raise RuntimeError(f"Log Analytics query failed with HTTP {response.status_code}") + table = response.json()["tables"][0] + counts = dict(zip([column["name"] for column in table["columns"]], table["rows"][0])) + # Log Analytics can show a line some minutes after the app writes it. + if counts["started"]: + return counts + await asyncio.sleep(60) diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py index e733aead7..4bb609779 100644 --- a/e2e_tests/test_gitea.py +++ b/e2e_tests/test_gitea.py @@ -12,6 +12,7 @@ from e2e_tests.conftest import get_workspace_owner_token from e2e_tests.resources import strings from e2e_tests.resources.gitea_probe import run_probe +from e2e_tests.resources.gitea_secrets import console_log_counts, key_vault_references from e2e_tests.resources.resource import get_resource, post_resource from e2e_tests.resources.sql_lifecycle import sql_lifecycle from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, secret_access, validate_vm @@ -109,6 +110,16 @@ async def test_private_gitea_repository_survives_update(verify): await arm.request("POST", f"{site_id}/restart", WEB_API) await probe(phase="read") LOGGER.info("The Gitea repository survived the update and the restart") + # The web app sends its console log to Log Analytics, so the log must not show a secret. + references = await key_vault_references(arm, site_id) + assert set(references.values()) == {"Resolved"}, f"Gitea Key Vault references: {references}" + log_workspace = ( + f"/subscriptions/{os.environ['ARM_SUBSCRIPTION_ID']}/resourceGroups/rg-{config.TRE_ID}" + f"/providers/Microsoft.OperationalInsights/workspaces/log-{config.TRE_ID}" + ) + counts = await console_log_counts(arm, log_workspace, site_id) + LOGGER.info("Gitea console log lines: %s", counts) + assert counts["leaks"] == 0, f"The Gitea console log has {counts['leaks']} lines with a secret" # Check the network settings last, so that a failure here keeps the functional evidence above. app = await arm.request("GET", site_id, WEB_API) server = await arm.request( diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py index d234bf291..c2f5658ca 100644 --- a/e2e_tests/unit_tests/test_gitea_probe.py +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -5,7 +5,7 @@ import re from types import SimpleNamespace import unittest -from unittest.mock import AsyncMock +from unittest.mock import AsyncMock, Mock, patch from uuid import uuid4 from jsonschema import validate @@ -14,6 +14,7 @@ from api_app.services.schema_service import enrich_user_resource_template, enrich_workspace_service_template from e2e_tests import test_gitea as gitea from e2e_tests.resources import gitea_probe as probe +from e2e_tests.resources import gitea_secrets as secrets ROOT = Path(__file__).resolve().parents[2] @@ -142,3 +143,45 @@ async def test_failed_command_is_removed(self): "Phase": "write", }, ) + + +class SecretLogTests(unittest.IsolatedAsyncioTestCase): + SITE = f"{GROUP.replace('/sub/', '/00000000-0000-0000-0000-000000000000/')}/providers/Microsoft.Web/sites/gitea-tretest-ws-abcd-svc-1234" + WORKSPACE = ( + "/subscriptions/sub/resourceGroups/rg-tretest/providers/Microsoft.OperationalInsights/workspaces/log-tretest" + ) + + @staticmethod + def arm(*rows): + responses = [ + SimpleNamespace( + status_code=200, + json=Mock( + return_value={"tables": [{"columns": [{"name": "started"}, {"name": "leaks"}], "rows": [row]}]} + ), + ) + for row in rows + ] + return SimpleNamespace( + request=AsyncMock(return_value={"properties": {"customerId": "customer", "status": "Resolved"}}), + credential=Mock(get_token=Mock(return_value=SimpleNamespace(token="token"))), + client=SimpleNamespace(post=AsyncMock(side_effect=responses)), + ) + + async def test_counts_wait_until_the_start_lines_arrive(self): + arm = self.arm([0, 0], [2, 0]) + with patch("e2e_tests.resources.gitea_secrets.asyncio.sleep", AsyncMock()) as sleep: + counts = await secrets.console_log_counts(arm, self.WORKSPACE, self.SITE) + self.assertEqual(counts, {"started": 2, "leaks": 0}) + sleep.assert_awaited_once() + self.assertIn(self.SITE.lower(), arm.client.post.call_args.kwargs["json"]["query"]) + + async def test_counts_reject_a_site_id_that_changes_the_query(self): + arm = self.arm() + with self.assertRaises(ValueError): + await secrets.console_log_counts(arm, self.WORKSPACE, f"{self.SITE}' or true or '") + arm.client.post.assert_not_awaited() + + async def test_references_report_the_status_of_each_secret_setting(self): + statuses = await secrets.key_vault_references(self.arm(), self.SITE) + self.assertEqual(statuses, {"GITEA_PASSWD": "Resolved", "GITEA_OPENID_CLIENT_SECRET": "Resolved"}) From 361b9b87506f155b4ac6d387873c765132c32c8e Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:43:38 +0000 Subject: [PATCH 07/23] Wrap a long CHANGELOG entry The Markdown lint rejects a line with more than 400 characters. The entry now continues on lines that start with two spaces, as an earlier entry in the file does. The rendered text does not change. --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebb6c6a23..9ab0ec4bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,7 +47,8 @@ BUG FIXES: * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) -* Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) +* Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. + The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) From 5c5fd11f3004bd6cdf78bd36183191b46b165aa5 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:09:21 +0000 Subject: [PATCH 08/23] Give the Gitea identity a role on its own secrets only The Gitea workspace service identity had the role "Key Vault Secrets User" on the whole workspace Key Vault. The web app reads only three secrets through its Key Vault references: the admin password, the workspace client secret and the database password. The identity now has the role on each of these three secrets, and the web app waits for these roles. The two secrets no longer wait for the old role. The resource processor writes them with its own identity. A contract test makes sure that no Gitea role has the whole vault as its scope, and that each Key Vault reference has a role. Version: `tre-workspace-service-gitea` 1.3.11. --- CHANGELOG.md | 1 + .../unit_tests/test_gitea_bundle_contract.py | 13 +++++++++++++ templates/workspace_services/gitea/porter.yaml | 2 +- .../gitea/terraform/gitea-webapp.tf | 17 ++++++++++------- .../workspace_services/gitea/terraform/mysql.tf | 4 ---- 5 files changed, 25 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ab0ec4bb..2f3a949fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,7 @@ BUG FIXES: (`tre-workspace-base` 2.12.4, `tre-workspace-unrestricted` 0.15.3, `tre-workspace-airlock-import-review` 0.18.3) ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) +* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 1.3.11) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) diff --git a/e2e_tests/unit_tests/test_gitea_bundle_contract.py b/e2e_tests/unit_tests/test_gitea_bundle_contract.py index a2535abbc..2b3eaf3ea 100644 --- a/e2e_tests/unit_tests/test_gitea_bundle_contract.py +++ b/e2e_tests/unit_tests/test_gitea_bundle_contract.py @@ -82,3 +82,16 @@ def test_openid_secret_is_a_key_vault_reference(self): value = re.search(r"\n GITEA_OPENID_CLIENT_SECRET\s*=\s*(.+)\n", webapp).group(1) self.assertRegex(value, r'^"@Microsoft\.KeyVault\(SecretUri=.+secrets/workspace-client-secret\)"$') self.assertNotIn('data "azurerm_key_vault_secret" "client_secret"', (BUNDLE / "data.tf").read_text()) + + def test_identity_reads_only_its_key_vault_references(self): + # The workspace Key Vault also has the secrets of the other workspace services. + webapp = (BUNDLE / "gitea-webapp.tf").read_text() + for name in re.findall(r'resource "azurerm_role_assignment" "(\w+)"', webapp): + assignment = block(webapp, f'resource "azurerm_role_assignment" "{name}"') + with self.subTest(role=name): + self.assertNotRegex(assignment, r"\n scope\s*=\s*data\.azurerm_key_vault\.ws\.id\n") + roles = block(webapp, 'resource "azurerm_role_assignment" "keyvault_gitea_secret_role"') + scopes = re.findall(r"\n \w+\s*=\s*(.+)", roles.split("for_each = {", 1)[1].split("\n }", 1)[0]) + references = re.findall(r"@Microsoft\.KeyVault\(SecretUri=(.+?)\)", webapp) + self.assertEqual(len(scopes), len(references)) + self.assertIn('"${data.azurerm_key_vault.ws.id}/secrets/workspace-client-secret"', scopes) diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index 735e18fdb..b98638de5 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.10 +version: 1.3.11 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/terraform/gitea-webapp.tf b/templates/workspace_services/gitea/terraform/gitea-webapp.tf index db23f9461..db9333546 100644 --- a/templates/workspace_services/gitea/terraform/gitea-webapp.tf +++ b/templates/workspace_services/gitea/terraform/gitea-webapp.tf @@ -110,7 +110,7 @@ resource "azurerm_linux_web_app" "gitea" { } depends_on = [ - azurerm_key_vault_secret.gitea_password + azurerm_role_assignment.keyvault_gitea_secret_role ] } @@ -183,8 +183,15 @@ resource "azurerm_monitor_diagnostic_setting" "gitea" { } } -resource "azurerm_role_assignment" "keyvault_gitea_ws_role" { - scope = data.azurerm_key_vault.ws.id +# The web app reads only the secrets of its Key Vault references, so its identity gets a role on each of them. +resource "azurerm_role_assignment" "keyvault_gitea_secret_role" { + for_each = { + admin_password = azurerm_key_vault_secret.gitea_password.resource_versionless_id + client_secret = "${data.azurerm_key_vault.ws.id}/secrets/workspace-client-secret" + db_password = azurerm_key_vault_secret.db_password.resource_versionless_id + } + + scope = each.value role_definition_name = "Key Vault Secrets User" principal_id = azurerm_user_assigned_identity.gitea_id.principal_id } @@ -195,10 +202,6 @@ resource "azurerm_key_vault_secret" "gitea_password" { key_vault_id = data.azurerm_key_vault.ws.id tags = local.workspace_service_tags - depends_on = [ - azurerm_role_assignment.keyvault_gitea_ws_role - ] - lifecycle { ignore_changes = [tags] } } diff --git a/templates/workspace_services/gitea/terraform/mysql.tf b/templates/workspace_services/gitea/terraform/mysql.tf index bc5b1fd8d..06b649468 100644 --- a/templates/workspace_services/gitea/terraform/mysql.tf +++ b/templates/workspace_services/gitea/terraform/mysql.tf @@ -66,9 +66,5 @@ resource "azurerm_key_vault_secret" "db_password" { key_vault_id = data.azurerm_key_vault.ws.id tags = local.workspace_service_tags - depends_on = [ - azurerm_role_assignment.keyvault_gitea_ws_role - ] - lifecycle { ignore_changes = [tags] } } From 7f310dc99eee5b38cf6b3847e5560cb8e14e80a2 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:13:08 +0000 Subject: [PATCH 09/23] Give the shared Gitea identity a role on its own secrets only The Gitea shared service identity had the role "Key Vault Secrets User" on the whole core Key Vault. The web app reads only two secrets through its Key Vault references: the admin password and the database password. The identity now has the role on each of these two secrets, and the web app waits for these roles. The two secrets no longer wait for the old role. The resource processor writes them with its own identity. Version: `tre-shared-service-gitea` 1.2.6. --- CHANGELOG.md | 1 + .../gitea/terraform/gitea-webapp.tf | 16 +++++++++------- .../shared_services/gitea/terraform/mysql.tf | 4 ---- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f3a949fd..49019e267 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,7 @@ BUG FIXES: * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) +* Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/templates/shared_services/gitea/terraform/gitea-webapp.tf b/templates/shared_services/gitea/terraform/gitea-webapp.tf index 068b11b39..0ef3602db 100644 --- a/templates/shared_services/gitea/terraform/gitea-webapp.tf +++ b/templates/shared_services/gitea/terraform/gitea-webapp.tf @@ -98,7 +98,7 @@ resource "azurerm_linux_web_app" "gitea" { } depends_on = [ - azurerm_key_vault_secret.gitea_password + azurerm_role_assignment.keyvault_gitea_secret_role ] } @@ -172,8 +172,14 @@ resource "azurerm_monitor_diagnostic_setting" "webapp_gitea" { } } -resource "azurerm_role_assignment" "keyvault_gitea_role" { - scope = data.azurerm_key_vault.keyvault.id +# The web app reads only the secrets of its Key Vault references, so its identity gets a role on each of them. +resource "azurerm_role_assignment" "keyvault_gitea_secret_role" { + for_each = { + admin_password = azurerm_key_vault_secret.gitea_password.resource_versionless_id + db_password = azurerm_key_vault_secret.db_password.resource_versionless_id + } + + scope = each.value role_definition_name = "Key Vault Secrets User" principal_id = azurerm_user_assigned_identity.gitea_id.principal_id } @@ -184,10 +190,6 @@ resource "azurerm_key_vault_secret" "gitea_password" { key_vault_id = data.azurerm_key_vault.keyvault.id tags = local.tre_shared_service_tags - depends_on = [ - azurerm_role_assignment.keyvault_gitea_role - ] - lifecycle { ignore_changes = [tags] } } diff --git a/templates/shared_services/gitea/terraform/mysql.tf b/templates/shared_services/gitea/terraform/mysql.tf index 3a1b9425c..a180185ea 100644 --- a/templates/shared_services/gitea/terraform/mysql.tf +++ b/templates/shared_services/gitea/terraform/mysql.tf @@ -64,9 +64,5 @@ resource "azurerm_key_vault_secret" "db_password" { key_vault_id = data.azurerm_key_vault.keyvault.id tags = local.tre_shared_service_tags - depends_on = [ - azurerm_role_assignment.keyvault_gitea_role - ] - lifecycle { ignore_changes = [tags] } } From efe6e9d9033c39424dfd1d42874ad3bb31f7e2f5 Mon Sep 17 00:00:00 2001 From: Jay W Date: Sun, 25 Oct 2026 22:14:17 +0000 Subject: [PATCH 10/23] Give the Nexus identity a role on its own secrets only The Nexus VM identity had the role "Key Vault Secrets User" on the whole core Key Vault. The VM reads only two secrets: the Nexus admin password and the secret of the Nexus TLS certificate. The identity now has the role on each of these two secrets. A new test makes sure that the shared Gitea and Nexus identities have no Key Vault secret role on the whole vault. Version: `tre-shared-service-sonatype-nexus` 3.11.3. --- CHANGELOG.md | 1 + devops/tests/test_key_vault_scopes.py | 35 +++++++++++++++++++ .../sonatype-nexus-vm/terraform/vm.tf | 12 +++++-- 3 files changed, 45 insertions(+), 3 deletions(-) create mode 100644 devops/tests/test_key_vault_scopes.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 49019e267..203d246c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,7 @@ BUG FIXES: * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) * Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) +* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.3) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/devops/tests/test_key_vault_scopes.py b/devops/tests/test_key_vault_scopes.py new file mode 100644 index 000000000..3531e1b66 --- /dev/null +++ b/devops/tests/test_key_vault_scopes.py @@ -0,0 +1,35 @@ +"""Make sure that the shared service identities read only their own Key Vault secrets.""" + +from pathlib import Path +import re +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +# Each of these identities reads a fixed list of secrets from the core Key Vault. +FILES = ( + ROOT / "templates/shared_services/gitea/terraform/gitea-webapp.tf", + ROOT / "templates/shared_services/sonatype-nexus-vm/terraform/vm.tf", +) +ASSIGNMENT = re.compile(r'^resource "azurerm_role_assignment" "(\w+)" \{(\n.*?)^\}', re.MULTILINE | re.DOTALL) +VAULT_SCOPE = re.compile(r"\n scope\s*=\s*data\.azurerm_key_vault\.\w+\.id\n") + + +class KeyVaultScopeTests(unittest.TestCase): + def test_secret_roles_are_not_on_the_whole_vault(self): + for path in FILES: + roles = [ + (name, body) + for name, body in ASSIGNMENT.findall(path.read_text()) + if '"Key Vault Secrets User"' in body + ] + with self.subTest(path=str(path.relative_to(ROOT))): + self.assertTrue(roles) + for name, body in roles: + with self.subTest(path=str(path.relative_to(ROOT)), role=name): + self.assertNotRegex(body, VAULT_SCOPE) + self.assertIn("for_each = {", body) + + +if __name__ == "__main__": + unittest.main() diff --git a/templates/shared_services/sonatype-nexus-vm/terraform/vm.tf b/templates/shared_services/sonatype-nexus-vm/terraform/vm.tf index d2723d1e1..e16812e05 100644 --- a/templates/shared_services/sonatype-nexus-vm/terraform/vm.tf +++ b/templates/shared_services/sonatype-nexus-vm/terraform/vm.tf @@ -87,8 +87,14 @@ resource "azurerm_user_assigned_identity" "nexus_msi" { lifecycle { ignore_changes = [tags] } } -resource "azurerm_role_assignment" "keyvault_nexus_role" { - scope = data.azurerm_key_vault.kv.id +# The VM reads only the Nexus admin password and its TLS certificate, so its identity gets a role on each of them. +resource "azurerm_role_assignment" "keyvault_nexus_secret_role" { + for_each = { + admin_password = azurerm_key_vault_secret.nexus_admin_password.resource_versionless_id + ssl_certificate = "${data.azurerm_key_vault.kv.id}/secrets/${data.azurerm_key_vault_certificate.nexus_cert.name}" + } + + scope = each.value role_definition_name = "Key Vault Secrets User" principal_id = azurerm_user_assigned_identity.nexus_msi.principal_id } @@ -148,7 +154,7 @@ resource "azurerm_linux_virtual_machine" "nexus" { } depends_on = [ - azurerm_role_assignment.keyvault_nexus_role + azurerm_role_assignment.keyvault_nexus_secret_role ] connection { From d1c5822f10f40c21eaafbcc71e41816276b74f84 Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 21:58:29 +0000 Subject: [PATCH 11/23] Use one workspace owner token in the Gitea test (#5141) The MySQL lifecycle helpers no longer take a cleanup token. The owner token is a TokenProvider, and get_auth_header gets a current token from it for each request. So the Gitea test uses one token for the whole case and its cleanup, as the MySQL test does. --- e2e_tests/test_gitea.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py index 4bb609779..1eb59772e 100644 --- a/e2e_tests/test_gitea.py +++ b/e2e_tests/test_gitea.py @@ -54,14 +54,9 @@ async def test_private_gitea_repository_survives_update(verify): raise ValueError("The Gitea probe currently supports AzureCloud only") async with sql_lifecycle(verify, deadline_variable="GITEA_VALIDATION_DEADLINE") as resources: workspace_path, workspace_id, guacamole_path, guacamole_id = await resources.setup() - new_token = partial(get_workspace_owner_token, workspace_id, verify) - gitea_path, gitea_id = await resources.create( - GITEA_PAYLOAD, f"/api{workspace_path}/workspace-services", await new_token(), cleanup_token=new_token - ) - vm_path, vm_id = await resources.create( - VM_PAYLOAD, f"/api{guacamole_path}/user-resources", await new_token(), cleanup_token=new_token - ) - token = await new_token() + token = await get_workspace_owner_token(workspace_id, verify) + gitea_path, gitea_id = await resources.create(GITEA_PAYLOAD, f"/api{workspace_path}/workspace-services", token) + vm_path, vm_id = await resources.create(VM_PAYLOAD, f"/api{guacamole_path}/user-resources", token) gitea = (await get_resource(f"/api{gitea_path}", token, verify))["workspaceService"] vm_properties = (await get_resource(f"/api{vm_path}", token, verify))["userResource"]["properties"] subscription_id = vm_properties.get("workspace_subscription_id") or os.environ["ARM_SUBSCRIPTION_ID"] @@ -95,7 +90,6 @@ async def test_private_gitea_repository_survives_update(verify): ) await probe(phase="write") LOGGER.info("Gitea is private, rejects anonymous and wrong-password requests, and stores a commit") - token = await new_token() gitea = (await get_resource(f"/api{gitea_path}", token, verify))["workspaceService"] async with asyncio.timeout(60 * 60): await post_resource( From 9cd14a9911c2ece4e7851e88b0c2cb6d628bf63f Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 21:58:29 +0000 Subject: [PATCH 12/23] Use one version bump for each Gitea and Nexus bundle The four combined changes bumped the Gitea workspace service three times. Upstream main has version 1.3.7, so this pull request uses 1.3.8. The two upgrade sections of the Gitea document now share one heading for 1.3.8. Upstream main already has Nexus 3.11.3, so the Nexus change uses 3.11.4. --- CHANGELOG.md | 8 ++++---- docs/tre-templates/workspace-services/gitea.md | 10 ++++++---- .../shared_services/sonatype-nexus-vm/porter.yaml | 2 +- templates/workspace_services/gitea/porter.yaml | 2 +- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 203d246c7..03cb21fc0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,15 +43,15 @@ BUG FIXES: (`tre-workspace-base` 2.12.4, `tre-workspace-unrestricted` 0.15.3, `tre-workspace-airlock-import-review` 0.18.3) ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) -* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 1.3.11) +* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 1.3.8) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) -* Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.9) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. - The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.10, `tre-shared-service-gitea` 1.2.6) + The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.8, `tre-shared-service-gitea` 1.2.6) * Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) -* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.3) +* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 231feb179..f78d7259d 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -29,9 +29,11 @@ The Gitea worskpace service opens outbound access to: Migrating existing Gitea services to the major version 1.0.0 is not currently supported. This is due to the breaking change in the Terraform to migrate from the deprecated mysql_server to the new mysql_flexible_server. -## Upgrading to version 1.3.9 +## Upgrading to version 1.3.8 -Before version 1.3.9, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 1.3.9, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. +### Repositories on the file share + +Before version 1.3.8, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 1.3.8, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. To keep the repositories, do these steps from a workspace VM: @@ -41,9 +43,9 @@ To keep the repositories, do these steps from a workspace VM: 4. Create each repository again in Gitea. 5. Push each mirror into its new repository with `git push --mirror `. -## Upgrading to version 1.3.10 +### Secrets in the console log -Before version 1.3.10, the Gitea workspace service wrote secrets to the console log of its web app. The web app sends this log to Log Analytics, so all people who can read the Log Analytics workspace of the TRE can read these secrets: +Before version 1.3.8, the Gitea workspace service wrote secrets to the console log of its web app. The web app sends this log to Log Analytics, so all people who can read the Log Analytics workspace of the TRE can read these secrets: - The client secret of the workspace application. - The password of the Gitea admin user. diff --git a/templates/shared_services/sonatype-nexus-vm/porter.yaml b/templates/shared_services/sonatype-nexus-vm/porter.yaml index 8af9e4553..2c32e4586 100644 --- a/templates/shared_services/sonatype-nexus-vm/porter.yaml +++ b/templates/shared_services/sonatype-nexus-vm/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-shared-service-sonatype-nexus -version: 3.11.3 +version: 3.11.4 description: "A Sonatype Nexus shared service" dockerfile: Dockerfile.tmpl registry: azuretre diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index b98638de5..379186665 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.11 +version: 1.3.8 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre From d6e0cf27644f56d8da7b3f19e569204755023b27 Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 23:00:13 +0000 Subject: [PATCH 13/23] Hide the shared Gitea admin password and reload changed OIDC settings The shared Gitea image ran the admin user create at each start. After the first start, the user existed and the command failed. Gitea then printed its full command line, with the admin password, to the console log. The shared image now has its own configure_gitea_admin.sh. The script waits for the web service, creates the admin user only if it does not exist, and hides the secret values in the Gitea output. Gitea reads the settings of an existing OIDC source only at its start. So update-oauth stored a new client secret that the running Gitea did not use. The workspace script now keeps a hash of the OIDC settings on the file share. If the settings change, the script exits with status 10, and configure_gitea.sh stops Gitea. The image then stops the container, and App Service starts it again. The script requests the stop only if it stored the hash, so a failed write cannot cause a restart loop. If a secret setting is empty or still has the Key Vault reference text, both scripts stop before any Gitea command. The Gitea documents have new upgrade steps: - Repository recovery that keeps the issues and pull requests: Gitea makes the missing repositories again from their records, and the operator pushes the mirrors back. A wiki needs one new page before its push. - A method to rotate the workspace client secret, the update of the other services that use it, and a refresh of the Key Vault references. - The removal of the gitea-admin access token, and a note that the Key Vault keeps the old admin password after a manual change. --- CHANGELOG.md | 5 +- docs/tre-templates/shared-services/gitea.md | 12 ++ .../tre-templates/workspace-services/gitea.md | 28 ++-- .../unit_tests/test_gitea_bundle_contract.py | 125 +++++++++++++++++- .../shared_services/gitea/docker/Dockerfile | 5 +- .../gitea/docker/configure_gitea_admin.sh | 37 ++++++ .../gitea/docker/configure_gitea.sh | 6 + .../gitea/docker/configure_gitea_admin.sh | 25 +++- 8 files changed, 225 insertions(+), 18 deletions(-) create mode 100644 templates/shared_services/gitea/docker/configure_gitea_admin.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 03cb21fc0..8b4aaa527 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,11 +45,12 @@ BUG FIXES: * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) * Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 1.3.8) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) -* Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install could fail, and after a delete removed the private endpoint, each later update or delete failed. +* Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. - The admin user create in both Gitea bundles also made an access token and printed it. The start script now hides the secrets in the Gitea output, and it gives the OIDC source the current client secret at each start. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.8, `tre-shared-service-gitea` 1.2.6) + The admin user create in both Gitea bundles also made an access token and printed it, and the shared service printed its admin password at each restart. The start scripts of both bundles now hide the secrets in the Gitea output and create the admin user only if it does not exist. + The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.8, `tre-shared-service-gitea` 1.2.6) * Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) * Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) diff --git a/docs/tre-templates/shared-services/gitea.md b/docs/tre-templates/shared-services/gitea.md index efa234ebc..c84bac115 100644 --- a/docs/tre-templates/shared-services/gitea.md +++ b/docs/tre-templates/shared-services/gitea.md @@ -51,3 +51,15 @@ Gitea needs to be able to access the following resource outside the Azure TRE VN ## Upgrading to version 1.0.0 Migrating existing Gitea services to the major version 1.0.0 is not currently supported. This is due to the breaking change in the Terraform to migrate from the deprecated mysql_server to the new mysql_flexible_server. + +## Upgrading to version 1.2.6 + +Before version 1.2.6, the Gitea shared service wrote secrets to the console log of its web app. The web app sends this log to the Log Analytics workspace of the TRE core, so all people who can read it can read these secrets: + +- An access token of the `giteaadmin` user, with the name `gitea-admin`. Gitea printed it at the first start. +- The password of the `giteaadmin` user. Gitea printed it at each later start, because the user create failed. + +If a Gitea shared service ran an earlier version, do these steps after the upgrade: + +1. Sign in to Gitea as `giteaadmin`. In the user settings, open Applications and delete the access token `gitea-admin`. Keep the token with the name `giteaadmin`, because `gitea_migrate_repo.sh` uses it. +2. If the people who can read Log Analytics must not get admin access to Gitea, change the password of `giteaadmin` in Gitea. Keep the new password in a safe location. The Key Vault secret `gitea--administrator-password` keeps the old value, because the start script does not change the password of an existing admin user. diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index f78d7259d..3f8ff982a 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -34,14 +34,18 @@ Migrating existing Gitea services to the major version 1.0.0 is not currently su ### Repositories on the file share Before version 1.3.8, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 1.3.8, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. +The issues, pull requests and releases stay in MySQL, and the LFS files stay on the file share. To keep the repositories, do these steps from a workspace VM: 1. Clone each repository that you want to keep with `git clone --mirror `. -2. Upgrade the workspace service. -3. In Gitea, open Site Administration, then Dashboard. Run "Delete all repositories missing their Git files". -4. Create each repository again in Gitea. -5. Push each mirror into its new repository with `git push --mirror `. +2. If a repository has a wiki, also clone its wiki with `git clone --mirror `. The wiki URL ends in `.wiki.git` in place of `.git`. +3. Upgrade the workspace service. +4. In Gitea, open Site Administration, then Dashboard. Run "Reinitialize all missing Git repositories for which records exist". +5. Push each mirror into the same repository with `git push --mirror `. +6. For each wiki, create one page in Gitea. Then push the wiki mirror with `git push --mirror `. + +Do not run "Delete all repositories missing their Git files". This task also deletes the records of the repositories, for example their issues and pull requests. ### Secrets in the console log @@ -49,14 +53,20 @@ Before version 1.3.8, the Gitea workspace service wrote secrets to the console l - The client secret of the workspace application. - The password of the Gitea admin user. -- An access token of the Gitea admin user. +- An access token of the Gitea admin user, with the name `gitea-admin`. If a Gitea workspace service ran an earlier version, do these steps after the upgrade: -1. Rotate the client secret of the workspace application. Make sure that the workspace Key Vault secret `workspace-client-secret` has the new value. -2. Restart the Gitea web app. At each start, Gitea gives its OIDC source the client secret from the Key Vault. -3. Sign in to Gitea as the admin user. In the user settings, open Applications and delete the access tokens. -4. If the people who can read Log Analytics must not get admin access to Gitea, change the password of the admin user in Gitea. +1. Make a new client secret for the workspace application: + - If Terraform registered the workspace application (`register_aad_application` is `true`), delete its client secret in Microsoft Entra ID. Then update the workspace. Terraform makes a new client secret and writes it to the workspace Key Vault secret `workspace-client-secret`. + - If you registered the workspace application, add a new client secret in Microsoft Entra ID. Update the workspace property `client_secret` with the new value. Then delete the old client secret. +2. Update each Guacamole and OHDSI workspace service in the workspace. These services use the version of the secret that was current when they deployed. +3. Make the Gitea web app read the new client secret from the Key Vault. Send a POST request to `https://management.azure.com//config/configreferences/appsettings/refresh?api-version=2022-03-01`, for example with `az rest`. +4. Restart the Gitea web app. At the start, the start script gives the OIDC source the new client secret. Then it stops Gitea one time, and App Service starts it again with the new secret. +5. Sign in to Gitea as the admin user. In the user settings, open Applications and delete the access token `gitea-admin`. +6. If the people who can read Log Analytics must not get admin access to Gitea, change the password of the admin user in Gitea. Keep the new password in a safe location. The Key Vault secret `-administrator-password` keeps the old value, because the start script does not change the password of an existing admin user. + +From version 1.3.8, the start script keeps a hash of the OIDC settings on the file share. If the settings change, the script stops Gitea one time, because Gitea reads them only at its start. The first start after the upgrade also does this, because the hash does not exist yet. ## Repairing a service that cannot reach its file share diff --git a/e2e_tests/unit_tests/test_gitea_bundle_contract.py b/e2e_tests/unit_tests/test_gitea_bundle_contract.py index 2b3eaf3ea..a9fad421d 100644 --- a/e2e_tests/unit_tests/test_gitea_bundle_contract.py +++ b/e2e_tests/unit_tests/test_gitea_bundle_contract.py @@ -12,6 +12,7 @@ START_SCRIPT = ROOT / "templates/workspace_services/gitea/docker/configure_gitea.sh" ADMIN_SCRIPT = ROOT / "templates/workspace_services/gitea/docker/configure_gitea_admin.sh" SHARED_DOCKERFILE = ROOT / "templates/shared_services/gitea/docker/Dockerfile" +SHARED_ADMIN_SCRIPT = ROOT / "templates/shared_services/gitea/docker/configure_gitea_admin.sh" SECRETS = ("GITEA_PASSWD", "GITEA_OPENID_CLIENT_SECRET") @@ -19,6 +20,32 @@ def block(source, header): return source.split(header + " {", 1)[1].split("\n}", 1)[0] +def run_script(script, folder, stubs, environment): + """Run a start script with stub commands in front of PATH.""" + for name, body in stubs.items(): + stub = Path(folder) / name + stub.write_text("#!/bin/bash\n" + body) + stub.chmod(0o755) + environment = {**environment, "PATH": f"{folder}{os.pathsep}{os.environ['PATH']}"} + return subprocess.run(["bash", str(script)], env=environment, capture_output=True, text=True, check=False) + + +# Gitea prints a table with a header line for each list command. +GITEA_STUB = """case "$*" in + "admin user list --admin") printf 'ID Username Email\\n1 giteaadmin giteaadmin@example.com\\n' ;; + "admin auth list") printf 'ID Name Type Enabled\\n%s' "${STUB_SOURCES:-}" ;; +esac +""" +OIDC_ENVIRONMENT = { + "GITEA_USERNAME": "giteaadmin", + "GITEA_PASSWD": "password", + "GITEA_EMAIL": "giteaadmin@example.com", + "GITEA_OPENID_CLIENT_ID": "client-id", + "GITEA_OPENID_CLIENT_SECRET": "first-secret", + "GITEA_OPENID_AUTHORITY": "https://login.example.com/tenant/v2.0", +} + + class GiteaBundleContractTests(unittest.TestCase): def test_share_uses_resource_manager(self): # With storage_account_name, the provider uses the file endpoint. The TRE firewall blocks it @@ -33,12 +60,13 @@ def test_server_disables_public_network_access(self): def test_start_script_prints_no_secret(self): # The web app sends the console output to Log Analytics. - for script in (START_SCRIPT, ADMIN_SCRIPT): + for script in (START_SCRIPT, ADMIN_SCRIPT, SHARED_ADMIN_SCRIPT): for line in script.read_text().splitlines(): if line.lstrip().startswith(("echo", "printf")) or "set -x" in line: for secret in SECRETS: with self.subTest(script=script.name, line=line, secret=secret): - self.assertNotIn(secret, line) + # A line can name a setting, but it must not expand its value. + self.assertNotRegex(line, r"\$\{?" + secret) def test_admin_script_hides_secrets_in_gitea_output(self): # Gitea prints its full command line when a command fails. @@ -58,6 +86,7 @@ def test_admin_script_hides_secrets_in_gitea_output(self): "GITEA_EMAIL": "giteaadmin@example.com", "GITEA_OPENID_CLIENT_ID": "client-id", "GITEA_OPENID_AUTHORITY": "https://login.example.com/tenant/v2.0", + "GITEA_CUSTOM": folder, } result = subprocess.run( ["bash", str(ADMIN_SCRIPT)], env=environment, capture_output=True, text=True, check=False @@ -69,9 +98,99 @@ def test_admin_script_hides_secrets_in_gitea_output(self): self.assertIn("user create --admin --username=giteaadmin --password=***", output) self.assertIn("add-oauth --name oidc --provider openidConnect --key client-id --secret ***", output) + def test_shared_admin_script_hides_secrets_in_gitea_output(self): + # Before the fix, the shared image printed the admin password at each restart. + secrets = {"GITEA_PASSWD": "pass$word[1]", "GITEA__database__PASSWD": "db.secret"} + with tempfile.TemporaryDirectory() as folder: + result = run_script( + SHARED_ADMIN_SCRIPT, + folder, + {"curl": "echo 200\n", "gitea": 'echo "[F] Failed to run app with [gitea $*]"\nexit 1\n'}, + {**secrets, "GITEA_USERNAME": "giteaadmin", "GITEA_EMAIL": "giteaadmin@example.com"}, + ) + output = result.stdout + result.stderr + for name, value in secrets.items(): + with self.subTest(secret=name): + self.assertNotIn(value, output) + self.assertIn("user create --admin --username=giteaadmin --password=***", output) + + def test_admin_scripts_skip_an_unresolved_key_vault_reference(self): + # If App Service cannot read a reference, the value is the reference text. + reference = "@Microsoft.KeyVault(SecretUri=https://kv.vault.azure.net/secrets/name)" + for script, name in ( + (ADMIN_SCRIPT, "GITEA_PASSWD"), + (ADMIN_SCRIPT, "GITEA_OPENID_CLIENT_SECRET"), + (SHARED_ADMIN_SCRIPT, "GITEA_PASSWD"), + ): + with ( + self.subTest(script=script.parent.parent.parent.name, setting=name), + tempfile.TemporaryDirectory() as folder, + ): + calls = Path(folder) / "calls" + result = run_script( + script, + folder, + {"curl": "echo 200\n", "gitea": f'echo "$*" >> "{calls}"\n'}, + {**OIDC_ENVIRONMENT, name: reference}, + ) + self.assertEqual(result.returncode, 1) + self.assertFalse(calls.exists()) + self.assertIn( + f"The value of {name} is missing or its Key Vault reference is not resolved", result.stdout + ) + + def test_admin_script_requests_a_restart_when_the_oidc_settings_change(self): + # Gitea reads the settings of an existing OIDC source only at its start. + cases = ( + ("new source", "", "first-secret", 0), + ("same settings", "1 oidc OAuth2 true\n", "first-secret", 0), + ("new client secret", "1 oidc OAuth2 true\n", "second-secret", 10), + ("same new secret", "1 oidc OAuth2 true\n", "second-secret", 0), + ) + with tempfile.TemporaryDirectory() as folder, tempfile.TemporaryDirectory() as custom: + for label, sources, secret, status in cases: + with self.subTest(case=label): + result = run_script( + ADMIN_SCRIPT, + folder, + {"gitea": GITEA_STUB}, + { + **OIDC_ENVIRONMENT, + "GITEA_OPENID_CLIENT_SECRET": secret, + "GITEA_CUSTOM": custom, + "STUB_SOURCES": sources, + }, + ) + self.assertEqual(result.returncode, status, result.stdout + result.stderr) + self.assertNotIn("second-secret", (Path(custom) / "oidc-settings.sha256").read_text()) + + def test_admin_script_requests_a_restart_after_an_upgrade(self): + # Earlier versions did not store the hash, so the first start after the upgrade restarts Gitea. + with tempfile.TemporaryDirectory() as folder, tempfile.TemporaryDirectory() as custom: + result = run_script( + ADMIN_SCRIPT, + folder, + {"gitea": GITEA_STUB}, + {**OIDC_ENVIRONMENT, "GITEA_CUSTOM": custom, "STUB_SOURCES": "1 oidc OAuth2 true\n"}, + ) + self.assertEqual(result.returncode, 10) + + def test_start_script_stops_gitea_only_after_an_oidc_change(self): + # The Gitea image stops the container when Gitea stops, and App Service starts it again. + for status, stops in ((0, False), (1, False), (10, True)): + with self.subTest(status=status), tempfile.TemporaryDirectory() as folder: + calls = Path(folder) / "calls" + run_script( + START_SCRIPT, + folder, + {"curl": "echo 200\n", "su": f"exit {status}\n", "s6-svc": f'echo "$*" >> "{calls}"\n'}, + {"GITEA_USERNAME": "giteaadmin"}, + ) + self.assertEqual(calls.read_text() if calls.exists() else "", "-d /etc/s6/gitea\n" if stops else "") + def test_admin_create_makes_no_access_token(self): # Gitea prints a new access token to the console output. - for source in (START_SCRIPT, SHARED_DOCKERFILE): + for source in (START_SCRIPT, ADMIN_SCRIPT, SHARED_ADMIN_SCRIPT, SHARED_DOCKERFILE): commands = [line for line in source.read_text().splitlines() if not line.lstrip().startswith("#")] with self.subTest(source=source.name): self.assertNotIn("--access-token", "\n".join(commands)) diff --git a/templates/shared_services/gitea/docker/Dockerfile b/templates/shared_services/gitea/docker/Dockerfile index b9e002fe1..267b98ddf 100644 --- a/templates/shared_services/gitea/docker/Dockerfile +++ b/templates/shared_services/gitea/docker/Dockerfile @@ -5,5 +5,6 @@ FROM gitea/gitea:${GITEA_TAG} # need to pass args to stage ARG CERTIFICATE_URL RUN wget -nv -O /usr/local/share/ca-certificates/mysql.crt.pem ${CERTIFICATE_URL} && update-ca-certificates -# Do not add --access-token. Gitea prints the new token to the console log. -ENTRYPOINT ["/bin/bash", "-c", "sleep 120 && gitea admin user create --admin --username=\"${GITEA_USERNAME}\" --password=\"${GITEA_PASSWD}\" --email=\"${GITEA_EMAIL}\" --must-change-password=false & /usr/bin/entrypoint"] +COPY configure_gitea_admin.sh / +# The script hides the secrets in the Gitea output, which goes to the console log. +ENTRYPOINT ["/bin/bash", "-c", "bash /configure_gitea_admin.sh & /usr/bin/entrypoint"] diff --git a/templates/shared_services/gitea/docker/configure_gitea_admin.sh b/templates/shared_services/gitea/docker/configure_gitea_admin.sh new file mode 100644 index 000000000..a29afa609 --- /dev/null +++ b/templates/shared_services/gitea/docker/configure_gitea_admin.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# The Dockerfile runs this script at each start. The web app sends its output to Log Analytics. + +# Gitea prints the full command line when a command fails, so hide the secrets in its output. +gitea_cli() { + local output status secret + output=$(gitea "$@" 2>&1) + status=$? + for secret in "$GITEA_PASSWD" "${GITEA__database__PASSWD:-}"; do + if [ -n "$secret" ]; then + output=${output//"$secret"/"***"} + fi + done + if [ -n "$output" ]; then + printf '%s\n' "$output" + fi + return "$status" +} + +while [[ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:3000/api/swagger)" != @("200"|"302") ]]; do + echo "Waiting for web service" + sleep 5 +done + +# If App Service cannot read a Key Vault reference, the value is the reference text. Do not store it in Gitea. +if [[ -z "$GITEA_PASSWD" || "$GITEA_PASSWD" == @Microsoft.KeyVault* ]]; then + echo "The value of GITEA_PASSWD is missing or its Key Vault reference is not resolved" + exit 1 +fi + +echo "Adding admin user" +if gitea_cli admin user list --admin | awk 'NR > 1 { print $2 }' | grep -qxF "$GITEA_USERNAME"; then + echo "The admin user exists" +else + # Do not add --access-token. Gitea prints the new token. + gitea_cli admin user create --admin --username="$GITEA_USERNAME" --password="$GITEA_PASSWD" --email="$GITEA_EMAIL" --must-change-password=false +fi diff --git a/templates/workspace_services/gitea/docker/configure_gitea.sh b/templates/workspace_services/gitea/docker/configure_gitea.sh index d5a3af772..97f527218 100755 --- a/templates/workspace_services/gitea/docker/configure_gitea.sh +++ b/templates/workspace_services/gitea/docker/configure_gitea.sh @@ -13,3 +13,9 @@ fi # The admin commands contain secrets. The script hides them in the Gitea output. su gitea -s /bin/bash -c "bash /configure_gitea_admin.sh" +# Exit status 10 means that the OIDC settings changed. Gitea reads them only at its start. +# When Gitea stops, the image stops the container, and App Service starts it again. +if [ $? -eq 10 ]; then + echo "Stopping Gitea, so that it starts again with the new OIDC settings" + s6-svc -d /etc/s6/gitea +fi diff --git a/templates/workspace_services/gitea/docker/configure_gitea_admin.sh b/templates/workspace_services/gitea/docker/configure_gitea_admin.sh index 5e9d4d10e..9c86def92 100755 --- a/templates/workspace_services/gitea/docker/configure_gitea_admin.sh +++ b/templates/workspace_services/gitea/docker/configure_gitea_admin.sh @@ -1,5 +1,6 @@ #!/bin/bash # configure_gitea.sh runs this script as the gitea user. The web app sends its output to Log Analytics. +# Exit status 10 means that the OIDC settings changed and that Gitea must restart to use them. # Gitea prints the full command line when a command fails, so hide the secrets in its output. gitea_cli() { @@ -17,6 +18,14 @@ gitea_cli() { return "$status" } +# If App Service cannot read a Key Vault reference, the value is the reference text. Do not store it in Gitea. +for name in GITEA_PASSWD GITEA_OPENID_CLIENT_SECRET; do + if [[ -z "${!name}" || "${!name}" == @Microsoft.KeyVault* ]]; then + echo "The value of $name is missing or its Key Vault reference is not resolved" + exit 1 + fi +done + echo "Adding admin user" if gitea_cli admin user list --admin | awk 'NR > 1 { print $2 }' | grep -qxF "$GITEA_USERNAME"; then echo "The admin user exists" @@ -30,7 +39,19 @@ oidc=(--name oidc --provider openidConnect --key "$GITEA_OPENID_CLIENT_ID" --sec source_id=$(gitea_cli admin auth list | awk '$2 == "oidc" { print $1 }') # Update an existing source, so that it gets the current values, for example a new client secret. if [ -z "$source_id" ]; then - gitea_cli admin auth add-oauth "${oidc[@]}" + gitea_cli admin auth add-oauth "${oidc[@]}" || exit else - gitea_cli admin auth update-oauth --id "$source_id" "${oidc[@]}" + gitea_cli admin auth update-oauth --id "$source_id" "${oidc[@]}" || exit +fi + +# The running Gitea keeps the OIDC settings of an existing source from its start. It loads a new source when it needs it. +# The file share keeps a hash of the settings, so that the script can find a change. +hash_file="${GITEA_CUSTOM:-/data/gitea}/oidc-settings.sha256" +hash=$(printf '%s\n' "${oidc[@]}" | sha256sum | cut -d ' ' -f 1) +if [ -n "$source_id" ] && [ "$hash" != "$(cat "$hash_file" 2>/dev/null)" ]; then + changed=true +fi +# Request a restart only if the hash is stored. Otherwise each start requests a restart again. +if printf '%s\n' "$hash" > "$hash_file" && [ "$changed" = true ]; then + exit 10 fi From f4dd202307e7d29848e3dfa91d89ee8fb7caf1ac Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 23:00:22 +0000 Subject: [PATCH 14/23] Stop and start the Gitea app in the test and read all its references The restart call returns before the app restarts, so the old container could serve the read. The case now stops and starts the web app, so that a new container serves the read. The case also reads the Key Vault reference of the database password. The coverage list names the reference check and the console log check. --- CHANGELOG.md | 2 +- docs/tre-developers/end-to-end-tests.md | 6 +++--- e2e_tests/bundle_coverage.json | 4 +++- e2e_tests/resources/gitea_secrets.py | 2 +- e2e_tests/test_gitea.py | 10 ++++++---- e2e_tests/unit_tests/test_gitea_probe.py | 9 ++++++++- 6 files changed, 22 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b4aaa527..e5b232550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) -* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update and an app restart. The check also makes sure that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update, a stop and a start of the app. The check also makes sure that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255)) * Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050)) * Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111)) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index 6c09e1e2e..acbaede4f 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -194,9 +194,9 @@ It gives the VM identity a temporary Key Vault Secrets User role on the secret t From the VM, the case makes sure that the Gitea host resolves to a private address and that the Gitea API is available. It also makes sure that Gitea rejects a repository create without credentials (HTTP 401) and a request with a wrong password (HTTP 401). The VM then creates a private repository with the admin account and commits a file. An anonymous read of the repository gets HTTP 404. -An update changes the display name, and the case restarts the web app. The VM then reads the file again and deletes the repository. -Then the case makes sure that the web app resolves its two Key Vault references, for the admin password and the OIDC client secret. -It also counts lines in the web app console log in the TRE Log Analytics workspace. The log must have the start script lines, and no line may show a password, a client secret or a new access token. +An update changes the display name, and the case stops and starts the web app. The VM then reads the file again and deletes the repository. +Then the case makes sure that the web app resolves its three Key Vault references, for the admin password, the OIDC client secret and the database password. +It also counts lines in the web app console log in the TRE Log Analytics workspace. The log must have the start script lines, and no line can show a password, a client secret or a new access token. The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index 16907481a..ec9ba9115 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -244,7 +244,9 @@ "private Gitea API through private DNS", "anonymous and wrong-password requests denied", "private repository write/read with the admin account", - "update and app restart with retained repository data", + "update, app stop and app start with retained repository data", + "Key Vault references resolved for the admin, OIDC client and database secrets", + "no secret or new access token in the web app console log", "public network access disabled", "uninstall" ], diff --git a/e2e_tests/resources/gitea_secrets.py b/e2e_tests/resources/gitea_secrets.py index d181fec24..3937a6c5b 100644 --- a/e2e_tests/resources/gitea_secrets.py +++ b/e2e_tests/resources/gitea_secrets.py @@ -6,7 +6,7 @@ LOGS = "https://api.loganalytics.io" LOG_API = "2022-10-01" WEB_API = "2022-09-01" -SECRET_SETTINGS = ("GITEA_PASSWD", "GITEA_OPENID_CLIENT_SECRET") +SECRET_SETTINGS = ("GITEA_PASSWD", "GITEA_OPENID_CLIENT_SECRET", "GITEA__database__PASSWD") SITE_ID = re.compile(r"^/subscriptions/[0-9a-f-]+/resourceGroups/[\w.-]+/providers/Microsoft\.Web/sites/[\w-]+$") # The start script hides the secret values as ***. A line that shows a value, or a new access token, is a leak. LEAK_QUERY = """AppServiceConsoleLogs diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py index 1eb59772e..833130290 100644 --- a/e2e_tests/test_gitea.py +++ b/e2e_tests/test_gitea.py @@ -1,4 +1,4 @@ -"""Exercise a private Gitea repository across an update and an app restart.""" +"""Exercise a private Gitea repository across an update, a stop and a start of the app.""" import asyncio from functools import partial @@ -100,10 +100,12 @@ async def test_private_gitea_repository_survives_update(verify): method="PATCH", etag=gitea["_etag"], ) - # A restart makes the app read the repository from the file share and MySQL again. - await arm.request("POST", f"{site_id}/restart", WEB_API) + # A stop and a start make a new container read the repository from the file share and MySQL. + # A restart call returns before the restart, so the old container can serve the read. + await arm.request("POST", f"{site_id}/stop", WEB_API) + await arm.request("POST", f"{site_id}/start", WEB_API) await probe(phase="read") - LOGGER.info("The Gitea repository survived the update and the restart") + LOGGER.info("The Gitea repository survived the update, the stop and the start") # The web app sends its console log to Log Analytics, so the log must not show a secret. references = await key_vault_references(arm, site_id) assert set(references.values()) == {"Resolved"}, f"Gitea Key Vault references: {references}" diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py index c2f5658ca..1e90e1131 100644 --- a/e2e_tests/unit_tests/test_gitea_probe.py +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -184,4 +184,11 @@ async def test_counts_reject_a_site_id_that_changes_the_query(self): async def test_references_report_the_status_of_each_secret_setting(self): statuses = await secrets.key_vault_references(self.arm(), self.SITE) - self.assertEqual(statuses, {"GITEA_PASSWD": "Resolved", "GITEA_OPENID_CLIENT_SECRET": "Resolved"}) + self.assertEqual( + statuses, + { + "GITEA_PASSWD": "Resolved", + "GITEA_OPENID_CLIENT_SECRET": "Resolved", + "GITEA__database__PASSWD": "Resolved", + }, + ) From eaa5571eac414fc56dcfd41398ad299f4fc7bde4 Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 23:12:56 +0000 Subject: [PATCH 15/23] Make the Gitea workspace service 2.0.0, because its upgrade needs steps The upgrade restarts the web app, so the repositories on the container disk are lost unless the operator mirrors them first. The operator must also rotate the workspace client secret. The versioning rules ask for a major version for potential data loss and for changes that need an action. A major version also makes the TRE block the upgrade until an operator uses a force version update. The CHANGELOG has a breaking change entry with links to the upgrade steps and to the force version update. --- CHANGELOG.md | 10 ++++++---- docs/tre-templates/workspace-services/gitea.md | 12 +++++++----- templates/workspace_services/gitea/porter.yaml | 2 +- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e5b232550..6bcb14074 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ * Existing resources with legacy Porter parameter overrides must complete a resource upgrade before custom actions or uninstall. The current template version can be reused. See [upgrading resources](docs/tre-admins/upgrading-resources.md#upgrade-existing-resources-for-document-based-porter-parameters). (`resource_processor` 0.13.10) ([#4904](https://github.com/microsoft/AzureTRE/pull/4904)) * Set `enable_legacy_airlock` explicitly to `true` in your `config.yaml`. It currently defaults to `true` but will default to `false` in a future release; Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048)) +* `tre-workspace-service-gitea` 2.0.0 keeps the Git repositories on its file share. The upgrade restarts the web app, so the repositories on the container disk are lost. + Mirror the repositories before the upgrade, and rotate the workspace client secret after it. See the [Gitea upgrade steps](docs/tre-templates/workspace-services/gitea.md#upgrading-to-version-200). The upgrade needs a [force version update](docs/tre-admins/upgrading-resources.md#force-version-update). ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) @@ -43,14 +45,14 @@ BUG FIXES: (`tre-workspace-base` 2.12.4, `tre-workspace-unrestricted` 0.15.3, `tre-workspace-airlock-import-review` 0.18.3) ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) -* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 1.3.8) +* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. - Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) -* Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 1.3.8) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) + Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it, and the shared service printed its admin password at each restart. The start scripts of both bundles now hide the secrets in the Gitea output and create the admin user only if it does not exist. - The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 1.3.8, `tre-shared-service-gitea` 1.2.6) + The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 2.0.0, `tre-shared-service-gitea` 1.2.6) * Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) * Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 3f8ff982a..28c758588 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -29,11 +29,13 @@ The Gitea worskpace service opens outbound access to: Migrating existing Gitea services to the major version 1.0.0 is not currently supported. This is due to the breaking change in the Terraform to migrate from the deprecated mysql_server to the new mysql_flexible_server. -## Upgrading to version 1.3.8 +## Upgrading to version 2.0.0 + +Version 2.0.0 is a major version, because the upgrade needs the steps below. The TRE blocks the upgrade until you use a [force version update](../../tre-admins/upgrading-resources.md#force-version-update). ### Repositories on the file share -Before version 1.3.8, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 1.3.8, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. +Before version 2.0.0, Gitea kept the Git repositories on the container disk of the web app. Each restart of the app deleted them, but MySQL kept their records. From version 2.0.0, the repositories are on the `gitea-data` file share. The upgrade restarts the app, so the repositories on the container disk are lost. The issues, pull requests and releases stay in MySQL, and the LFS files stay on the file share. To keep the repositories, do these steps from a workspace VM: @@ -49,7 +51,7 @@ Do not run "Delete all repositories missing their Git files". This task also del ### Secrets in the console log -Before version 1.3.8, the Gitea workspace service wrote secrets to the console log of its web app. The web app sends this log to Log Analytics, so all people who can read the Log Analytics workspace of the TRE can read these secrets: +Before version 2.0.0, the Gitea workspace service wrote secrets to the console log of its web app. The web app sends this log to Log Analytics, so all people who can read the Log Analytics workspace of the TRE can read these secrets: - The client secret of the workspace application. - The password of the Gitea admin user. @@ -66,11 +68,11 @@ If a Gitea workspace service ran an earlier version, do these steps after the up 5. Sign in to Gitea as the admin user. In the user settings, open Applications and delete the access token `gitea-admin`. 6. If the people who can read Log Analytics must not get admin access to Gitea, change the password of the admin user in Gitea. Keep the new password in a safe location. The Key Vault secret `-administrator-password` keeps the old value, because the start script does not change the password of an existing admin user. -From version 1.3.8, the start script keeps a hash of the OIDC settings on the file share. If the settings change, the script stops Gitea one time, because Gitea reads them only at its start. The first start after the upgrade also does this, because the hash does not exist yet. +From version 2.0.0, the start script keeps a hash of the OIDC settings on the file share. If the settings change, the script stops Gitea one time, because Gitea reads them only at its start. The first start after the upgrade also does this, because the hash does not exist yet. ## Repairing a service that cannot reach its file share -Before version 1.3.8, Terraform used the file endpoint of the Gitea storage account to manage the `gitea-data` share. If a delete removed the private endpoint of the account and then failed, each later update or delete of the service fails with `executing request: Get "https://stgsvc.file.core.windows.net/gitea-data?restype=share": EOF`. +Before version 2.0.0, Terraform used the file endpoint of the Gitea storage account to manage the `gitea-data` share. If a delete removed the private endpoint of the account and then failed, each later update or delete of the service fails with `executing request: Get "https://stgsvc.file.core.windows.net/gitea-data?restype=share": EOF`. If you want to delete such a service: diff --git a/templates/workspace_services/gitea/porter.yaml b/templates/workspace_services/gitea/porter.yaml index 379186665..8dfa72c65 100644 --- a/templates/workspace_services/gitea/porter.yaml +++ b/templates/workspace_services/gitea/porter.yaml @@ -1,7 +1,7 @@ --- schemaVersion: 1.0.0 name: tre-workspace-service-gitea -version: 1.3.8 +version: 2.0.0 description: "A Gitea workspace service" dockerfile: Dockerfile.tmpl registry: azuretre From 3e298919164e9045c64fac33ffd85d0d65a42b9f Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 23:13:02 +0000 Subject: [PATCH 16/23] Wait for the Gitea log lines of the start after the stop The install already wrote the start script lines. So the console log check accepted zero leaks before Log Analytics had the lines of the start after the stop. The check now waits for an OIDC line with a time after the stop. Then it waits two more minutes for later lines of the same start, and it counts the leaks over the full test. --- docs/tre-developers/end-to-end-tests.md | 3 +- e2e_tests/resources/gitea_secrets.py | 44 +++++++++++++++--------- e2e_tests/test_gitea.py | 4 ++- e2e_tests/unit_tests/test_gitea_probe.py | 25 +++++++++----- 4 files changed, 49 insertions(+), 27 deletions(-) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index acbaede4f..c0105e589 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -196,7 +196,8 @@ It also makes sure that Gitea rejects a repository create without credentials (H The VM then creates a private repository with the admin account and commits a file. An anonymous read of the repository gets HTTP 404. An update changes the display name, and the case stops and starts the web app. The VM then reads the file again and deletes the repository. Then the case makes sure that the web app resolves its three Key Vault references, for the admin password, the OIDC client secret and the database password. -It also counts lines in the web app console log in the TRE Log Analytics workspace. The log must have the start script lines, and no line can show a password, a client secret or a new access token. +It also counts lines in the web app console log in the TRE Log Analytics workspace. No line can show a password, a client secret or a new access token. +The count waits for the start script lines of the start after the stop, and then for two more minutes, because the lines of one start can arrive at different times. The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. diff --git a/e2e_tests/resources/gitea_secrets.py b/e2e_tests/resources/gitea_secrets.py index 3937a6c5b..13769d356 100644 --- a/e2e_tests/resources/gitea_secrets.py +++ b/e2e_tests/resources/gitea_secrets.py @@ -1,6 +1,7 @@ """Make sure that the Gitea web app keeps its secrets out of its console log.""" import asyncio +from datetime import timezone import re LOGS = "https://api.loganalytics.io" @@ -12,6 +13,7 @@ LEAK_QUERY = """AppServiceConsoleLogs | where _ResourceId =~ '{site_id}' | summarize started = countif(ResultDescription contains 'Adding admin user'), + restarted = countif(TimeGenerated >= datetime({since}) and ResultDescription contains 'Configuring OIDC'), leaks = countif((ResultDescription contains '--password=' and ResultDescription !contains '--password=***') or (ResultDescription contains '--secret ' and ResultDescription !contains '--secret ***') or ResultDescription contains 'Access token was successfully created')""" @@ -26,25 +28,33 @@ async def key_vault_references(arm, site_id): return statuses -async def console_log_counts(arm, workspace_id, site_id, minutes=15): - """Count the start lines and the secret lines in the console log. The query returns counts only.""" +async def console_log_counts(arm, workspace_id, site_id, since, minutes=15, settle=120): + """Count the start lines and the secret lines in the console log. The query returns counts only. + + Wait for the OIDC line of a start after `since`. Then wait `settle` seconds more, because the lines + of one start can arrive in Log Analytics at different times. + """ if not SITE_ID.match(site_id): raise ValueError("Expected a web app resource ID") customer_id = (await arm.request("GET", workspace_id, LOG_API))["properties"]["customerId"] - query = LEAK_QUERY.format(site_id=site_id.lower()) + start = since.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + query = LEAK_QUERY.format(site_id=site_id.lower(), since=start) + + async def counts(): + token = await asyncio.to_thread(arm.credential.get_token, f"{LOGS}/.default") + response = await arm.client.post( + f"{LOGS}/v1/workspaces/{customer_id}/query", + headers={"Authorization": f"Bearer {token.token}"}, + json={"query": query, "timespan": "P1D"}, + ) + if response.status_code != 200: + raise RuntimeError(f"Log Analytics query failed with HTTP {response.status_code}") + table = response.json()["tables"][0] + return dict(zip([column["name"] for column in table["columns"]], table["rows"][0])) + async with asyncio.timeout(minutes * 60): - while True: - token = await asyncio.to_thread(arm.credential.get_token, f"{LOGS}/.default") - response = await arm.client.post( - f"{LOGS}/v1/workspaces/{customer_id}/query", - headers={"Authorization": f"Bearer {token.token}"}, - json={"query": query, "timespan": "P1D"}, - ) - if response.status_code != 200: - raise RuntimeError(f"Log Analytics query failed with HTTP {response.status_code}") - table = response.json()["tables"][0] - counts = dict(zip([column["name"] for column in table["columns"]], table["rows"][0])) - # Log Analytics can show a line some minutes after the app writes it. - if counts["started"]: - return counts + # Log Analytics can show a line some minutes after the app writes it. + while not (await counts())["restarted"]: await asyncio.sleep(60) + await asyncio.sleep(settle) + return await counts() diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py index 833130290..b40fcc43a 100644 --- a/e2e_tests/test_gitea.py +++ b/e2e_tests/test_gitea.py @@ -1,6 +1,7 @@ """Exercise a private Gitea repository across an update, a stop and a start of the app.""" import asyncio +from datetime import datetime, timezone from functools import partial import logging import os @@ -102,6 +103,7 @@ async def test_private_gitea_repository_survives_update(verify): ) # A stop and a start make a new container read the repository from the file share and MySQL. # A restart call returns before the restart, so the old container can serve the read. + restarted_at = datetime.now(timezone.utc) await arm.request("POST", f"{site_id}/stop", WEB_API) await arm.request("POST", f"{site_id}/start", WEB_API) await probe(phase="read") @@ -113,7 +115,7 @@ async def test_private_gitea_repository_survives_update(verify): f"/subscriptions/{os.environ['ARM_SUBSCRIPTION_ID']}/resourceGroups/rg-{config.TRE_ID}" f"/providers/Microsoft.OperationalInsights/workspaces/log-{config.TRE_ID}" ) - counts = await console_log_counts(arm, log_workspace, site_id) + counts = await console_log_counts(arm, log_workspace, site_id, restarted_at) LOGGER.info("Gitea console log lines: %s", counts) assert counts["leaks"] == 0, f"The Gitea console log has {counts['leaks']} lines with a secret" # Check the network settings last, so that a failure here keeps the functional evidence above. diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py index 1e90e1131..28708b953 100644 --- a/e2e_tests/unit_tests/test_gitea_probe.py +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -3,6 +3,7 @@ import json from pathlib import Path, PurePosixPath import re +from datetime import datetime, timedelta, timezone from types import SimpleNamespace import unittest from unittest.mock import AsyncMock, Mock, patch @@ -150,6 +151,7 @@ class SecretLogTests(unittest.IsolatedAsyncioTestCase): WORKSPACE = ( "/subscriptions/sub/resourceGroups/rg-tretest/providers/Microsoft.OperationalInsights/workspaces/log-tretest" ) + SINCE = datetime(2026, 10, 28, 23, 30, 5, tzinfo=timezone(timedelta(hours=1))) @staticmethod def arm(*rows): @@ -157,7 +159,11 @@ def arm(*rows): SimpleNamespace( status_code=200, json=Mock( - return_value={"tables": [{"columns": [{"name": "started"}, {"name": "leaks"}], "rows": [row]}]} + return_value={ + "tables": [ + {"columns": [{"name": "started"}, {"name": "restarted"}, {"name": "leaks"}], "rows": [row]} + ] + } ), ) for row in rows @@ -168,18 +174,21 @@ def arm(*rows): client=SimpleNamespace(post=AsyncMock(side_effect=responses)), ) - async def test_counts_wait_until_the_start_lines_arrive(self): - arm = self.arm([0, 0], [2, 0]) + async def test_counts_wait_for_the_start_after_the_restart(self): + # The install lines arrive first. The lines of the later start can arrive some minutes after them. + arm = self.arm([1, 0, 0], [2, 1, 0], [2, 1, 1]) with patch("e2e_tests.resources.gitea_secrets.asyncio.sleep", AsyncMock()) as sleep: - counts = await secrets.console_log_counts(arm, self.WORKSPACE, self.SITE) - self.assertEqual(counts, {"started": 2, "leaks": 0}) - sleep.assert_awaited_once() - self.assertIn(self.SITE.lower(), arm.client.post.call_args.kwargs["json"]["query"]) + counts = await secrets.console_log_counts(arm, self.WORKSPACE, self.SITE, self.SINCE) + self.assertEqual(counts, {"started": 2, "restarted": 1, "leaks": 1}) + self.assertEqual([call.args[0] for call in sleep.await_args_list], [60, 120]) + query = arm.client.post.call_args.kwargs["json"]["query"] + self.assertIn(self.SITE.lower(), query) + self.assertIn("TimeGenerated >= datetime(2026-10-28T22:30:05Z)", query) async def test_counts_reject_a_site_id_that_changes_the_query(self): arm = self.arm() with self.assertRaises(ValueError): - await secrets.console_log_counts(arm, self.WORKSPACE, f"{self.SITE}' or true or '") + await secrets.console_log_counts(arm, self.WORKSPACE, f"{self.SITE}' or true or '", self.SINCE) arm.client.post.assert_not_awaited() async def test_references_report_the_status_of_each_secret_setting(self): From 8329404a177a584e95f05c4a128405842cdc7663 Mon Sep 17 00:00:00 2001 From: Jay W Date: Wed, 28 Oct 2026 23:19:45 +0000 Subject: [PATCH 17/23] Remove the leaked service principal password in the Gitea steps If Terraform registered the workspace application, the client secret is a password of the service principal (azuread_service_principal_password), not of the app registration. A delete of an app registration secret left the leaked password valid. The steps now list the passwords of the service principal through Microsoft Graph, find the leaked one by its hint, and remove it. azuread 3.3.0 then removes the password from the state, so the next workspace update makes a new one. --- docs/tre-templates/workspace-services/gitea.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 28c758588..06db75ce3 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -60,7 +60,11 @@ Before version 2.0.0, the Gitea workspace service wrote secrets to the console l If a Gitea workspace service ran an earlier version, do these steps after the upgrade: 1. Make a new client secret for the workspace application: - - If Terraform registered the workspace application (`register_aad_application` is `true`), delete its client secret in Microsoft Entra ID. Then update the workspace. Terraform makes a new client secret and writes it to the workspace Key Vault secret `workspace-client-secret`. + - If Terraform registered the workspace application (`register_aad_application` is `true`), the client secret is a password of its service principal, not of the app registration. The Microsoft Entra admin center does not show it. Do these steps: + 1. List the passwords of the service principal with `az rest --method GET --url "https://graph.microsoft.com/v1.0/servicePrincipals(appId='')" --query "passwordCredentials[].{keyId:keyId,hint:hint}"`. + 2. Find the password whose `hint` is the same as the first three characters of the workspace Key Vault secret `workspace-client-secret`. + 3. Remove it with `az rest --method POST --url "https://graph.microsoft.com/v1.0/servicePrincipals(appId='')/removePassword" --body '{"keyId": ""}'`. + 4. Update the workspace. Terraform makes a new password and writes it to `workspace-client-secret`. - If you registered the workspace application, add a new client secret in Microsoft Entra ID. Update the workspace property `client_secret` with the new value. Then delete the old client secret. 2. Update each Guacamole and OHDSI workspace service in the workspace. These services use the version of the secret that was current when they deployed. 3. Make the Gitea web app read the new client secret from the Key Vault. Send a POST request to `https://management.azure.com//config/configreferences/appsettings/refresh?api-version=2022-03-01`, for example with `az rest`. From 95a2efaf18dba25f7f66bf7397924ed0bc0cbd40 Mon Sep 17 00:00:00 2001 From: Jay W Date: Thu, 29 Oct 2026 09:59:20 +0000 Subject: [PATCH 18/23] Test the Gitea stop when the OIDC settings change The Gitea check now gives the web app a new OIDC client ID after the update. The start script must find the new hash of the OIDC settings, stop Gitea one time, and App Service must start it again. The check reads the stop line and the next start line in the console log. After the stop and the start of the app, the check makes sure that the log shows only one stop. The last start has the same OIDC settings, so it must not stop Gitea again. --- CHANGELOG.md | 3 +- docs/tre-developers/end-to-end-tests.md | 5 ++- e2e_tests/bundle_coverage.json | 1 + e2e_tests/resources/gitea_secrets.py | 53 ++++++++++++++++++++---- e2e_tests/test_gitea.py | 26 +++++++++--- e2e_tests/unit_tests/test_gitea_probe.py | 52 +++++++++++++++++++---- 6 files changed, 114 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6bcb14074..178344931 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,8 @@ ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) * Add a private MySQL lifecycle check. A Nexus-backed Linux VM writes a row through private DNS, the SKU changes from Burstable to General Purpose, and the VM reads the row again. ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Run the `stop`, `start` and `reset_password` actions in the Guacamole Linux VM check, and make sure of the VM power state and the new Key Vault secret version through Azure. ([#5145](https://github.com/microsoft/AzureTRE/issues/5145)) -* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. The VM reads the file again after an update, a stop and a start of the app. The check also makes sure that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) +* Add a private Gitea lifecycle check. A workspace VM uses the Gitea API through private DNS, gets HTTP 401 without credentials and with a wrong password, and commits a file to a private repository. + The VM reads the file again after an update, a change of the OIDC client ID, a stop and a start of the app. The check also makes sure that the OIDC change stops Gitea one time and that the console log of the web app shows no secret. ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Select individual bundle tests and record lifecycle evidence for base workspaces, private Azure SQL, Airlock, Linux VMs and certificate reuse across reinstallation. ([#5255](https://github.com/microsoft/AzureTRE/pull/5255)) * Enable `user_management_enabled`, `auto_workspace_group_creation`, and `auto_grant_workspace_consent` by default in `config.sample.yaml` for a better out-of-the-box experience ([#5050](https://github.com/microsoft/AzureTRE/issues/5050)) * Add shared repository instructions and TRE-prefixed component skills for agent-assisted development, security review, validation and troubleshooting. ([#5111](https://github.com/microsoft/AzureTRE/issues/5111)) diff --git a/docs/tre-developers/end-to-end-tests.md b/docs/tre-developers/end-to-end-tests.md index c0105e589..306f4ca15 100644 --- a/docs/tre-developers/end-to-end-tests.md +++ b/docs/tre-developers/end-to-end-tests.md @@ -194,10 +194,13 @@ It gives the VM identity a temporary Key Vault Secrets User role on the secret t From the VM, the case makes sure that the Gitea host resolves to a private address and that the Gitea API is available. It also makes sure that Gitea rejects a repository create without credentials (HTTP 401) and a request with a wrong password (HTTP 401). The VM then creates a private repository with the admin account and commits a file. An anonymous read of the repository gets HTTP 404. -An update changes the display name, and the case stops and starts the web app. The VM then reads the file again and deletes the repository. +An update changes the display name. Then the case gives the web app a new OIDC client ID. +The start script must stop Gitea one time for the new OIDC settings, and App Service must start it again. +Then the case stops and starts the web app. The VM reads the file again and deletes the repository. Then the case makes sure that the web app resolves its three Key Vault references, for the admin password, the OIDC client secret and the database password. It also counts lines in the web app console log in the TRE Log Analytics workspace. No line can show a password, a client secret or a new access token. The count waits for the start script lines of the start after the stop, and then for two more minutes, because the lines of one start can arrive at different times. +The last start has the same OIDC settings, so the log must show only one stop for the new OIDC settings. The query returns counts only. The CI identity needs read access to the Log Analytics workspace. At the end, the case makes sure that the web app and MySQL have public network access disabled, and that the storage firewall denies by default. The Gitea job uses the SQL time limits with its own `GITEA_VALIDATION_DEADLINE`. diff --git a/e2e_tests/bundle_coverage.json b/e2e_tests/bundle_coverage.json index ec9ba9115..49194451c 100644 --- a/e2e_tests/bundle_coverage.json +++ b/e2e_tests/bundle_coverage.json @@ -245,6 +245,7 @@ "anonymous and wrong-password requests denied", "private repository write/read with the admin account", "update, app stop and app start with retained repository data", + "one Gitea stop after an OIDC setting change", "Key Vault references resolved for the admin, OIDC client and database secrets", "no secret or new access token in the web app console log", "public network access disabled", diff --git a/e2e_tests/resources/gitea_secrets.py b/e2e_tests/resources/gitea_secrets.py index 13769d356..2b93996fd 100644 --- a/e2e_tests/resources/gitea_secrets.py +++ b/e2e_tests/resources/gitea_secrets.py @@ -3,6 +3,7 @@ import asyncio from datetime import timezone import re +from uuid import uuid4 LOGS = "https://api.loganalytics.io" LOG_API = "2022-10-01" @@ -17,6 +18,14 @@ leaks = countif((ResultDescription contains '--password=' and ResultDescription !contains '--password=***') or (ResultDescription contains '--secret ' and ResultDescription !contains '--secret ***') or ResultDescription contains 'Access token was successfully created')""" +# The start script writes this line when the OIDC settings change, and then stops Gitea. +OIDC_STOP = "Stopping Gitea, so that it starts again with the new OIDC settings" +OIDC_QUERY = """AppServiceConsoleLogs +| where _ResourceId =~ '{site_id}' and TimeGenerated >= datetime({since}) +| summarize stops = countif(ResultDescription contains '{stop}'), + last_stop = maxif(TimeGenerated, ResultDescription contains '{stop}'), + last_start = maxif(TimeGenerated, ResultDescription contains 'Configuring OIDC') +| project stops, started_after_stop = iff(isnotnull(last_stop) and isnotnull(last_start) and last_start > last_stop, 1, 0)""" async def key_vault_references(arm, site_id): @@ -28,17 +37,19 @@ async def key_vault_references(arm, site_id): return statuses -async def console_log_counts(arm, workspace_id, site_id, since, minutes=15, settle=120): - """Count the start lines and the secret lines in the console log. The query returns counts only. - - Wait for the OIDC line of a start after `since`. Then wait `settle` seconds more, because the lines - of one start can arrive in Log Analytics at different times. - """ +async def change_oidc_client_id(arm, site_id): + """Give the web app a new OIDC client ID. A change of the app settings restarts the web app.""" if not SITE_ID.match(site_id): raise ValueError("Expected a web app resource ID") + # The secret settings are Key Vault references, so the list does not contain a secret value. + settings = (await arm.request("POST", f"{site_id}/config/appsettings/list", WEB_API))["properties"] + settings["GITEA_OPENID_CLIENT_ID"] = str(uuid4()) + await arm.request("PUT", f"{site_id}/config/appsettings", WEB_API, body={"properties": settings}) + + +async def wait_for_counts(arm, workspace_id, query, ready, minutes, settle): + """Run a count query until `ready` accepts the counts. Then wait `settle` seconds and count again.""" customer_id = (await arm.request("GET", workspace_id, LOG_API))["properties"]["customerId"] - start = since.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") - query = LEAK_QUERY.format(site_id=site_id.lower(), since=start) async def counts(): token = await asyncio.to_thread(arm.credential.get_token, f"{LOGS}/.default") @@ -54,7 +65,31 @@ async def counts(): async with asyncio.timeout(minutes * 60): # Log Analytics can show a line some minutes after the app writes it. - while not (await counts())["restarted"]: + while not ready(await counts()): await asyncio.sleep(60) await asyncio.sleep(settle) return await counts() + + +async def console_log_counts(arm, workspace_id, site_id, since, minutes=15, settle=120): + """Count the start lines and the secret lines in the console log. The query returns counts only. + + Wait for the OIDC line of a start after `since`. Then wait `settle` seconds more, because the lines + of one start can arrive in Log Analytics at different times. + """ + if not SITE_ID.match(site_id): + raise ValueError("Expected a web app resource ID") + start = since.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + query = LEAK_QUERY.format(site_id=site_id.lower(), since=start) + return await wait_for_counts(arm, workspace_id, query, lambda counts: counts["restarted"], minutes, settle) + + +async def oidc_stop_counts(arm, workspace_id, site_id, since, minutes=20, settle=120): + """Count the stops for new OIDC settings after `since`. Wait for a stop and for a start after it.""" + if not SITE_ID.match(site_id): + raise ValueError("Expected a web app resource ID") + start = since.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + query = OIDC_QUERY.format(site_id=site_id.lower(), since=start, stop=OIDC_STOP) + return await wait_for_counts( + arm, workspace_id, query, lambda counts: counts["stops"] and counts["started_after_stop"], minutes, settle + ) diff --git a/e2e_tests/test_gitea.py b/e2e_tests/test_gitea.py index b40fcc43a..12c6b050a 100644 --- a/e2e_tests/test_gitea.py +++ b/e2e_tests/test_gitea.py @@ -13,7 +13,12 @@ from e2e_tests.conftest import get_workspace_owner_token from e2e_tests.resources import strings from e2e_tests.resources.gitea_probe import run_probe -from e2e_tests.resources.gitea_secrets import console_log_counts, key_vault_references +from e2e_tests.resources.gitea_secrets import ( + change_oidc_client_id, + console_log_counts, + key_vault_references, + oidc_stop_counts, +) from e2e_tests.resources.resource import get_resource, post_resource from e2e_tests.resources.sql_lifecycle import sql_lifecycle from e2e_tests.resources.sql_probe import arm_client, COMPUTE_API, secret_access, validate_vm @@ -67,6 +72,10 @@ async def test_private_gitea_repository_survives_update(verify): assert gitea["properties"]["connection_uri"] == f"{gitea_url}/" vault = f"kv-{f'{config.TRE_ID}-ws-{workspace_id[-4:]}'[-20:]}".lower() azure_vm_id = vm_properties["azure_resource_id"] + log_workspace = ( + f"/subscriptions/{os.environ['ARM_SUBSCRIPTION_ID']}/resourceGroups/rg-{config.TRE_ID}" + f"/providers/Microsoft.OperationalInsights/workspaces/log-{config.TRE_ID}" + ) async with arm_client() as arm: vm = await arm.request("GET", azure_vm_id, COMPUTE_API) group = validate_vm(vm, azure_vm_id, subscription_id, config.TRE_ID, workspace_id, guacamole_id, vm_id) @@ -101,23 +110,28 @@ async def test_private_gitea_repository_survives_update(verify): method="PATCH", etag=gitea["_etag"], ) + # A new OIDC client ID changes the hash of the OIDC settings on the file share. The start script + # then stops Gitea one time, and App Service starts it again. + changed_at = datetime.now(timezone.utc) + await change_oidc_client_id(arm, site_id) + await oidc_stop_counts(arm, log_workspace, site_id, changed_at, settle=0) + LOGGER.info("Gitea stopped for the new OIDC settings and started again") # A stop and a start make a new container read the repository from the file share and MySQL. # A restart call returns before the restart, so the old container can serve the read. restarted_at = datetime.now(timezone.utc) await arm.request("POST", f"{site_id}/stop", WEB_API) await arm.request("POST", f"{site_id}/start", WEB_API) await probe(phase="read") - LOGGER.info("The Gitea repository survived the update, the stop and the start") + LOGGER.info("The Gitea repository survived the update, the OIDC change, the stop and the start") # The web app sends its console log to Log Analytics, so the log must not show a secret. references = await key_vault_references(arm, site_id) assert set(references.values()) == {"Resolved"}, f"Gitea Key Vault references: {references}" - log_workspace = ( - f"/subscriptions/{os.environ['ARM_SUBSCRIPTION_ID']}/resourceGroups/rg-{config.TRE_ID}" - f"/providers/Microsoft.OperationalInsights/workspaces/log-{config.TRE_ID}" - ) counts = await console_log_counts(arm, log_workspace, site_id, restarted_at) LOGGER.info("Gitea console log lines: %s", counts) assert counts["leaks"] == 0, f"The Gitea console log has {counts['leaks']} lines with a secret" + # The start after the stop has the same OIDC settings, so it must not stop Gitea again. + stops = await oidc_stop_counts(arm, log_workspace, site_id, changed_at, settle=0) + assert stops["stops"] == 1, f"Gitea stopped {stops['stops']} times for one change of the OIDC settings" # Check the network settings last, so that a failure here keeps the functional evidence above. app = await arm.request("GET", site_id, WEB_API) server = await arm.request( diff --git a/e2e_tests/unit_tests/test_gitea_probe.py b/e2e_tests/unit_tests/test_gitea_probe.py index 28708b953..5d0aa2f4e 100644 --- a/e2e_tests/unit_tests/test_gitea_probe.py +++ b/e2e_tests/unit_tests/test_gitea_probe.py @@ -7,7 +7,7 @@ from types import SimpleNamespace import unittest from unittest.mock import AsyncMock, Mock, patch -from uuid import uuid4 +from uuid import UUID, uuid4 from jsonschema import validate import yaml @@ -154,17 +154,11 @@ class SecretLogTests(unittest.IsolatedAsyncioTestCase): SINCE = datetime(2026, 10, 28, 23, 30, 5, tzinfo=timezone(timedelta(hours=1))) @staticmethod - def arm(*rows): + def arm(*rows, columns=("started", "restarted", "leaks")): responses = [ SimpleNamespace( status_code=200, - json=Mock( - return_value={ - "tables": [ - {"columns": [{"name": "started"}, {"name": "restarted"}, {"name": "leaks"}], "rows": [row]} - ] - } - ), + json=Mock(return_value={"tables": [{"columns": [{"name": name} for name in columns], "rows": [row]}]}), ) for row in rows ] @@ -201,3 +195,43 @@ async def test_references_report_the_status_of_each_secret_setting(self): "GITEA__database__PASSWD": "Resolved", }, ) + + async def test_oidc_counts_wait_for_a_start_after_the_stop(self): + # The stop line arrives first. The lines of the start after the stop can arrive some minutes after it. + arm = self.arm([0, 0], [1, 0], [1, 1], [1, 1], columns=("stops", "started_after_stop")) + with patch("e2e_tests.resources.gitea_secrets.asyncio.sleep", AsyncMock()) as sleep: + counts = await secrets.oidc_stop_counts(arm, self.WORKSPACE, self.SITE, self.SINCE) + self.assertEqual(counts, {"stops": 1, "started_after_stop": 1}) + self.assertEqual([call.args[0] for call in sleep.await_args_list], [60, 60, 120]) + query = arm.client.post.call_args.kwargs["json"]["query"] + self.assertIn(self.SITE.lower(), query) + self.assertIn("TimeGenerated >= datetime(2026-10-28T22:30:05Z)", query) + self.assertIn("contains 'Stopping Gitea, so that it starts again with the new OIDC settings'", query) + + async def test_oidc_stop_line_is_the_line_of_the_start_script(self): + script = (ROOT / "templates/workspace_services/gitea/docker/configure_gitea.sh").read_text() + self.assertIn(f'echo "{secrets.OIDC_STOP}"', script) + + async def test_oidc_counts_reject_a_site_id_that_changes_the_query(self): + arm = self.arm() + with self.assertRaises(ValueError): + await secrets.oidc_stop_counts(arm, self.WORKSPACE, f"{self.SITE}' or true or '", self.SINCE) + arm.client.post.assert_not_awaited() + + async def test_oidc_change_keeps_the_other_app_settings(self): + settings = {"GITEA_OPENID_CLIENT_ID": "old", "GITEA_PASSWD": "@Microsoft.KeyVault(SecretUri=x)"} + arm = SimpleNamespace(request=AsyncMock(side_effect=[{"properties": dict(settings)}, {}])) + await secrets.change_oidc_client_id(arm, self.SITE) + listed, put = arm.request.await_args_list + self.assertEqual(listed.args[:2], ("POST", f"{self.SITE}/config/appsettings/list")) + self.assertEqual(put.args[:2], ("PUT", f"{self.SITE}/config/appsettings")) + changed = put.kwargs["body"]["properties"] + self.assertEqual(changed["GITEA_PASSWD"], settings["GITEA_PASSWD"]) + self.assertNotEqual(changed["GITEA_OPENID_CLIENT_ID"], "old") + self.assertEqual(str(UUID(changed["GITEA_OPENID_CLIENT_ID"])), changed["GITEA_OPENID_CLIENT_ID"]) + + async def test_oidc_change_rejects_a_site_id_that_is_not_a_web_app(self): + arm = SimpleNamespace(request=AsyncMock()) + with self.assertRaises(ValueError): + await secrets.change_oidc_client_id(arm, VM_ID) + arm.request.assert_not_awaited() From 7853874a7a339dc238c3a0821aec678baa4e9d26 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 16:25:10 +0000 Subject: [PATCH 19/23] Rotate a registered workspace secret through the Key Vault The Gitea document now tells the operator of a registered workspace application to write the new client secret to the workspace Key Vault secret workspace-client-secret. The Gitea web app reads that secret through a Key Vault reference, and the other steps refresh it. --- docs/tre-templates/workspace-services/gitea.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tre-templates/workspace-services/gitea.md b/docs/tre-templates/workspace-services/gitea.md index 06db75ce3..27efd665d 100644 --- a/docs/tre-templates/workspace-services/gitea.md +++ b/docs/tre-templates/workspace-services/gitea.md @@ -65,7 +65,7 @@ If a Gitea workspace service ran an earlier version, do these steps after the up 2. Find the password whose `hint` is the same as the first three characters of the workspace Key Vault secret `workspace-client-secret`. 3. Remove it with `az rest --method POST --url "https://graph.microsoft.com/v1.0/servicePrincipals(appId='')/removePassword" --body '{"keyId": ""}'`. 4. Update the workspace. Terraform makes a new password and writes it to `workspace-client-secret`. - - If you registered the workspace application, add a new client secret in Microsoft Entra ID. Update the workspace property `client_secret` with the new value. Then delete the old client secret. + - If you registered the workspace application, add a new client secret in Microsoft Entra ID. Write the new value to the workspace Key Vault secret `workspace-client-secret`. You need a role that can set secrets and network access to the Key Vault. Then delete the old client secret. 2. Update each Guacamole and OHDSI workspace service in the workspace. These services use the version of the secret that was current when they deployed. 3. Make the Gitea web app read the new client secret from the Key Vault. Send a POST request to `https://management.azure.com//config/configreferences/appsettings/refresh?api-version=2022-03-01`, for example with `az rest`. 4. Restart the Gitea web app. At the start, the start script gives the OIDC source the new client secret. Then it stops Gitea one time, and App Service starts it again with the new secret. From 72ea49666a693ea8330f2d0f9a28bf0e431b38bb Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 16:28:00 +0000 Subject: [PATCH 20/23] Link the new Gitea CHANGELOG entries to their issue Five new entries had no issue link. They now link to #5283. --- CHANGELOG.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 178344931..decc94575 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ * Set `enable_legacy_airlock` explicitly to `true` in your `config.yaml`. It currently defaults to `true` but will default to `false` in a future release; Setting to `false` will delete existing airlock storage accounts and must only be done once all workspaces use the v2 airlock. ([#5048](https://github.com/microsoft/AzureTRE/pull/5048)) * `tre-workspace-service-gitea` 2.0.0 keeps the Git repositories on its file share. The upgrade restarts the web app, so the repositories on the container disk are lost. - Mirror the repositories before the upgrade, and rotate the workspace client secret after it. See the [Gitea upgrade steps](docs/tre-templates/workspace-services/gitea.md#upgrading-to-version-200). The upgrade needs a [force version update](docs/tre-admins/upgrading-resources.md#force-version-update). + Mirror the repositories before the upgrade, and rotate the workspace client secret after it. See the [Gitea upgrade steps](docs/tre-templates/workspace-services/gitea.md#upgrading-to-version-200). The upgrade needs a [force version update](docs/tre-admins/upgrading-resources.md#force-version-update). ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) ENHANCEMENTS: * Register the OpenAI bundle for branch validation, add GPT-5.1 as the default model and check regional model and quota prerequisites before its private lifecycle test. (`tre-workspace-service-openai` 1.1.0) ([#5150](https://github.com/microsoft/AzureTRE/issues/5150)) @@ -46,16 +46,16 @@ BUG FIXES: (`tre-workspace-base` 2.12.4, `tre-workspace-unrestricted` 0.15.3, `tre-workspace-airlock-import-review` 0.18.3) ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Clean up backup-test workspaces after failed provisioning. ([#5135](https://github.com/microsoft/AzureTRE/issues/5135)) * Create the OpenAI private endpoint after the model deployment, so that Azure does not reject it while the deployment updates the account. (`tre-workspace-service-openai` 1.1.0) ([#5249](https://github.com/microsoft/AzureTRE/issues/5249)) -* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) +* Give the Gitea workspace service identity the Key Vault role only on the three secrets that its web app reads. It had the role on the whole workspace Key Vault. (`tre-workspace-service-gitea` 2.0.0) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Disable public network access on the MySQL flexible server, so that clients can connect only through the private endpoint. Move the MySQL bundle to azurerm 4.27.0, which has this setting. An upgrade changes the setting in place. (`tre-workspace-service-mysql` 1.0.16) ([#5148](https://github.com/microsoft/AzureTRE/issues/5148)) * Create the Gitea workspace service file share through Resource Manager. Terraform used the file endpoint of the storage account, which the TRE firewall blocks until the private endpoint and its DNS record exist. So an install can fail, and after a delete removes the private endpoint, each later update or delete fails. Also disable public network access on the Gitea MySQL server. An upgrade keeps the existing share and changes the server setting in place. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Keep the Git repositories of the Gitea workspace service on its file share. Gitea used its image default `/data/git/repositories`, which is on the container disk. So each restart of the web app deleted all repositories. MySQL kept their records, and Gitea returned HTTP 500 for them. The upgrade restarts the app, so copy the repositories before the upgrade. (`tre-workspace-service-gitea` 2.0.0) ([#5141](https://github.com/microsoft/AzureTRE/issues/5141)) * Stop writing Gitea secrets to the console log, which the web app sends to Log Analytics. The start script of the Gitea workspace service printed the admin password and the client secret of the workspace application. Gitea also prints the full command line when a command fails, so each restart printed the admin password again. The admin user create in both Gitea bundles also made an access token and printed it, and the shared service printed its admin password at each restart. The start scripts of both bundles now hide the secrets in the Gitea output and create the admin user only if it does not exist. - The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 2.0.0, `tre-shared-service-gitea` 1.2.6) -* Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) -* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) + The workspace start script gives the OIDC source the current client secret at each start, and it stops Gitea one time when the OIDC settings change. The web app reads the client secret through a Key Vault reference. (`tre-workspace-service-gitea` 2.0.0, `tre-shared-service-gitea` 1.2.6) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) +* Give the Gitea shared service identity the Key Vault role only on the two secrets that its web app reads. It had the role on the whole core Key Vault. (`tre-shared-service-gitea` 1.2.6) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) +* Give the Nexus VM identity the Key Vault role only on the Nexus admin password and the Nexus TLS certificate. It had the role on the whole core Key Vault. (`tre-shared-service-sonatype-nexus` 3.11.4) ([#5283](https://github.com/microsoft/AzureTRE/issues/5283)) * Update `vite` from 7.3.6 to 7.3.7 and the UI Vitest packages from 4.1.11 to 4.1.12 to address security advisories in the development server and the test mocker. (`ui` 0.10.4) ([#5274](https://github.com/microsoft/AzureTRE/pull/5274)) * Update `aiohttp` from 3.14.3 to 3.14.4 in `api_app`, `resource_processor`, and the CLI to address security advisories. (`API` 0.28.7, `resource_processor` 0.13.15, `CLI` 0.2.13) ([#5270](https://github.com/microsoft/AzureTRE/pull/5270)) * Rename a test variable in the E2E polling tests so that CodeQL stops reporting false clear-text logging alerts. ([#5266](https://github.com/microsoft/AzureTRE/pull/5266)) From dabfe28c79c480afa58073d9b41cea474c9fd264 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 16:37:56 +0000 Subject: [PATCH 21/23] Make the shared Gitea admin script executable The Lint job requires an executable bit on each shell script. The Dockerfile runs the script with bash, so the image does not change its behaviour. --- templates/shared_services/gitea/docker/configure_gitea_admin.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 templates/shared_services/gitea/docker/configure_gitea_admin.sh diff --git a/templates/shared_services/gitea/docker/configure_gitea_admin.sh b/templates/shared_services/gitea/docker/configure_gitea_admin.sh old mode 100644 new mode 100755 From c31938d8cb566547b3c0fe5b479258cfc457d0f6 Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 17:06:33 +0000 Subject: [PATCH 22/23] Explain the bundle test deadlines in the workflow A reviewer asked what 270 * 60 means. The comment gives the job timeout, the deadline and the time that the clean-up and the report keep. --- .github/workflows/deploy_tre_reusable.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/deploy_tre_reusable.yml b/.github/workflows/deploy_tre_reusable.yml index 5db123799..65ebe1539 100644 --- a/.github/workflows/deploy_tre_reusable.yml +++ b/.github/workflows/deploy_tre_reusable.yml @@ -1069,6 +1069,9 @@ jobs: [deploy_shared_services, register_bundles, register_user_resource_bundles, e2e_tests_smoke] timeout-minutes: 300 steps: + # The job timeout is 300 minutes. Each step below sets an absolute deadline 270 minutes + # (270 * 60 seconds) after the job start. The lifecycle helper keeps the last 120 minutes + # before the deadline for the clean-up. The 30 minutes after the deadline are for the report. - name: Reserve SQL cleanup and reporting time if: inputs.e2eBundle == 'tre-workspace-service-azuresql' run: echo "SQL_VALIDATION_DEADLINE=$(( $(date +%s) + 270 * 60 ))" >> "$GITHUB_ENV" From 30f9562274aeb1185a36c012dd8bac9fa4e1e7ad Mon Sep 17 00:00:00 2001 From: Jay W Date: Fri, 30 Oct 2026 17:06:47 +0000 Subject: [PATCH 23/23] Tell operators how to replace the migration token after a password change gitea_migrate_repo.sh makes a new access token with the admin password from the Key Vault. After an operator changes the password in Gitea, that sign-in fails with HTTP 401. The shared Gitea document now tells the operator to make the token with the new password and store it in the Key Vault, and not to write the new password to the Key Vault, because the next update writes the old value back. --- docs/tre-templates/shared-services/gitea.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/tre-templates/shared-services/gitea.md b/docs/tre-templates/shared-services/gitea.md index c84bac115..dcd17785d 100644 --- a/docs/tre-templates/shared-services/gitea.md +++ b/docs/tre-templates/shared-services/gitea.md @@ -63,3 +63,12 @@ If a Gitea shared service ran an earlier version, do these steps after the upgra 1. Sign in to Gitea as `giteaadmin`. In the user settings, open Applications and delete the access token `gitea-admin`. Keep the token with the name `giteaadmin`, because `gitea_migrate_repo.sh` uses it. 2. If the people who can read Log Analytics must not get admin access to Gitea, change the password of `giteaadmin` in Gitea. Keep the new password in a safe location. The Key Vault secret `gitea--administrator-password` keeps the old value, because the start script does not change the password of an existing admin user. +3. If you changed the password, `gitea_migrate_repo.sh` cannot make a new access token, because it signs in with the old password from the Key Vault: + - Do not set the Key Vault secret `gitea--admin-token` to `null`. + - Do not write the new password to `gitea--administrator-password`. The next update of the service writes the old value back. + - To replace the token, make it with the new password on the jumpbox, and store it in the Key Vault. `curl` asks for the password: + + ```bash + curl -X POST -H "Content-Type: application/json" -u giteaadmin -d '{"name": "giteaadmin-"}' https://gitea-.azurewebsites.net/api/v1/users/giteaadmin/tokens + az keyvault secret set --name gitea--admin-token --vault-name kv- --value + ```