Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ BUG FIXES:
Reject failed parameter discovery and keep request parameters out of completion logs.
(`resource_processor` 0.13.10, `tre-shared-service-firewall` 1.6.4) ([#4903](https://github.com/microsoft/AzureTRE/issues/4903))
* Allow up to 30 checks at ten-second intervals when enabling or disabling storage public access during deployment. (`devops` 0.6.10) ([#5113](https://github.com/microsoft/AzureTRE/pull/5113))
* Name the workspace storage account of new `airlock_version` 2 workspaces from a random 12-character suffix (`stgws{suffix}`) checked for availability before creation, instead of the last 4 characters of the workspace ID,
to prevent `StorageAccountAlreadyTaken` deployment failures. Legacy (v1) workspaces, including those later moved to v2, keep their existing names. The workspace exposes `storage_account_name`, which OHDSI and Guacamole VMs now use.
(`API` 0.28.3, `tre-workspace-base` 2.13.2, `tre-workspace-unrestricted` 0.15.2, `tre-workspace-airlock-import-review` 0.18.2, `tre-workspace-service-ohdsi` 0.4.0, `tre-service-guacamole-linuxvm` 1.5.0, `tre-service-guacamole-windowsvm` 3.1.0) ([#2893](https://github.com/microsoft/AzureTRE/issues/2893), [#3666](https://github.com/microsoft/AzureTRE/issues/3666))
* Include Azure cloud and location in isolated CI environment names to avoid retained Key Vault name conflicts after region changes. Find previous CI environments by ownership tags during explicit cleanup. (`devops` 0.6.9) ([#5106](https://github.com/microsoft/AzureTRE/issues/5106))
* Validate Certbot before `make letsencrypt` prepares Azure access, check the resource group after loading Terraform outputs, and disable optional Super-Linter publishing with read-only GitHub tokens. ([#5110](https://github.com/microsoft/AzureTRE/pull/5110))
* Preserve loaded UI views during background refresh, ignore stale workspace responses, refresh on every tab return, and keep internal Connect behavior and unavailable cost displays consistent. (`ui` 0.10.1) ([#5105](https://github.com/microsoft/AzureTRE/pull/5105))
Expand Down
2 changes: 1 addition & 1 deletion api_app/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.28.2"
__version__ = "0.28.3"
74 changes: 52 additions & 22 deletions api_app/db/repositories/workspaces.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import uuid
import secrets
import string
from typing import List, Tuple
import asyncio
from azure.mgmt.storage.aio import StorageManagementClient
Expand Down Expand Up @@ -114,9 +116,13 @@ async def get_workspace_by_id(self, workspace_id: str) -> Workspace:
raise EntityDoesNotExist
return TypeAdapter(Workspace).validate_python(workspaces[0])

# Remove this method once not using last 4 digits for naming - https://github.com/microsoft/AzureTRE/issues/3666
async def is_workspace_storage_account_available(self, credential, workspace_id: str) -> bool:
name = f"stgws{workspace_id[-4:]}"
@staticmethod
def generate_unique_identifier_suffix() -> str:
alphabet = string.ascii_lowercase + string.digits
return "".join(secrets.choice(alphabet) for _ in range(constants.UNIQUE_IDENTIFIER_SUFFIX_LENGTH))

async def is_workspace_storage_account_available(self, credential, name_suffix: str) -> bool:
name = constants.STORAGE_ACCOUNT_NAME_WORKSPACE.format(name_suffix)
storage_client = StorageManagementClient(
credential,
config.SUBSCRIPTION_ID,
Expand All @@ -136,25 +142,6 @@ async def create_workspace_item(
self, workspace_input: WorkspaceInCreate, auth_info: dict, workspace_owner_object_id: str, user_roles: List[str]
) -> Tuple[Workspace, ResourceTemplate]:

full_workspace_id = str(uuid.uuid4())

# Ensure workspace with last four digits of ID does not already exist - remove when https://github.com/microsoft/AzureTRE/issues/3666 is resolved
async with credentials.get_credential_async_context() as credential:

async def name_check():
nonlocal full_workspace_id
while not await self.is_workspace_storage_account_available(credential, full_workspace_id):
full_workspace_id = str(uuid.uuid4())

try:
await asyncio.wait_for(name_check(), timeout=45.0)
except asyncio.TimeoutError:
raise StorageAccountNameGenerationTimeout(
"Unable to generate a unique storage account name within the timeout limit."
)
except HttpResponseError as e:
raise StorageAccountNameCheckFailed("Storage name availability check failed.") from e

template = await self.validate_input_against_template(
workspace_input.templateName, workspace_input, ResourceType.Workspace, user_roles
)
Expand Down Expand Up @@ -183,6 +170,15 @@ async def name_check():
airlock_version = 1
airlock_version_param = {"airlock_version": airlock_version}

# Only templates that consume unique_identifier_suffix get a random storage account name. Older
# airlock v2-capable templates keep the legacy naming, so no suffix is stored that the bundle would ignore.
supports_storage_suffix = "unique_identifier_suffix" in template.properties
full_workspace_id, unique_identifier_suffix = await self.generate_workspace_storage_naming(
airlock_version if supports_storage_suffix else 1
)
# The suffix is set once at creation, so legacy workspaces keep their names even after moving to v2.
unique_identifier_suffix_param = {"unique_identifier_suffix": unique_identifier_suffix}

# JSON Schema defaults validate input but are not materialised by the API. Persist the
# Airlock default so review-workspace templates that default it to false do not later get
# interpreted as having Airlock enabled.
Expand All @@ -203,6 +199,7 @@ async def name_check():
**workspace_owner_param,
**airlock_version_param,
**enable_airlock_param,
**unique_identifier_suffix_param,
**auth_info,
**self.get_workspace_spec_params(full_workspace_id),
}
Expand All @@ -218,6 +215,39 @@ async def name_check():

return workspace, template

async def generate_workspace_storage_naming(self, airlock_version: int) -> Tuple[str, str]:
"""Generate a workspace id and storage account name suffix with an available storage account name.

airlock_version >= 2 workspaces name their storage account from a random unique_identifier_suffix.
Legacy (v1) workspaces keep naming storage accounts from the last 4 characters of the workspace id
(https://github.com/microsoft/AzureTRE/issues/3666), and return an empty suffix.
"""
full_workspace_id = str(uuid.uuid4())
unique_identifier_suffix = ""

async with credentials.get_credential_async_context() as credential:

async def name_check():
nonlocal full_workspace_id, unique_identifier_suffix
if airlock_version >= 2:
unique_identifier_suffix = self.generate_unique_identifier_suffix()
while not await self.is_workspace_storage_account_available(credential, unique_identifier_suffix):
unique_identifier_suffix = self.generate_unique_identifier_suffix()
else:
while not await self.is_workspace_storage_account_available(credential, full_workspace_id[-4:]):
full_workspace_id = str(uuid.uuid4())

try:
await asyncio.wait_for(name_check(), timeout=45.0)
except asyncio.TimeoutError:
raise StorageAccountNameGenerationTimeout(
"Unable to generate a unique storage account name within the timeout limit."
)
except HttpResponseError as e:
raise StorageAccountNameCheckFailed("Storage name availability check failed.") from e

return full_workspace_id, unique_identifier_suffix

def get_workspace_owner(self, workspace_properties: dict, workspace_owner_object_id: str) -> str:
# Add the objectId of the user that will become the workspace owner. If it is not present in
# the request, we can assume the logged in user will be WorkspaceOwner
Expand Down
5 changes: 5 additions & 0 deletions api_app/resources/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
CORE_RESOURCE_GROUP_NAME = "rg-{}"
WORKSPACE_RESOURCE_GROUP_NAME = "rg-{}-ws-{}"

# Workspace storage account (templates/workspaces/base/terraform/locals.tf)
STORAGE_ACCOUNT_NAME_WORKSPACE = "stgws{}"
# Length of the random suffix used to name the storage account of airlock_version >= 2 workspaces
UNIQUE_IDENTIFIER_SUFFIX_LENGTH = 12

IMPORT_TYPE = "import"
EXPORT_TYPE = "export"

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import re
import copy
from unittest.mock import AsyncMock
import pytest
Expand Down Expand Up @@ -292,6 +293,98 @@ async def test_create_workspace_item_defaults_legacy_template_to_airlock_v1(
assert workspace.properties["airlock_version"] == 1


@pytest.mark.asyncio
@patch("db.repositories.workspaces.generate_new_cidr")
@patch("db.repositories.workspaces.WorkspaceRepository.validate_input_against_template")
@patch("db.repositories.workspaces.WorkspaceRepository.is_workspace_storage_account_available")
@patch("core.config.RESOURCE_LOCATION", "useast2")
@patch("core.config.TRE_ID", "9876")
async def test_create_workspace_item_v2_uses_available_random_storage_suffix(
mock_is_workspace_storage_account_available,
validate_input_mock,
new_cidr_mock,
workspace_repo,
basic_workspace_request,
basic_resource_template,
):
workspace_to_create = basic_workspace_request
workspace_to_create.properties["airlock_version"] = 2
workspace_to_create.properties["unique_identifier_suffix"] = "userinput"
# first generated suffix is taken, second is available
mock_is_workspace_storage_account_available.side_effect = [False, True]
basic_resource_template.properties["airlock_version"] = Property(default=2, enum=[1, 2])
# the template must declare the suffix to consume it
basic_resource_template.properties["unique_identifier_suffix"] = Property(type="string")
validate_input_mock.return_value = basic_resource_template
new_cidr_mock.return_value = "1.2.3.4/24"

workspace, _ = await workspace_repo.create_workspace_item(workspace_to_create, {}, "test_object_id", ["test_role"])

suffix = workspace.properties["unique_identifier_suffix"]
assert re.fullmatch(r"[a-z0-9]{12}", suffix)
assert mock_is_workspace_storage_account_available.call_count == 2
assert mock_is_workspace_storage_account_available.call_args_list[-1].args[1] == suffix
assert mock_is_workspace_storage_account_available.call_args_list[0].args[1] != suffix
assert workspace.properties["workspace_id"] == workspace.id[-4:]


@pytest.mark.asyncio
@patch("db.repositories.workspaces.generate_new_cidr")
@patch("db.repositories.workspaces.WorkspaceRepository.validate_input_against_template")
@patch("db.repositories.workspaces.WorkspaceRepository.is_workspace_storage_account_available")
@patch("core.config.RESOURCE_LOCATION", "useast2")
@patch("core.config.TRE_ID", "9876")
async def test_create_workspace_item_v2_template_without_suffix_keeps_legacy_storage_naming(
mock_is_workspace_storage_account_available,
validate_input_mock,
new_cidr_mock,
workspace_repo,
basic_workspace_request,
basic_resource_template,
):
workspace_to_create = basic_workspace_request
workspace_to_create.properties["airlock_version"] = 2
mock_is_workspace_storage_account_available.return_value = True
basic_resource_template.properties["airlock_version"] = Property(default=2, enum=[1, 2])
validate_input_mock.return_value = basic_resource_template
new_cidr_mock.return_value = "1.2.3.4/24"

workspace, _ = await workspace_repo.create_workspace_item(workspace_to_create, {}, "test_object_id", ["test_role"])

assert workspace.properties["unique_identifier_suffix"] == ""
assert mock_is_workspace_storage_account_available.call_args.args[1] == workspace.id[-4:]


@pytest.mark.asyncio
@patch("db.repositories.workspaces.generate_new_cidr")
@patch("db.repositories.workspaces.WorkspaceRepository.validate_input_against_template")
@patch("db.repositories.workspaces.WorkspaceRepository.is_workspace_storage_account_available")
@patch("core.config.RESOURCE_LOCATION", "useast2")
@patch("core.config.TRE_ID", "9876")
async def test_create_workspace_item_v1_keeps_workspace_id_storage_naming(
mock_is_workspace_storage_account_available,
validate_input_mock,
new_cidr_mock,
workspace_repo,
basic_workspace_request,
basic_resource_template,
):
workspace_to_create = basic_workspace_request
workspace_to_create.properties["airlock_version"] = 1
workspace_to_create.properties["unique_identifier_suffix"] = "userinput"
# first generated workspace id is taken, second is available
mock_is_workspace_storage_account_available.side_effect = [False, True]
validate_input_mock.return_value = basic_resource_template
new_cidr_mock.return_value = "1.2.3.4/24"

workspace, _ = await workspace_repo.create_workspace_item(workspace_to_create, {}, "test_object_id", ["test_role"])

assert workspace.properties["airlock_version"] == 1
assert workspace.properties["unique_identifier_suffix"] == ""
assert mock_is_workspace_storage_account_available.call_count == 2
assert mock_is_workspace_storage_account_available.call_args_list[-1].args[1] == workspace.id[-4:]


@pytest.mark.asyncio
@patch("core.config.RESOURCE_LOCATION", "useast2")
@patch("core.config.TRE_ID", "9876")
Expand Down Expand Up @@ -486,16 +579,15 @@ def test_workspace_owner_is_not_overwritten_if_present_in_workspace_properties(w
@pytest.mark.asyncio
@patch("db.repositories.workspaces.StorageManagementClient")
async def test_is_workspace_storage_account_available_when_name_available(mock_storage_client):
workspace_id = "workspace1234"
suffix = workspace_id[-4:]
suffix = "abc123def456"
mock_storage_client_instance = MagicMock()
mock_storage_client_instance.storage_accounts.check_name_availability = AsyncMock()
mock_storage_client_instance.storage_accounts.check_name_availability.return_value.name_available = True
mock_storage_client_instance.close = AsyncMock()
mock_storage_client.return_value = mock_storage_client_instance
workspace_repo = WorkspaceRepository()

result = await workspace_repo.is_workspace_storage_account_available(MagicMock(), workspace_id)
result = await workspace_repo.is_workspace_storage_account_available(MagicMock(), suffix)

assert result is True
assert mock_storage_client_instance.storage_accounts.check_name_availability.call_count == 1
Expand All @@ -507,8 +599,7 @@ async def test_is_workspace_storage_account_available_when_name_available(mock_s
@pytest.mark.asyncio
@patch("db.repositories.workspaces.StorageManagementClient")
async def test_is_workspace_storage_account_available_when_name_not_available(mock_storage_client):
workspace_id = "workspace1234"
suffix = workspace_id[-4:]
suffix = "abc123def456"
mock_storage_client_instance = MagicMock()
mock_storage_client_instance.storage_accounts.check_name_availability = AsyncMock()
mock_storage_client_instance.close = AsyncMock()
Expand All @@ -518,7 +609,7 @@ async def test_is_workspace_storage_account_available_when_name_not_available(mo
mock_storage_client.return_value = mock_storage_client_instance
workspace_repo = WorkspaceRepository()

result = await workspace_repo.is_workspace_storage_account_available(MagicMock(), workspace_id)
result = await workspace_repo.is_workspace_storage_account_available(MagicMock(), suffix)

assert result is False
assert mock_storage_client_instance.storage_accounts.check_name_availability.call_count == 1
Expand All @@ -530,7 +621,7 @@ async def test_is_workspace_storage_account_available_when_name_not_available(mo
@pytest.mark.asyncio
@patch("db.repositories.workspaces.StorageManagementClient")
async def test_is_workspace_storage_account_available_when_check_raises_exception(mock_storage_client):
workspace_id = "workspace1234"
suffix = "abc123def456"
mock_storage_client_instance = MagicMock()
mock_storage_client_instance.storage_accounts.check_name_availability = AsyncMock()
mock_storage_client_instance.storage_accounts.check_name_availability.side_effect = Exception("ARM error")
Expand All @@ -539,7 +630,7 @@ async def test_is_workspace_storage_account_available_when_check_raises_exceptio
workspace_repo = WorkspaceRepository()

with pytest.raises(Exception, match="ARM error"):
await workspace_repo.is_workspace_storage_account_available(MagicMock(), workspace_id)
await workspace_repo.is_workspace_storage_account_available(MagicMock(), suffix)


@pytest.mark.asyncio
Expand Down Expand Up @@ -577,7 +668,7 @@ async def mock_wait_for_se(coro, timeout=None):
@pytest.mark.asyncio
@patch("db.repositories.workspaces.StorageManagementClient")
async def test_is_workspace_storage_account_available_when_check_times_out(mock_storage_client):
workspace_id = "workspace1234"
suffix = "abc123def456"
mock_storage_client_instance = MagicMock()
mock_storage_client_instance.storage_accounts.check_name_availability = AsyncMock()
mock_storage_client_instance.close = AsyncMock()
Expand All @@ -586,7 +677,7 @@ async def test_is_workspace_storage_account_available_when_check_times_out(mock_
workspace_repo = WorkspaceRepository()

with pytest.raises(asyncio.TimeoutError):
await workspace_repo.is_workspace_storage_account_available(MagicMock(), workspace_id)
await workspace_repo.is_workspace_storage_account_available(MagicMock(), suffix)

assert mock_storage_client_instance.storage_accounts.check_name_availability.call_count == 1

Expand Down
2 changes: 1 addition & 1 deletion docs/azure-tre-overview/tre-resources-breakdown.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ A TRE Workspace will be provisioned in a separate Resource Group along with its
| kv-{TRE_ID}-ws-XXXX | Azure Key Vault | Management of TRE workspace secrets & certificates | [Azure Key Vault](https://docs.microsoft.com/en-us/azure/key-vault/general/overview) |
| osdisk-windowsvm8f45 | Disk | Azure VM storage disk | [Managed Disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) |
| plan-09d0ba4f-f79f-4047-aa2c-03fc9df7b318 | App Service plan | Compute resources in which the workspace app services (Gitea) run | [App Hosting Plans](https://docs.microsoft.com/en-us/azure/app-service/overview-hosting-plans) |
| stgwsb318 | Storage account | Workspace Storage account | [Storage Blobs](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) |
| stgwsb318 | Storage account | Workspace Storage account. Workspaces created with `airlock_version` 2 use a random suffix instead (e.g. `stgwsk3x9q2m7a1bz`) | [Storage Blobs](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) |
| vnet-{TRE_ID}-ws-XXXX | Virtual Network | [Azure TRE VNET spoke](../networking) | [Virtual Networks](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview) |
| windowsvm8f45 | Virtual Machine | [Windows VM instance](../../tre-templates/user-resources/guacamole-windows-vm) for research | [Windows Virtual Machine](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/overview) |

Expand Down
Loading