You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Redesign airlock to reduce number of storage accounts used #4358
The airlock uses a large number of storage accounts. The number of accounts (particularly the workspace ones) are a bind on scalability, and also increase the overall cost.
Is it possible to consolidate some of these accounts, and use containers to segregate data instead?
Per account:
Private endpoint $7.30/account/month
Defender scanning $10/account/month
E.g. TRE with 10 workspaces = 6 core airlock accounts, 50 workspace airlock accounts
It's not clear to me that the rej SAs serve any purpose at all. If a file is rejected, why is there any need to keep it at all? What happens to it after it goes to the rej SA, does it just sit there gathering dust until the workspace is destroyed? Can't we just get rid of that one altogether?
I believe RBAC can be applied at the container level.
To do RBAC at the container level the storage accounts need Hierarchical namespace Enabled and the storage account needs to be a Data Lake Storage Gen2 I believe.
What is possible now might not have been at the time it was created.
Enabling HNS on the storage accounts might have implications on events and subscription possibilities
I remember something around repeat defender scans. If we copy (move) files between containers on the same account it might trigger a repeat scan. Maybe at the time we could only turn on defender on the account level and maybe that has changed.
Networking wise I think looking at the core vnet isn't enough as I think different subnets have different access, so need to confirm what merging accounts will do to network security.
p.s. I haven't checked this now, this is just thoughts that I had.
Our AzureTRE installation is considering disabling airlock for a workspace when it is not needed to cut costs. Ideally this would not need to be managed by our TRE Admin.
What are the technical limitations on AzureTRE doing this automatically someday, only creating these storage accounts when needed?
Matthew Dunlap (@matthew-a-dunlap) as in spinning them up when needed? Hmm, there is infra that needs to be deployed, but its an option, how would you implement it? Storage accounts dont cost, its the data in them. Some people have also been discussing moving the airlock accounts to a shared model.
Marcus Robinson (@marrobi) The motivation for considering disabling/enabling airlock are mostly that the private endpoint costs add up when you are considering hundreds of research workspaces. Any solution that makes the workspaces leaner is one we'd be interested in.
I am unsure of implementation at this point, I mostly wanted to test the waters about whether we are off-base on some fundamentals about this. We are in early stages of talking through options.
Matthew Dunlap (@matthew-a-dunlap) hmm, didbt realy think about the $0.01 an hour base cost. Eveb if centralised, would likely need private endpoints, as need a way to restrict access based on traffic source.
Creting PEs on demand might be an option, but DNS caching etc is likely to get messy.
Say:
"Microsoft recommends use of Azure Private Link and private endpoints for secure and private access to services hosted on the Azure platform. Azure Private Link deploys a network interface into a virtual network of your choosing for Azure services such as Azure Storage or Azure SQL. For more information, see Azure Private Link and What is a private endpoint?."
Question: Migrating a workspace from legacy (v1) to the new (v2) Airlock storage architecture
When we upgrade an existing workspace to the new v2 Airlock storage model, the migration would (by default) remove the legacy v1 storage accounts (in that workspace) as part of the Terraform apply. This means any in-progress or approved Airlock requests tied to those legacy accounts would be lost — e.g. approved imports and draft/in-flight exports.
We'd like to understand your expectations:
Would it be acceptable to drain/complete all in-progress and approved Airlock requests before upgrading a workspace, treating the upgrade as a maintenance window where no requests are pending?
If not, would losing those in-flight requests on upgrade be a blocker for you — meaning we'd need to build a more complex migration path that preserves or moves the legacy accounts (and their Terraform state) before removing them?
Understanding whether "flush before upgrade" is workable, or whether zero data loss is a hard requirement, will help us decide whether to invest in the more complex state-migration approach.
In reality I think will caveat the behaviour, and if people want something more complex then we can maybe provide some scripts...
Marcus Robinson (@marrobi) On our end, completing the in-progress/approved airlock requests is fine. We are fine with the "data loss' for these, and will notify our users with ample warning. We we also potentially recreate a few of the requests afterwards.
What is most important for our data brokering team is that the v1 airlock metadata is still present and complete in v2. So that the requests can be audited going forward.
Hi Marcus Robinson (@marrobi) We would also be happy if airlocks had to be lost to migrate over, our only request would be to integrate a warning when migrating to airlock v2.
Activity
TonyWildish-BH commentedon Feb 12, 2025
It's not clear to me that the
rejSAs serve any purpose at all. If a file is rejected, why is there any need to keep it at all? What happens to it after it goes to therejSA, does it just sit there gathering dust until the workspace is destroyed? Can't we just get rid of that one altogether?jonnyry commentedon Feb 13, 2025
Indeed - or if necessary to keep could there be a rejected container, rather than a whole seperate storage account?
In fact could some of the accounts that share the same networking be consolidated, and containers used instead of seperate accounts... e.g.
I believe RBAC can be applied at the container level.
marrobi commentedon Feb 13, 2025
Rejected was there in case of false positives, the fact that occasionally there might be a need to go retrieve the data.
I'm trying to recall why they were separate accounts, I agree if the share the networking and RBAC can be done at a container level, that might work.
Liza Shakury (@LizaShak) (or even Elad Iwanir (@eladiw) ), can you remember?
West-P commentedon Feb 13, 2025
To do RBAC at the container level the storage accounts need Hierarchical namespace Enabled and the storage account needs to be a Data Lake Storage Gen2 I believe.
tamirkamara commentedon Mar 27, 2025
p.s. I haven't checked this now, this is just thoughts that I had.
matthew-a-dunlap commentedon Feb 2, 2026
Our AzureTRE installation is considering disabling airlock for a workspace when it is not needed to cut costs. Ideally this would not need to be managed by our TRE Admin.
What are the technical limitations on AzureTRE doing this automatically someday, only creating these storage accounts when needed?
marrobi commentedon Feb 2, 2026
Matthew Dunlap (@matthew-a-dunlap) as in spinning them up when needed? Hmm, there is infra that needs to be deployed, but its an option, how would you implement it? Storage accounts dont cost, its the data in them. Some people have also been discussing moving the airlock accounts to a shared model.
matthew-a-dunlap commentedon Feb 3, 2026
Marcus Robinson (@marrobi) The motivation for considering disabling/enabling airlock are mostly that the private endpoint costs add up when you are considering hundreds of research workspaces. Any solution that makes the workspaces leaner is one we'd be interested in.
I am unsure of implementation at this point, I mostly wanted to test the waters about whether we are off-base on some fundamentals about this. We are in early stages of talking through options.
marrobi commentedon Feb 3, 2026
Matthew Dunlap (@matthew-a-dunlap) hmm, didbt realy think about the $0.01 an hour base cost. Eveb if centralised, would likely need private endpoints, as need a way to restrict access based on traffic source.
Creting PEs on demand might be an option, but DNS caching etc is likely to get messy.
marrobi commentedon Feb 3, 2026
Could look at https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-attributes#subnet, but would be using service endpoints, not private link, whick may raise security questions. Traffic is restricted by source, abd stays on the Azure network, but the storage account has a public IP.
marrobi commentedon Feb 5, 2026
Looking more, the docs at https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview:
Say:
"Microsoft recommends use of Azure Private Link and private endpoints for secure and private access to services hosted on the Azure platform. Azure Private Link deploys a network interface into a virtual network of your choosing for Azure services such as Azure Storage or Azure SQL. For more information, see Azure Private Link and What is a private endpoint?."
So I really think we need to stick to private endpoint. The storage account limit is an issue, so propose we look at using this: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-examples?tabs=portal-visual-editor#example-allow-access-to-a-container-only-from-a-specific-private-endpoint . This doesn't resolve the fact we need multiple private endpoints, but does reduce the storage account quota issues.
Thoughts?
6 remaining items
TonyWildish-BH commentedon Apr 2, 2026
thanks Marcus Robinson (@marrobi) , I'll test this after Easter!
rudolphjacksonm commentedon Jul 3, 2026
Tony Wildish (@TonyWildish-BH) did you get a chance to test this? Keen to pick this up and collect any feedback you might have on it
TonyWildish-BH commentedon Jul 3, 2026
sorry, no, I've not had the chance to test this at all. It's not clear when I'll have time available in the next 2-3 months either.
Fix import-review workspace template to support both v1 and v2 airloc…
marrobi commentedon Aug 18, 2026
Question: Migrating a workspace from legacy (v1) to the new (v2) Airlock storage architecture
When we upgrade an existing workspace to the new v2 Airlock storage model, the migration would (by default) remove the legacy v1 storage accounts (in that workspace) as part of the Terraform apply. This means any in-progress or approved Airlock requests tied to those legacy accounts would be lost — e.g. approved imports and draft/in-flight exports.
We'd like to understand your expectations:
Would it be acceptable to drain/complete all in-progress and approved Airlock requests before upgrading a workspace, treating the upgrade as a maintenance window where no requests are pending?
If not, would losing those in-flight requests on upgrade be a blocker for you — meaning we'd need to build a more complex migration path that preserves or moves the legacy accounts (and their Terraform state) before removing them?
Understanding whether "flush before upgrade" is workable, or whether zero data loss is a hard requirement, will help us decide whether to invest in the more complex state-migration approach.
In reality I think will caveat the behaviour, and if people want something more complex then we can maybe provide some scripts...
matthew-a-dunlap commentedon Aug 18, 2026
Marcus Robinson (@marrobi) On our end, completing the in-progress/approved airlock requests is fine. We are fine with the "data loss' for these, and will notify our users with ample warning. We we also potentially recreate a few of the requests afterwards.
What is most important for our data brokering team is that the v1 airlock metadata is still present and complete in v2. So that the requests can be audited going forward.
JaimieWi commentedon Aug 18, 2026
Hi Marcus Robinson (@marrobi) We would also be happy if airlocks had to be lost to migrate over, our only request would be to integrate a warning when migrating to airlock v2.
Thank you, this will be a lifesaver!