Skip to content

Redesign airlock to reduce number of storage accounts used #4358

Description

The airlock uses a large number of storage accounts. The number of accounts (particularly the workspace ones) are a bind on scalability, and also increase the overall cost.

Is it possible to consolidate some of these accounts, and use containers to segregate data instead?


Per account:

  • Private endpoint $7.30/account/month
  • Defender scanning $10/account/month

E.g. TRE with 10 workspaces = 6 core airlock accounts, 50 workspace airlock accounts

Airlock storage accounts - core

Name Description
st + airlockp + <TRE_ID> Airlock Processor
st + alexapp + <TRE_ID> Airlock Export Approved
st + alimblocked + <TRE_ID> Airlock Import Blocked
st + alimex + <TRE_ID> Airlock Import External
st + alimip + <TRE_ID> Airlock Import In Progress
st + alimrej + <TRE_ID> Airlock Import Rejected

Airlock storage accounts - per workspace

Name Description
st + alexblocked + ws + <WS_ID> Airlock Export Blocked
st + alexint + ws + <WS_ID> Airlock Export Internal
st + alexip + ws + <WS_ID> Airlock Export In Progress
st + alexrej + ws + <WS_ID> Airlock Export Rejected
st + alimapp + ws + <WS_ID> Airlock Import Approved

Activity

  1. TonyWildish-BH commented on Feb 12, 2025

    @TonyWildish-BH
    Contributor

    It's not clear to me that the rej SAs serve any purpose at all. If a file is rejected, why is there any need to keep it at all? What happens to it after it goes to the rej SA, does it just sit there gathering dust until the workspace is destroyed? Can't we just get rid of that one altogether?

  2. jonnyry commented on Feb 13, 2025

    @jonnyry
    ContributorAuthor

    Indeed - or if necessary to keep could there be a rejected container, rather than a whole seperate storage account?

    In fact could some of the accounts that share the same networking be consolidated, and containers used instead of seperate accounts... e.g.

    I believe RBAC can be applied at the container level.

    Image

  3. marrobi commented on Feb 13, 2025

    @marrobi
    Member

    Rejected was there in case of false positives, the fact that occasionally there might be a need to go retrieve the data.

    I'm trying to recall why they were separate accounts, I agree if the share the networking and RBAC can be done at a container level, that might work.

    Liza Shakury (@LizaShak) (or even Elad Iwanir (@eladiw) ), can you remember?

  4. West-P commented on Feb 13, 2025

    @West-P

    I believe RBAC can be applied at the container level.

    To do RBAC at the container level the storage accounts need Hierarchical namespace Enabled and the storage account needs to be a Data Lake Storage Gen2 I believe.

  5. tamirkamara commented on Mar 27, 2025

    @tamirkamara
    Collaborator
    1. What is possible now might not have been at the time it was created.
    2. Enabling HNS on the storage accounts might have implications on events and subscription possibilities
    3. I remember something around repeat defender scans. If we copy (move) files between containers on the same account it might trigger a repeat scan. Maybe at the time we could only turn on defender on the account level and maybe that has changed.
    4. Networking wise I think looking at the core vnet isn't enough as I think different subnets have different access, so need to confirm what merging accounts will do to network security.

    p.s. I haven't checked this now, this is just thoughts that I had.

  6. matthew-a-dunlap commented on Feb 2, 2026

    @matthew-a-dunlap
    Contributor

    Our AzureTRE installation is considering disabling airlock for a workspace when it is not needed to cut costs. Ideally this would not need to be managed by our TRE Admin.

    What are the technical limitations on AzureTRE doing this automatically someday, only creating these storage accounts when needed?

  7. marrobi commented on Feb 2, 2026

    @marrobi
    Member

    Matthew Dunlap (@matthew-a-dunlap) as in spinning them up when needed? Hmm, there is infra that needs to be deployed, but its an option, how would you implement it? Storage accounts dont cost, its the data in them. Some people have also been discussing moving the airlock accounts to a shared model.

  8. matthew-a-dunlap commented on Feb 3, 2026

    @matthew-a-dunlap
    Contributor

    Marcus Robinson (@marrobi) The motivation for considering disabling/enabling airlock are mostly that the private endpoint costs add up when you are considering hundreds of research workspaces. Any solution that makes the workspaces leaner is one we'd be interested in.

    I am unsure of implementation at this point, I mostly wanted to test the waters about whether we are off-base on some fundamentals about this. We are in early stages of talking through options.

  9. marrobi commented on Feb 3, 2026

    @marrobi
    Member

    Matthew Dunlap (@matthew-a-dunlap) hmm, didbt realy think about the $0.01 an hour base cost. Eveb if centralised, would likely need private endpoints, as need a way to restrict access based on traffic source.

    Creting PEs on demand might be an option, but DNS caching etc is likely to get messy.

  10. marrobi commented on Feb 3, 2026

    @marrobi
    Member

    Could look at https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-attributes#subnet, but would be using service endpoints, not private link, whick may raise security questions. Traffic is restricted by source, abd stays on the Azure network, but the storage account has a public IP.

  11. marrobi commented on Feb 5, 2026

    @marrobi
    Member

    Looking more, the docs at https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview:

    Say:
    "Microsoft recommends use of Azure Private Link and private endpoints for secure and private access to services hosted on the Azure platform. Azure Private Link deploys a network interface into a virtual network of your choosing for Azure services such as Azure Storage or Azure SQL. For more information, see Azure Private Link and What is a private endpoint?."

    So I really think we need to stick to private endpoint. The storage account limit is an issue, so propose we look at using this: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-examples?tabs=portal-visual-editor#example-allow-access-to-a-container-only-from-a-specific-private-endpoint . This doesn't resolve the fact we need multiple private endpoints, but does reduce the storage account quota issues.

    Thoughts?

  12. 6 remaining items

  13. TonyWildish-BH commented on Apr 2, 2026

    @TonyWildish-BH
    Contributor

    thanks Marcus Robinson (@marrobi) , I'll test this after Easter!

  14. moved this from Up Next to In Progress in Azure TRE - Engineeringon Jul 2, 2026
  15. rudolphjacksonm commented on Jul 3, 2026

    @rudolphjacksonm
    Collaborator

    Tony Wildish (@TonyWildish-BH) did you get a chance to test this? Keen to pick this up and collect any feedback you might have on it

  16. TonyWildish-BH commented on Jul 3, 2026

    @TonyWildish-BH
    Contributor

    sorry, no, I've not had the chance to test this at all. It's not clear when I'll have time available in the next 2-3 months either.

  17. added a commit that references this issue on Jul 10, 2026
    f5683ca
  18. moved this from Up Next to In Progress in Azure TRE - Engineeringon Aug 17, 2026
  19. marrobi commented on Aug 18, 2026

    @marrobi
    Member

    Question: Migrating a workspace from legacy (v1) to the new (v2) Airlock storage architecture

    When we upgrade an existing workspace to the new v2 Airlock storage model, the migration would (by default) remove the legacy v1 storage accounts (in that workspace) as part of the Terraform apply. This means any in-progress or approved Airlock requests tied to those legacy accounts would be lost — e.g. approved imports and draft/in-flight exports.

    We'd like to understand your expectations:

    Would it be acceptable to drain/complete all in-progress and approved Airlock requests before upgrading a workspace, treating the upgrade as a maintenance window where no requests are pending?

    If not, would losing those in-flight requests on upgrade be a blocker for you — meaning we'd need to build a more complex migration path that preserves or moves the legacy accounts (and their Terraform state) before removing them?

    Understanding whether "flush before upgrade" is workable, or whether zero data loss is a hard requirement, will help us decide whether to invest in the more complex state-migration approach.

    In reality I think will caveat the behaviour, and if people want something more complex then we can maybe provide some scripts...

  20. matthew-a-dunlap commented on Aug 18, 2026

    @matthew-a-dunlap
    Contributor

    Marcus Robinson (@marrobi) On our end, completing the in-progress/approved airlock requests is fine. We are fine with the "data loss' for these, and will notify our users with ample warning. We we also potentially recreate a few of the requests afterwards.

    What is most important for our data brokering team is that the v1 airlock metadata is still present and complete in v2. So that the requests can be audited going forward.

  21. JaimieWi commented on Aug 18, 2026

    @JaimieWi
    Contributor

    Hi Marcus Robinson (@marrobi) We would also be happy if airlocks had to be lost to migrate over, our only request would be to integrate a warning when migrating to airlock v2.

    Thank you, this will be a lifesaver!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions