Skip to content

Lock the concept set row while saving a renamed concept set (#2430) - #2594

Open
konstde00 wants to merge 2 commits into
OHDSI:masterfrom
konstde00:fix/2430-conceptset-version-race
Open

konstde00 wants to merge 2 commits into
OHDSI:masterfrom
konstde00:fix/2430-conceptset-version-race

Conversation

@konstde00

Copy link
Copy Markdown

Hi!

I could get the error from #2430 only by deleting a concept set while another request renames it. The steps in the issue have no delete, so could one have come from another tab or user?

Currently updateConceptSet loads the concept set and then saves its version in a separate REQUIRES_NEW transaction, so if the row is deleted in between, the insert violates fk_concept_set_version_asset_id, VersionService retries it 10 times and the client gets a 500.

With this change the concept set is loaded with a pessimistic write lock (FOR NO KEY UPDATE on PostgreSQL, so the inner insert isn't blocked). A concurrent delete now waits, and if it goes first - the update returns 404. Concurrent renames of one concept set now wait for each other too.

The new renameWhileDeleting test fails without the change (in 40 rounds, 17 renames got "Error during saving version") and passes with it. mvn -B test passes, integration tests included. I didn't touch the 2.15 branch or the other asset types that use VersionService.

Please let me know if this matches what you see on your side.

Thank you!

updateConceptSet loads the concept set and then saves its version in a
separate REQUIRES_NEW transaction. If the concept set is deleted between
the load and the version insert, the insert violates
fk_concept_set_version_asset_id. Depending on timing, the request then
fails with "Error during saving version" after VersionService has retried
the failing insert ten times, or with an optimistic locking error when
the update of the concept set finds the row gone. This is one way to get
the error reported in OHDSI#2430, not necessarily the steps of that report.

Load the concept set with a pessimistic write lock (FOR NO KEY UPDATE on
PostgreSQL). A concurrent delete now waits until the update has finished,
and the version insert in the inner transaction is not blocked by it. If
the delete wins, the update returns 404. Concurrent renames of one
concept set now wait for each other instead of racing on the version
number.

Add ConceptSetConcurrentSaveTest: a rename racing with a delete, and
concurrent renames and item saves of one concept set (this one passes
without the lock and checks that the lock does not deadlock).
@chrisknoll

chrisknoll commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Hi @konstde00 , I reviewed with the help of CoPilot, here is a summary:

Thanks for tracking this down. The diagnosis matches what I see: the race is between update and delete, not between the header save and the items save. updateConceptSet loads the row, and then VersionService.save() inserts the version on a separate connection (@Transactional(propagation = REQUIRES_NEW)). If a delete commits in that gap, the insert breaks fk_concept_set_version_asset_id. VersionService.create() then retries it as if it were a version-number clash, up to versioning.maxAttempt times, and the client gets a 500.

The fix works on PostgreSQL, which is all WebAPI 3 supports. I have a few concerns, and I'd like to discuss a purely transactional alternative that doesn't need a special findByIdForUpdate repository method.

Concerns with the current approach

  1. It only works because of the PostgreSQL lock mode. In Hibernate 6.5, PESSIMISTIC_WRITE on PostgreSQL becomes FOR NO KEY UPDATE. The REQUIRES_NEW insert's foreign-key check takes FOR KEY SHARE on the parent row, and those two don't conflict. A plain FOR UPDATE would conflict: the request would block on a lock held by its own outer transaction. Hibernate uses for update nowait when a no-wait timeout is set, so a lock-timeout hint would fail the request immediately. If we keep this approach, please add a comment explaining why the lock mode matters.
  2. Each update still uses two connections. REQUIRES_NEW holds two pooled connections at once, which can starve the pool under load.
  3. saveConceptSetItems isn't covered. It deletes all items and re-inserts them with no lock. Because deleteByConceptSetId is a derived Spring Data delete (it selects the items, then deletes each by id), two overlapping item saves usually make the second request fail with a stale-state / optimistic-lock error, which surfaces as a 500. If the concept set had no items yet, both requests insert and you end up with both sets. The client (Atlas3) sends the header PUT /conceptset/{id} and then a separate PUT /conceptset/{id}/items, so a lock taken in the first request is gone before the second runs.
  4. The retry loop still treats every error as a version-number clash. That loop is what turns one foreign-key error into a 500. A foreign-key violation should not be retried. Cohorts, characterizations, incidence rates and pathways use the same VersionService and have the same problem.
  5. Later edits still overwrite earlier ones silently. Concurrent renames now wait for each other, but the second save still replaces the first without telling the user.

Suggested alternative: write the version in the same transaction, after the update

Drop REQUIRES_NEW for version creation, flush the header UPDATE first, and then write the version in the same transaction. The UPDATE locks the row on any database, so no lock hint or special repository method is needed.

@Transactional
public ConceptSetDTO updateConceptSet(int id, ConceptSetDTO dto) {
    ConceptSet cs = repo.findById(id).orElseThrow(/* 404 */);
    ConceptSetVersion snapshot = buildVersionSnapshot(cs); // items JSON + previous modifiedBy/date, captured in memory
    updateConceptSetInternal(cs, conversionService.convert(dto, ConceptSet.class));
    entityManager.flush();   // the UPDATE takes the row lock; a concurrent delete or rename waits here
    versionService.createInCurrentTx(VersionType.CONCEPT_SET, snapshot); // max+1, insert, no REQUIRES_NEW, no retry loop
    // ...
}

How it behaves:

  • Concurrent renames: the second update waits at its UPDATE. When it continues, its "latest version + 1" query is a new statement and sees the first update's committed version, so version numbers can't collide and the retry loop isn't needed.
  • Delete committed first: the flush updates zero rows and Hibernate throws StaleStateException/OptimisticLockException, which we map to 404.
  • Delete arriving later: it waits behind the update's row lock.
  • Connections: one per request, and the lock mode no longer matters because a transaction can't block itself.
  • New method: VersionService needs a create method without REQUIRES_NEW that runs in the caller's transaction. Other asset types could adopt it later.

Larger follow-ups (probably separate PRs)

  • Optimistic locking: add a @Version column to concept_set (a migration plus a field in the DTO). Simultaneous edits then return 409 Conflict instead of silently overwriting each other (point 5).
  • One save endpoint: save the header, items and version in a single transaction, keeping the existing two endpoints for older clients. This fixes points 3 and 5 and makes a save all-or-nothing, so a failed second request can't leave a half-saved concept set.

Minimum ask if we keep this PR as is

  • Stop retrying foreign-key violations in VersionService.create. Only retry unique or primary-key clashes on the version number, and return 404/409 for anything else.
  • Add a comment on findByIdForUpdate explaining that it depends on PostgreSQL's FOR NO KEY UPDATE, given the REQUIRES_NEW insert.

The new ConceptSetConcurrentSaveTest is great. It would be a good regression check for whichever approach we choose.


More broadly, tho, when versioning was first introduced, I had a concern that it would slow down performance due to the saves now taking twice the time to complete (the main save + the version save). I had an idea that we could have asset-versioning offloaded to a sort of queue where the json representation of the asset would be saved as a version async while the main save completes and returns to the user as soon as the main save completes. This does lead to a minor delay of getting a version saved (that might not appear in the ui until a little after the save) but seemed to be a good trade off for app performance.

I am wondering if we just fix this issue you identified quickly, and then in version 3.1 we re-think how asset versioning works in a more async/queue fashon so each save happens when it happens and drops a queue'd msg to create the version for the asset (this will preserve order, while taking it out of the save-flow). This is a larger refactor, but I am trying to keep the code complexity lower (I'm sensing that this fix is introducing some complexity in saving due to the behavior of versioning).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants