Repository navigation
Conversation
updateConceptSet loads the concept set and then saves its version in a separate REQUIRES_NEW transaction. If the concept set is deleted between the load and the version insert, the insert violates fk_concept_set_version_asset_id. Depending on timing, the request then fails with "Error during saving version" after VersionService has retried the failing insert ten times, or with an optimistic locking error when the update of the concept set finds the row gone. This is one way to get the error reported in OHDSI#2430, not necessarily the steps of that report. Load the concept set with a pessimistic write lock (FOR NO KEY UPDATE on PostgreSQL). A concurrent delete now waits until the update has finished, and the version insert in the inner transaction is not blocked by it. If the delete wins, the update returns 404. Concurrent renames of one concept set now wait for each other instead of racing on the version number. Add ConceptSetConcurrentSaveTest: a rename racing with a delete, and concurrent renames and item saves of one concept set (this one passes without the lock and checks that the lock does not deadlock).
|
Hi @konstde00 , I reviewed with the help of CoPilot, here is a summary: Thanks for tracking this down. The diagnosis matches what I see: the race is between update and delete, not between the header save and the items save. The fix works on PostgreSQL, which is all WebAPI 3 supports. I have a few concerns, and I'd like to discuss a purely transactional alternative that doesn't need a special Concerns with the current approach
Suggested alternative: write the version in the same transaction, after the updateDrop @Transactional
public ConceptSetDTO updateConceptSet(int id, ConceptSetDTO dto) {
ConceptSet cs = repo.findById(id).orElseThrow(/* 404 */);
ConceptSetVersion snapshot = buildVersionSnapshot(cs); // items JSON + previous modifiedBy/date, captured in memory
updateConceptSetInternal(cs, conversionService.convert(dto, ConceptSet.class));
entityManager.flush(); // the UPDATE takes the row lock; a concurrent delete or rename waits here
versionService.createInCurrentTx(VersionType.CONCEPT_SET, snapshot); // max+1, insert, no REQUIRES_NEW, no retry loop
// ...
}How it behaves:
Larger follow-ups (probably separate PRs)
Minimum ask if we keep this PR as is
The new More broadly, tho, when versioning was first introduced, I had a concern that it would slow down performance due to the saves now taking twice the time to complete (the main save + the version save). I had an idea that we could have asset-versioning offloaded to a sort of queue where the json representation of the asset would be saved as a version async while the main save completes and returns to the user as soon as the main save completes. This does lead to a minor delay of getting a version saved (that might not appear in the ui until a little after the save) but seemed to be a good trade off for app performance. I am wondering if we just fix this issue you identified quickly, and then in version 3.1 we re-think how asset versioning works in a more async/queue fashon so each save happens when it happens and drops a queue'd msg to create the version for the asset (this will preserve order, while taking it out of the save-flow). This is a larger refactor, but I am trying to keep the code complexity lower (I'm sensing that this fix is introducing some complexity in saving due to the behavior of versioning). |
Hi!
I could get the error from #2430 only by deleting a concept set while another request renames it. The steps in the issue have no delete, so could one have come from another tab or user?
Currently updateConceptSet loads the concept set and then saves its version in a separate REQUIRES_NEW transaction, so if the row is deleted in between, the insert violates fk_concept_set_version_asset_id, VersionService retries it 10 times and the client gets a 500.
With this change the concept set is loaded with a pessimistic write lock (FOR NO KEY UPDATE on PostgreSQL, so the inner insert isn't blocked). A concurrent delete now waits, and if it goes first - the update returns 404. Concurrent renames of one concept set now wait for each other too.
The new renameWhileDeleting test fails without the change (in 40 rounds, 17 renames got "Error during saving version") and passes with it.
mvn -B testpasses, integration tests included. I didn't touch the 2.15 branch or the other asset types that use VersionService.Please let me know if this matches what you see on your side.
Thank you!