Skip to content

fix(storages)!: Drop autosaved values on KeyValueStore.purge() - #2467

Open
Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:kvs-purge-autosaved
Open

Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:kvs-purge-autosaved

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

KeyValueStore.purge() now also drops the values that get_auto_saved_value() and use_state keep in memory for the store, the same as drop(). A value obtained before the purge isn't saved anymore, and the next get_auto_saved_value() or use_state call starts from the default value. A later save used to write these values back into the purged store.

If a purge or a drop fails or is cancelled, the values stay and keep being saved. A get_auto_saved_value() call made during a purge or a drop waits for it to finish.

Depends on #2466. Without it, a PERSIST_STATE save that started before the purge can still write the value back.

Testing

  • Unit tests for a purge, a purge concurrent with get_auto_saved_value(), and a failed or cancelled purge and drop.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 17, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 17, 2026
@Mantisus
Mantisus requested a review from vdusek November 17, 2026 23:35
@vdusek vdusek changed the title fix(storages)!: Drop autosaved values on purge and keep them when a purge or drop fails fix(storages)!: Drop autosaved values on KeyValueStore.purge() Nov 18, 2026
@@ -141,12 +141,11 @@ async def drop(self) -> None:
storage_instance_manager = service_locator.storage_instance_manager
storage_instance_manager.remove_from_cache(self)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The store leaves the instance cache before _clear_store_and_cache(). When the backend drop fails, the autosaved values are restored but the store is no longer cached. A later KeyValueStore.open() then creates a second instance that shares _autosaved_values[id] but has its own _autosave_lock. Crawlee JS removes the store from the cache only after backend.drop() succeeds. Moving remove_from_cache(self) after the clear would do the same here.

✍️ Drafted by Claude Code

# The read gets time to load the record while the purge is parked.
read_task = asyncio.create_task(kvs.get_auto_saved_value('state', {'count': 0}))
await asyncio.wait([read_task], timeout=0.1)
release_purge.set()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The 0.1 s wait gives the read time to run, but nothing checks that it was blocked. An assert makes the check explicit. A timeout on purge_started.wait() (line 1457) would also make a regression fail fast instead of hanging.

Suggested change
release_purge.set()
assert not read_task.done()
release_purge.set()

✍️ Drafted by Claude Code

@vdusek

vdusek commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Note: On its own, this PR still lets a PERSIST_STATE save that already started write the value back after purge(). Repro: block the listener's set_value, call purge(), then release it. The record is None after the purge and {'count': 1} after the listener finishes. With #2466 applied on top, it stays None. So #2466 should merge first, or the description should mention the dependency.

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants