Skip to content

fix(redis)!: Close the Redis client and open storages per backend - #2463

Open
Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:redis-backend-close
Open

Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:redis-backend-close

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

RedisStorageBackend can now be closed with close() or used as an async context manager. It closes the Redis client it created from a connection string and leaves a client you pass in open. Storages opened through a closed backend raise an error instead of opening new connections.

Each RedisStorageBackend instance now opens its own storages, so a new backend doesn't get the storages of a closed one. With purge_on_start enabled, a new backend on the same database purges the unnamed storages again.

When two clients create the same storage at once, the one that loses the race now uses the winner's ID, and it raises an error if the winner's metadata never appears.

Testing

  • Unit tests for closing, separate storages per backend, Lua scripts after close, and the creation race.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 17, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 17, 2026
@Mantisus
Mantisus requested a review from vdusek November 17, 2026 13:33
@vdusek vdusek changed the title fix(storages)!: Close Redis connections and keep each backend's storages separate fix(redis)!: Close the Redis client and open storages per backend Nov 18, 2026
requests_json = await self._fetch_script(
keys=[self._queue_key, self._in_progress_key, self._data_key],
args=[self.client_key, blocked_until_timestamp, self._MAX_BATCH_FETCH_SIZE],
client=self.redis,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: fetch_next_request() returns from _pending_fetch_cache (line 366) before it touches self.redis. So after close(), requests fetched in an earlier batch are still returned without an error. Check that the backend is closed before reading that cache, so the behavior matches "storages opened through a closed backend raise an error".

✍️ Drafted by Claude Code

@override
def get_storage_backend_cache_key(self, configuration: Configuration) -> Hashable:
# Storages stop working when their backend closes, so a different backend instance must not get them.
return super().get_storage_backend_cache_key(configuration), id(self)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: With id(self) in the key, every backend instance adds its own entries to the global storage cache, and close() never removes them. A closed backend, its client and its storages stay in memory for the life of the process. Also, Dataset.open(storage_backend=closed_backend) returns the cached instance without an error, and the error only comes on the first operation. Should close() remove its cache entries?

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep the current behavior. Storage backends don't know about the storage cache, only the storages do, and an entry leaves the cache on drop(). Removing entries in close() would make the backend reach into the global cache and turn that dependency around. What stays behind is a few objects per backend instance with no open connections, and a process normally creates one backend.

Comment thread docs/06_upgrading/upgrading_to_v2.md Outdated
```

`Configuration.purge_on_start` isn't affected. Unnamed storages are still purged once per process, when they're first opened.
`Configuration.purge_on_start` isn't affected. Unnamed storages are still purged when a storage backend first opens them.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This holds for Redis and SQL only. MemoryStorageBackend and FileSystemStorageBackend (same storage_dir) still share cached storages across instances, so a second instance doesn't purge. #2464 changes the same line and adds a section at the same place, so the second PR to merge will conflict here.

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted the line here. #2464 updates that text and will cover which backends it applies to. The new sections still land at the same place, so I'll resolve that conflict when rebasing whichever PR merges second.

connection_string='redis://localhost:6379'
) as storage_backend:
# And pass it to the crawler.
crawler = ParselCrawler(storage_backend=storage_backend)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The async with block ends right after the crawler is created, so the backend is closed before anything runs. A user who adds await crawler.run(...) after the block gets "The RedisStorageBackend is closed". Running the crawler inside the block (as the upgrading guide does) would show the right pattern. Same in redis_storage_backend_configuration_example.py.

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants