Repository navigation
Conversation
| requests_json = await self._fetch_script( | ||
| keys=[self._queue_key, self._in_progress_key, self._data_key], | ||
| args=[self.client_key, blocked_until_timestamp, self._MAX_BATCH_FETCH_SIZE], | ||
| client=self.redis, |
There was a problem hiding this comment.
Suggestion: fetch_next_request() returns from _pending_fetch_cache (line 366) before it touches self.redis. So after close(), requests fetched in an earlier batch are still returned without an error. Check that the backend is closed before reading that cache, so the behavior matches "storages opened through a closed backend raise an error".
✍️ Drafted by Claude Code
| @override | ||
| def get_storage_backend_cache_key(self, configuration: Configuration) -> Hashable: | ||
| # Storages stop working when their backend closes, so a different backend instance must not get them. | ||
| return super().get_storage_backend_cache_key(configuration), id(self) |
There was a problem hiding this comment.
Question: With id(self) in the key, every backend instance adds its own entries to the global storage cache, and close() never removes them. A closed backend, its client and its storages stay in memory for the life of the process. Also, Dataset.open(storage_backend=closed_backend) returns the cached instance without an error, and the error only comes on the first operation. Should close() remove its cache entries?
✍️ Drafted by Claude Code
There was a problem hiding this comment.
I'd keep the current behavior. Storage backends don't know about the storage cache, only the storages do, and an entry leaves the cache on drop(). Removing entries in close() would make the backend reach into the global cache and turn that dependency around. What stays behind is a few objects per backend instance with no open connections, and a process normally creates one backend.
| ``` | ||
|
|
||
| `Configuration.purge_on_start` isn't affected. Unnamed storages are still purged once per process, when they're first opened. | ||
| `Configuration.purge_on_start` isn't affected. Unnamed storages are still purged when a storage backend first opens them. |
There was a problem hiding this comment.
Nit: This holds for Redis and SQL only. MemoryStorageBackend and FileSystemStorageBackend (same storage_dir) still share cached storages across instances, so a second instance doesn't purge. #2464 changes the same line and adds a section at the same place, so the second PR to merge will conflict here.
✍️ Drafted by Claude Code
There was a problem hiding this comment.
Reverted the line here. #2464 updates that text and will cover which backends it applies to. The new sections still land at the same place, so I'll resolve that conflict when rebasing whichever PR merges second.
| connection_string='redis://localhost:6379' | ||
| ) as storage_backend: | ||
| # And pass it to the crawler. | ||
| crawler = ParselCrawler(storage_backend=storage_backend) |
There was a problem hiding this comment.
Nit: The async with block ends right after the crawler is created, so the backend is closed before anything runs. A user who adds await crawler.run(...) after the block gets "The RedisStorageBackend is closed". Running the crawler inside the block (as the upgrading guide does) would show the right pattern. Same in redis_storage_backend_configuration_example.py.
✍️ Drafted by Claude Code
Description
RedisStorageBackendcan now be closed withclose()or used as an async context manager. It closes the Redis client it created from a connection string and leaves a client you pass in open. Storages opened through a closed backend raise an error instead of opening new connections.Each
RedisStorageBackendinstance now opens its own storages, so a new backend doesn't get the storages of a closed one. Withpurge_on_startenabled, a new backend on the same database purges the unnamed storages again.When two clients create the same storage at once, the one that loses the race now uses the winner's ID, and it raises an error if the winner's metadata never appears.
Testing
✍️ Drafted by Claude Code