Repository navigation
Conversation
| @override | ||
| def get_storage_backend_cache_key(self, configuration: Configuration) -> Hashable: | ||
| # Storages stop working when their backend closes, so a different backend instance must not get them. | ||
| return super().get_storage_backend_cache_key(configuration), id(self) |
There was a problem hiding this comment.
Question: With id(self) in the key, every backend instance adds its own entries to the global storage cache, and close() never removes them. A closed backend and its storages stay in memory for the life of the process. Also, Dataset.open(storage_backend=closed_backend) returns the cached instance without an error, and the error only comes on the first operation. Should close() remove its cache entries?
✍️ Drafted by Claude Code
There was a problem hiding this comment.
I'd keep the current behavior. Storage backends don't know about the storage cache, only the storages do, and an entry leaves the cache on drop(). Removing entries in close() would make the backend reach into the global cache and turn that dependency around. What stays behind is a few objects per backend instance with no open connections, and a process normally creates one backend.
| dataset = await Dataset.open(name='results', storage_backend=storage_backend, configuration=configuration) | ||
| await dataset.push_data({'key': 'value'}) | ||
|
|
||
| assert dataset is not old_dataset |
There was a problem hiding this comment.
Suggestion: The docs and the upgrading guide warn that a new backend with purge_on_start=True on the same database wipes the unnamed storages an earlier one wrote. No test covers it. A test could push an item with backend A, open the default dataset with backend B (purge_on_start=True), and assert it's empty.
✍️ Drafted by Claude Code
| ``` | ||
|
|
||
| `Configuration.purge_on_start` isn't affected. Unnamed storages are still purged once per process, when they're first opened. | ||
| `Configuration.purge_on_start` isn't affected. Unnamed storages are still purged when a storage backend first opens them. |
There was a problem hiding this comment.
Nit: This holds for SQL and Redis only. MemoryStorageBackend and FileSystemStorageBackend (same storage_dir) still share cached storages across instances, so a second instance doesn't purge. #2463 changes the same line and adds a section at the same place, so the second PR to merge will conflict here.
✍️ Drafted by Claude Code
Description
SqlStorageBackend.close()now disposes only the engine the backend created, from a connection string or for the default SQLite database. An engine you pass in stays open, so dispose it yourself. Afterclose(), the backend and the storages opened through it raise an error instead of silently opening a new connection pool.Each
SqlStorageBackendinstance now opens its own storages, so a new backend doesn't get the storages of a closed one. Withpurge_on_startenabled, a new backend on the same database purges the unnamed storages again.Testing
✍️ Drafted by Claude Code