Skip to content

fix(sql)!: Keep a passed-in engine open and open storages per backend - #2464

Open
Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:sql-backend-close
Open

Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:sql-backend-close

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

SqlStorageBackend.close() now disposes only the engine the backend created, from a connection string or for the default SQLite database. An engine you pass in stays open, so dispose it yourself. After close(), the backend and the storages opened through it raise an error instead of silently opening a new connection pool.

Each SqlStorageBackend instance now opens its own storages, so a new backend doesn't get the storages of a closed one. With purge_on_start enabled, a new backend on the same database purges the unnamed storages again.

Testing

  • Unit tests for disposing only the backend's own engine, the closed backend, a backend closed before its first use, and separate storages per backend.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 17, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 17, 2026
@Mantisus
Mantisus requested a review from vdusek November 17, 2026 15:10
@vdusek vdusek changed the title fix(storages)!: Close only the SQL backend's own engine and keep each backend's storages separate fix(sql)!: Keep a passed-in engine open and open storages per backend Nov 18, 2026
@override
def get_storage_backend_cache_key(self, configuration: Configuration) -> Hashable:
# Storages stop working when their backend closes, so a different backend instance must not get them.
return super().get_storage_backend_cache_key(configuration), id(self)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: With id(self) in the key, every backend instance adds its own entries to the global storage cache, and close() never removes them. A closed backend and its storages stay in memory for the life of the process. Also, Dataset.open(storage_backend=closed_backend) returns the cached instance without an error, and the error only comes on the first operation. Should close() remove its cache entries?

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep the current behavior. Storage backends don't know about the storage cache, only the storages do, and an entry leaves the cache on drop(). Removing entries in close() would make the backend reach into the global cache and turn that dependency around. What stays behind is a few objects per backend instance with no open connections, and a process normally creates one backend.

dataset = await Dataset.open(name='results', storage_backend=storage_backend, configuration=configuration)
await dataset.push_data({'key': 'value'})

assert dataset is not old_dataset

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The docs and the upgrading guide warn that a new backend with purge_on_start=True on the same database wipes the unnamed storages an earlier one wrote. No test covers it. A test could push an item with backend A, open the default dataset with backend B (purge_on_start=True), and assert it's empty.

✍️ Drafted by Claude Code

Comment thread docs/06_upgrading/upgrading_to_v2.md Outdated
```

`Configuration.purge_on_start` isn't affected. Unnamed storages are still purged once per process, when they're first opened.
`Configuration.purge_on_start` isn't affected. Unnamed storages are still purged when a storage backend first opens them.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This holds for SQL and Redis only. MemoryStorageBackend and FileSystemStorageBackend (same storage_dir) still share cached storages across instances, so a second instance doesn't purge. #2463 changes the same line and adds a section at the same place, so the second PR to merge will conflict here.

✍️ Drafted by Claude Code

@Mantisus
Mantisus requested a review from vdusek November 18, 2026 13:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants