Skip to content

feat(crawlers)!: Remove use_session_pool and always use sessions - #2439

Merged
vdusek merged 6 commits into
apify:v2from
Mantisus:remove-use-session-pool
Nov 17, 2026
Merged

vdusek merged 6 commits into
apify:v2from
Mantisus:remove-use-session-pool

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The use_session_pool option is removed. Every crawler now processes each request with a session from its SessionPool, so context.session is always a Session and handlers no longer need if context.session checks. A request bound by session_id to a session that's no longer usable still fails with RequestCollisionError. Its failed request handler gets that session while the pool still holds it, otherwise a retired session with the bound id.

For crawlers that ran without the pool, 401, 403 and 429 responses now rotate the session, and with proxy_urls a session keeps its proxy.

Issues

Testing

  • Updated the tests that ran without the pool and added a unit test for the failed request handler of a request whose bound session is gone.

✍️ Drafted by Claude Code

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 12, 2026
@Mantisus Mantisus self-assigned this Nov 12, 2026
@Mantisus
Mantisus requested a review from vdusek November 12, 2026 12:28
@vdusek vdusek changed the title feat(crawlers)!: Remove use_session_pool and always process requests with a session feat(crawlers)!: Remove use_session_pool and always use sessions Nov 16, 2026
Comment thread docs/06_upgrading/upgrading_to_v2.md Resolved
Comment thread docs/06_upgrading/upgrading_to_v2.md Resolved
Comment thread docs/06_upgrading/upgrading_to_v2.md Resolved
@Mantisus
Mantisus requested a review from vdusek November 16, 2026 16:50
…rror`

BREAKING CHANGE: The `RequestCollisionError` message for a request whose bound session is no longer usable changed from "... is no longer available in SessionPool" to "... is no longer usable".

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit efef3e9 into apify:v2 Nov 17, 2026
41 checks passed
vdusek pushed a commit that referenced this pull request Nov 18, 2026
### Description

- A crawler now enters and exits only the objects it creates. A session
pool, browser pool, HTTP client, rendering type predictor or PydanticAI
extractor passed to it is the caller's to enter before `run()`, and
`run()` raises otherwise. The crawler no longer closes it after a run or
resets it on the next one. `Statistics` stay the exception: the crawler
enters them on every run, so crawlers can share an instance one after
another, but not at the same time.
- `PydanticAiDirectExtractor` becomes an async context manager, and
session pools, statistics, HTTP clients and the default rendering type
predictor can be entered again after a failed or cancelled entry or
exit. For details, see "Crawlers manage only what they create" in the
upgrade guide. The docs examples and project templates follow in a
separate PR.

### Issues

- Part of: #2301
- Follow-up: #2439

### Testing

- New unit and integration tests for passed and own objects of every
crawler, and for failed or cancelled entries and exits.

*✍️ Drafted by Claude Code*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store the proxy in Session and drop session and request arguments from ProxyConfiguration Move blocked_status_codes from Session to the crawler

2 participants