Skip to content

refactor(crawlers)!: Enter and exit only what a crawler creates - #2445

Merged
vdusek merged 2 commits into
apify:v2from
Mantisus:ownership
Nov 18, 2026
Merged

vdusek merged 2 commits into
apify:v2from
Mantisus:ownership

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • A crawler now enters and exits only the objects it creates. A session pool, browser pool, HTTP client, rendering type predictor or PydanticAI extractor passed to it is the caller's to enter before run(), and run() raises otherwise. The crawler no longer closes it after a run or resets it on the next one. Statistics stay the exception: the crawler enters them on every run, so crawlers can share an instance one after another, but not at the same time.
  • PydanticAiDirectExtractor becomes an async context manager, and session pools, statistics, HTTP clients and the default rendering type predictor can be entered again after a failed or cancelled entry or exit. For details, see "Crawlers manage only what they create" in the upgrade guide. The docs examples and project templates follow in a separate PR.

Issues

Testing

  • New unit and integration tests for passed and own objects of every crawler, and for failed or cancelled entries and exits.

✍️ Drafted by Claude Code

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 14, 2026
@Mantisus Mantisus self-assigned this Nov 14, 2026
@Mantisus
Mantisus requested a review from vdusek November 14, 2026 20:24
@Mantisus
Mantisus requested a review from janbuchar November 15, 2026 18:39
@vdusek vdusek changed the title refactor(crawlers)!: Enter and exit only the objects a crawler creates fix(crawlers)!: Enter and exit only what a crawler creates Nov 16, 2026
Comment thread docs/06_upgrading/upgrading_to_v2.md Resolved
@Mantisus Mantisus changed the title fix(crawlers)!: Enter and exit only what a crawler creates refactor(crawlers)!: Enter and exit only what a crawler creates Nov 16, 2026
@Mantisus
Mantisus requested a review from vdusek November 17, 2026 00:18

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit fda3f40 into apify:v2 Nov 18, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let crawlers manage only the collaborators they create and extract per-run state

2 participants