Skip to content

fix(crawlers)!: Keep statistics and sessions between run() calls - #2449

Open
Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:keep-state-across-runs
Open

Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:keep-state-across-runs

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • A repeated run() continues the crawl: it adds to the statistics of the earlier runs, error trackers included, and keeps the sessions of the session pool. max_requests_per_crawl counts the requests of the earlier runs too, so a run that starts with the limit already reached stops right away. To count from zero, call crawler.statistics.reset() before the next run().
  • The same change for Crawlee JS is proposed in v4: A repeated run() continues the queue but resets the statistics the crawler built crawlee#4397.

Issues

Testing

  • Unit and integration tests for repeated runs with own and passed statistics and session pools.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 15, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from e3dc508 to 41411cc Compare November 15, 2026 22:39
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from 41411cc to 8dbf734 Compare November 18, 2026 14:50
@Mantisus
Mantisus marked this pull request as ready for review November 18, 2026 14:50
@Mantisus
Mantisus requested a review from vdusek November 18, 2026 14:50
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from 8dbf734 to 0316453 Compare November 18, 2026 15:10
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from 0316453 to dfe7541 Compare November 18, 2026 16:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant