Skip to content

feat(crawlers)!: Add per-route timeouts and extend_timeout() - #2429

Merged
vdusek merged 4 commits into
apify:v2from
Mantisus:timeouts-routes
Nov 17, 2026
Merged

vdusek merged 4 commits into
apify:v2from
Mantisus:timeouts-routes

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Router.handler() and Router.default_handler() take a request_handler_timeout that replaces the crawler's timeout for the requests the handler handles. context.extend_timeout() gives the running part of the request processing more time, as in Crawlee JS. Called from a navigation hook, it extends the navigation timeout. Called from the request handler or an error handler, it extends the timeout of that handler. It also extends the lock of the request in the request manager.

The crawler tells the request manager how long it holds a request, based on its longest request handler timeout, including the router handlers, so SqlStorageBackend and RedisStorageBackend keep the request locked long enough. HTTP crawlers bound the navigation request and reading its body with navigation_timeout themselves, so an extension reaches the body. With FileDownloadCrawler(stream=True), the body streamed in the request handler counts towards request_handler_timeout. The upgrade guide covers the changes.

Issues

✍️ Drafted by Claude Code

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 9, 2026
@Mantisus
Mantisus marked this pull request as draft November 9, 2026 22:17
@Mantisus Mantisus self-assigned this Nov 9, 2026
@Mantisus
Mantisus marked this pull request as ready for review November 12, 2026 13:46
@Mantisus
Mantisus requested a review from vdusek November 12, 2026 13:47
@vdusek vdusek changed the title feat(crawlers)!: Add per-route request handler timeouts and context.extend_timeout() feat(crawlers)!: Add per-route timeouts and extend_timeout() Nov 16, 2026
Comment thread src/crawlee/router.py Resolved
Comment thread docs/04_guides/11_secure_scraping.mdx Resolved
@Mantisus
Mantisus requested a review from vdusek November 16, 2026 22:03
@vdusek

vdusek commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Hi @Mantisus, maybe you forgot to push? 🙂 (there are no new commits)

@Mantisus

Copy link
Copy Markdown
Collaborator Author

Hi @Mantisus, maybe you forgot to push? 🙂 (there are no new commits)

Yes, sorry...🤦

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit 427d798 into apify:v2 Nov 17, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow per-route request handler timeouts and extending the timeout from the handler

2 participants