Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/03_concepts/03_crawling_context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ The context provides helpers that operate on the crawler's storages and request
- `get_key_value_store` opens a key-value store, the default one unless you pass `id`, `name` or `alias`.
- `use_state` returns a dictionary shared by all handler calls of the crawler. The crawler keeps it in the default key-value store and saves it automatically. This helper is experimental and its interface may change.
- `send_request` sends an extra HTTP request through the crawler's HTTP client, with the session and proxy of the current request. Its response isn't parsed, and the request doesn't go through the request queue.
- `extend_timeout` gives the running request handler, error handler or navigation hook more time and keeps the request locked in the request queue for longer. For details, see [Handler timeouts](./request-router#handler-timeouts).

The following example uses most of these helpers in a <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> handler and reads the results back from the crawler once the run finishes.

Expand Down
11 changes: 11 additions & 0 deletions docs/03_concepts/04_request_router.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import CodeBlock from '@theme/CodeBlock';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import BasicRequestHandlers from '!!raw-loader!roa-loader!./code_examples/request_router/basic_request_handlers.py';
import HandlerTimeouts from '!!raw-loader!roa-loader!./code_examples/request_router/handler_timeouts.py';
import SimpleDefaultHandler from '!!raw-loader!roa-loader!./code_examples/request_router/simple_default_handler.py';
import CustomRouterDefaultOnly from '!!raw-loader!roa-loader!./code_examples/request_router/custom_router_default_only.py';
import HttpPreNavigation from '!!raw-loader!roa-loader!./code_examples/request_router/http_pre_navigation.py';
Expand Down Expand Up @@ -58,6 +59,16 @@ More complex crawling projects often require different processing logic for vari
{BasicRequestHandlers}
</RunnableCodeBlock>

### Handler timeouts

The `request_handler_timeout` option of the crawler applies to every request handler. When one type of page needs more or less time than the others, pass `request_handler_timeout` to <ApiLink to="class/Router#handler">`router.handler()`</ApiLink> or <ApiLink to="class/Router#default_handler">`router.default_handler()`</ApiLink>. Handlers that don't set it keep the crawler's timeout. The timeout follows the label the request has before the [middlewares](#middleware) run. If a middleware changes the label, the new handler runs with the timeout of the old one.

When the time a request needs only becomes clear while the handler runs, await <ApiLink to="class/BasicCrawlingContext#extend_timeout">`context.extend_timeout()`</ApiLink> to give the handler more time. It also keeps the request locked in the request queue for longer. Called from a pre-navigation or post-navigation hook, it extends the navigation timeout instead. It raises an error when the delta isn't positive or when it's awaited on another event loop, such as one started in a separate thread. Callbacks that run outside the request processing, such as Playwright event handlers, can't extend the timeout.

<RunnableCodeBlock className="language-python" language="python">
{HandlerTimeouts}
</RunnableCodeBlock>

## Middleware

Middlewares are functions registered with <ApiLink to="class/Router#use">`router.use()`</ApiLink> that execute before the matched request handler on every request, regardless of the request label. Multiple middlewares can be registered and are executed sequentially in the order they were registered. If a middleware raises an exception, the execution chain is interrupted and the handler is not called.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,7 @@ async def main() -> None:
# been read yet.
crawler = FileDownloadCrawler(
stream=True,
# Bounds the whole download: the connection, the response headers and the body.
navigation_timeout=timedelta(minutes=5),
# The body is downloaded inside the handler, so this bounds the transfer as well.
# The body is downloaded inside the handler, so this bounds the transfer.
request_handler_timeout=timedelta(minutes=5),
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
Expand Down
55 changes: 55 additions & 0 deletions docs/03_concepts/code_examples/request_router/handler_timeouts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import asyncio
from datetime import timedelta
from urllib.parse import urljoin

from crawlee import Request
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.router import Router


async def main() -> None:
router = Router[ParselCrawlingContext]()

# Category pages list many products, so give them more time than the rest
@router.handler('CATEGORY', request_handler_timeout=timedelta(minutes=2))
async def category_handler(context: ParselCrawlingContext) -> None:
context.log.info('Processing category page: %s', context.request.url)
await context.enqueue_links(selector='a[href*="/products/"]', label='PRODUCT')

# Product pages keep the timeout of the crawler
@router.handler('PRODUCT')
async def product_handler(context: ParselCrawlingContext) -> None:
context.log.info('Processing product page: %s', context.request.url)
image_urls = set(context.selector.css('.product-gallery img::attr(src)').getall())

# The handler downloads every image below, and their number is known only now,
# so ask for 5 more seconds per image
if image_urls:
await context.extend_timeout(timedelta(seconds=5 * len(image_urls)))

image_sizes = []
for image_url in image_urls:
response = await context.send_request(urljoin(context.request.url, image_url))
image = await response.read()
image_sizes.append(len(image))

await context.push_data({'url': context.request.url, 'image_sizes': image_sizes})

crawler = ParselCrawler(
request_handler=router,
request_handler_timeout=timedelta(seconds=30),
max_requests_per_crawl=10, # Limit the max requests per crawl.
)

await crawler.run(
[
Request.from_url(
'https://warehouse-theme-metal.myshopify.com/collections/all',
label='CATEGORY',
),
]
)


if __name__ == '__main__':
asyncio.run(main())
2 changes: 1 addition & 1 deletion docs/04_guides/11_secure_scraping.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ If only your crawler's storage sees the response, the same request is a *blind*
A hostile site can try to use up your CPU, memory, or queue rather than read anything:

- A *crawler trap* is a maze of randomly generated URLs linking to more generated pages. It's built to keep a crawler running forever and grow its queue without bound. Bound how far link-following goes with <ApiLink to="class/BasicCrawlerOptions#max_crawl_depth">`max_crawl_depth`</ApiLink>, and cap a whole run with <ApiLink to="class/BasicCrawlerOptions#max_requests_per_crawl">`max_requests_per_crawl`</ApiLink>. Both are unlimited by default, so set them yourself.
- A single *oversized or slow response* can tie up a worker or blow up memory. Both the fetch and the handler are capped at 60 seconds by default. <ApiLink to="class/HttpCrawlerOptions#navigation_timeout">`navigation_timeout`</ApiLink> covers the fetch, including reading the body, and is passed straight to the HTTP client. <ApiLink to="class/BasicCrawlerOptions#request_handler_timeout">`request_handler_timeout`</ApiLink> covers the handler. Raise them if a legitimate target needs longer, but keep the cap as tight as your workload allows.
- A single *oversized or slow response* can tie up a worker or blow up memory. Both the fetch and the handler are capped at 60 seconds by default. <ApiLink to="class/HttpCrawlerOptions#navigation_timeout">`navigation_timeout`</ApiLink> covers the fetch, including reading the body. <ApiLink to="class/BasicCrawlerOptions#request_handler_timeout">`request_handler_timeout`</ApiLink> covers the handler. With <ApiLink to="class/FileDownloadCrawler">`FileDownloadCrawler`</ApiLink> in stream mode, reading the body counts towards the handler timeout instead. Raise them if a legitimate target needs longer, but keep the cap as tight as your workload allows.
- A *decompression bomb* is an archive that expands to gigabytes when unpacked. If your crawler downloads and extracts archives, cap the output stream during extraction. The declared size is part of the attack and easy to forge.

## Untrusted content
Expand Down
10 changes: 8 additions & 2 deletions docs/06_upgrading/upgrading_to_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,11 @@ The memory backend now handles values the same way as the file-system backend. I

`SqlStorageBackend` and `RedisStorageBackend` now lock a fetched request for 3 minutes by default, the same as Crawlee JS. In v1, the lock lasted 5 minutes. When a lock expires before the request is handled or reclaimed, another client sharing the queue can fetch the request and process it again.

If processing a request can take longer than 3 minutes, call `RequestQueue.set_expected_request_processing_time()` with the longest time you expect. Every request fetched after that call stays locked at least that long. To give a single request that's still being processed more time, call `RequestQueue.extend_request_processing_time()`.
A crawler keeps the lock at least as long as its longest request handler timeout plus 1 minute for the error handling, counting the timeouts of its router handlers. HTTP and Playwright crawlers add their navigation timeout, and `AdaptivePlaywrightCrawler` counts the request handler timeout twice, since it can run both sub crawlers on one request. The crawler caps this lock time at a day and logs a warning when its timeouts add up to more. If processing a request can take longer, call `RequestQueue.set_expected_request_processing_time()` with the longest time you expect. Every request fetched after that call stays locked at least that long. To give a single request that's still being processed more time, await `context.extend_timeout()` from the request handler. Outside a crawler, call `RequestQueue.extend_request_processing_time()`.

## `BasicCrawlingContext` requires `extend_timeout`

`BasicCrawlingContext` has a new required field, `extend_timeout`. If you build a crawling context yourself, such as in a test, pass it as well.

## Crawler `id` is a string

Expand Down Expand Up @@ -385,7 +389,7 @@ Responses returned by `context.send_request` are closed after the request is pro

The `timeout` of `send_request` is one deadline for the whole redirect chain and for reading the body, the same as Crawlee JS. A body read past it raises `TimeoutError`. By default, `timeout` is `'default'` and the default of the underlying library applies to each request of the chain. `ImpitHttpClient` allows 30 seconds, including reading the body. `CurlImpersonateHttpClient` allows 30 seconds to connect and 30 seconds without receiving data. `HttpxHttpClient` allows 5 seconds per operation. To change the default, pass `timeout` when you create the client.

The crawlers pass the remaining `navigation_timeout` to the HTTP client, so it bounds reading the body too. With `FileDownloadCrawler(stream=True)`, the deadline also covers the body streamed in the request handler, with every HTTP client. In v1, only `ImpitHttpClient` bounded the streamed body, while `HttpxHttpClient` and `CurlImpersonateHttpClient` bounded only the connection. For long downloads, raise `navigation_timeout` together with `request_handler_timeout`.
The crawlers bound the navigation request and reading its body with `navigation_timeout` themselves and send the request with `timeout=None`, so `context.extend_timeout()` can extend it. With `FileDownloadCrawler(stream=True)`, the body streamed in the request handler counts towards `request_handler_timeout`, so a body that stalls fails as a request handler timeout. In v1, `ImpitHttpClient` bounded the streamed body by the remaining `navigation_timeout`, while `HttpxHttpClient` and `CurlImpersonateHttpClient` bounded only the connection. For long downloads, raise `request_handler_timeout` or extend it from the handler.

### `timeout=None` disables the timeout

Expand Down Expand Up @@ -578,6 +582,8 @@ The warnings about an HTTP 429 that the request manager doesn't pace are reworde

The `ThrottlingRequestManager` warning about capping a delay now calls it a "requested wait" delay instead of a "Retry-After header" delay. Its DEBUG messages about a 429 ignored during an active backoff and about resetting a domain's rate limit state are removed, and its DEBUG message about a domain's crawl-delay shows the delay with one decimal place.

The error of an `AdaptivePlaywrightCrawler` sub crawler that timed out now reads `rendering_type=static timed out after 60.0 seconds`, with a space before `seconds`.

Log records hold the message template and its arguments separately. `record.msg` is the template, such as `Retrying request to %s due to: %s. %s`, and `record.args` holds the values. To read the final text, use `record.getMessage()`, which formatters and handlers already do.

## Session retirement is permanent
Expand Down
26 changes: 26 additions & 0 deletions src/crawlee/_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,29 @@ def __call__(
"""


@docs_group('Functions')
class ExtendTimeoutFunction(Protocol):
"""A function for giving the running part of the request processing more time.

Called from a navigation hook, it extends the navigation timeout. Called from the request handler or an error
handler, it extends the timeout of that handler. It also keeps the request locked in the request manager for longer.
Use it when the time a request needs only becomes apparent while it runs. Otherwise, set `request_handler_timeout`
on the crawler or on the router handler. It doesn't extend a timeout that has already expired, or any timeout when
called from a callback that runs outside the request processing, such as a Playwright event callback.
"""

def __call__(self, delta: timedelta) -> Coroutine[None, None, None]:
"""Call dunder method.

Args:
delta: How much more time to give.

Raises:
ValueError: If `delta` isn't positive.
RuntimeError: If awaited on another event loop, such as one started in a separate thread.
"""


@docs_group('Functions')
class ExtractLinksFunction(Protocol):
"""A function for extracting URLs to crawl based on elements selected by a given selector.
Expand Down Expand Up @@ -683,6 +706,9 @@ class BasicCrawlingContext:
register_deferred_cleanup: Callable[[DeferredCleanupCallback], None]
"""Register an async callback to be called after request processing completes (including error handlers)."""

extend_timeout: ExtendTimeoutFunction
"""Give the running part of the request processing more time."""

async def get_snapshot(self) -> PageSnapshot:
"""Get snapshot of crawled page."""
return PageSnapshot()
Expand Down
82 changes: 74 additions & 8 deletions src/crawlee/_utils/time.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,25 @@
import asyncio
import time
from contextlib import contextmanager
from contextvars import ContextVar, copy_context
from dataclasses import dataclass
from datetime import timedelta
from functools import partial
from typing import TYPE_CHECKING

if TYPE_CHECKING:
from collections.abc import Generator
from collections.abc import Callable, Generator
from contextvars import Context, Token
from types import TracebackType

_TimeoutExtender = Callable[[timedelta], bool]

_SECONDS_PER_MINUTE = 60
_SECONDS_PER_HOUR = 3600

_active_timeout_extenders: ContextVar[tuple[_TimeoutExtender, ...]] = ContextVar('active_timeout_extenders', default=())
"""Extenders of the `SharedTimeout`s entered in the current context, from the outermost to the innermost."""


@dataclass
class TimerResult:
Expand All @@ -40,13 +48,20 @@ def measure_time() -> Generator[TimerResult]:
class SharedTimeout:
"""Keeps track of a time budget shared by multiple independent async operations.

Provides a reusable, non-reentrant context manager interface.
Provides a reusable, non-reentrant context manager interface. While entered, `extend_active_timeouts` can extend it.
"""

def __init__(self, timeout: timedelta) -> None:
self._total_timeout = timeout
self._remaining_timeout = timeout
self._active_timeout: asyncio.Timeout | None = None
self._activation_timestamp: float | None = None
self._extenders_token: Token[tuple[_TimeoutExtender, ...]] | None = None

@property
def total_timeout(self) -> timedelta:
"""The whole time budget, including extensions."""
return self._total_timeout

async def __aenter__(self) -> timedelta:
if self._active_timeout is not None or self._activation_timestamp is not None:
Expand All @@ -55,6 +70,10 @@ async def __aenter__(self) -> timedelta:
self._activation_timestamp = time.monotonic()
self._active_timeout = new_timeout = asyncio.timeout(self._remaining_timeout.total_seconds())
await new_timeout.__aenter__()

# Entering and exiting must run in the same context. Otherwise, resetting the token fails.
extender = partial(self._extend, new_timeout)
self._extenders_token = _active_timeout_extenders.set((*_active_timeout_extenders.get(), extender))
return self._remaining_timeout

async def __aexit__(
Expand All @@ -63,15 +82,62 @@ async def __aexit__(
exc_value: BaseException | None,
exc_traceback: TracebackType | None,
) -> None:
if self._active_timeout is None or self._activation_timestamp is None:
if self._active_timeout is None or self._activation_timestamp is None or self._extenders_token is None:
raise RuntimeError('Logic error')

await self._active_timeout.__aexit__(exc_type, exc_value, exc_traceback)
elapsed = time.monotonic() - self._activation_timestamp
self._remaining_timeout = self._remaining_timeout - timedelta(seconds=elapsed)
try:
await self._active_timeout.__aexit__(exc_type, exc_value, exc_traceback)
finally:
elapsed = time.monotonic() - self._activation_timestamp
self._remaining_timeout = self._remaining_timeout - timedelta(seconds=elapsed)

extenders_token = self._extenders_token
self._active_timeout = None
self._activation_timestamp = None
self._extenders_token = None
_active_timeout_extenders.reset(extenders_token)

def _extend(self, entered_timeout: asyncio.Timeout, delta: timedelta) -> bool:
"""Extend the budget by `delta` and return whether the enclosing timeouts should be extended as well."""
# A timeout that has been exited since only gives its later uses the extra time.
is_running = entered_timeout is self._active_timeout
if is_running and entered_timeout.expired():
return False

if is_running:
deadline = entered_timeout.when()
if deadline is not None:
entered_timeout.reschedule(deadline + delta.total_seconds())

self._total_timeout = _add_saturating(self._total_timeout, delta)
self._remaining_timeout = _add_saturating(self._remaining_timeout, delta)
return is_running


def _add_saturating(duration: timedelta, delta: timedelta) -> timedelta:
"""Add `delta` to `duration`, staying at `timedelta.max` when the sum doesn't fit."""
try:
return duration + delta
except OverflowError:
return timedelta.max


def extend_active_timeouts(delta: timedelta) -> None:
"""Extend the `SharedTimeout`s entered in the current context by `delta`, from the innermost outwards.

Stops at a timeout that is no longer running, so the ones that still bound the caller stay as they are. Must be
called on the event loop the timeouts run on.
"""
for extend in reversed(_active_timeout_extenders.get()):
if not extend(delta):
return


self._active_timeout = None
self._activation_timestamp = None
def context_without_active_timeouts() -> Context:
"""Copy the current context without the entered `SharedTimeout`s, so the code running in it can't extend them."""
context = copy_context()
context.run(_active_timeout_extenders.set, ())
return context


def format_duration(duration: timedelta | None) -> str:
Expand Down
Loading
Loading