Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion docs/06_upgrading/upgrading_to_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,39 @@ class MyHttpClient(HttpClient):

`SitemapRequestLoader.fetch_next_request` skips a URL while its request is still in progress, for example when another sitemap lists it too. In v1, it returned a request for the URL again, and `get_total_count` counted the URL twice.

## Crawlers manage only what they create

A crawler now enters and exits only the objects it creates, the same as Crawlee JS. An object you pass is yours to enter and exit. The `Statistics` and the event managers are the exceptions, since the crawler enters them on every run. You can enter an event manager yourself as well. In v1, the crawler entered a passed object that wasn't active yet and exited it when its run finished, even while another crawler still used it.
Comment thread
Mantisus marked this conversation as resolved.
Resolved

What you do with an object you pass depends on its type:

- `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
- `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time.

For example:

```python
# v1
crawler = ParselCrawler(session_pool=SessionPool(max_pool_size=10))
await crawler.run(urls)

# v2
session_pool = SessionPool(max_pool_size=10)
crawler = ParselCrawler(session_pool=session_pool)

async with session_pool:
await crawler.run(urls)
```

When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.

`RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`.

`AdaptivePlaywrightCrawler.rendering_type_predictor` is now read-only. Pass your predictor to the constructor instead.

The private `_additional_context_managers` argument of `BasicCrawler` is removed. To have the crawler enter and exit an object on every run, a subclass builds the object with `self._own_or_borrow()`. A subclass that gives its parent class a default object, such as an HTTP client, builds it this way before it calls `super().__init__()`.

## Resetting statistics and the session pool

`Statistics.reset()` is now synchronous and resets only the in-memory statistics, the same as Crawlee JS. In v1, it also cleared the persisted record. To clear the record too, call the new `reset_store()` once the statistics are exited:
Expand Down Expand Up @@ -599,16 +632,21 @@ To keep the response cookies out of the sessions, pass `save_response_cookies=Fa
```python
# v1
crawler = ParselCrawler(use_session_pool=False)
await crawler.run(urls)

# v2
from crawlee.sessions import SessionCookies, SessionPool

session_pool = SessionPool(create_session_settings={'max_usage_count': 1})
crawler = ParselCrawler(
retry_on_blocked=False,
save_response_cookies=False,
session_pool=SessionPool(create_session_settings={'max_usage_count': 1}),
session_pool=session_pool,
)

async with session_pool:
await crawler.run(urls)

# In a request handler
response = await context.send_request(url, cookies=SessionCookies())
```
Expand Down
33 changes: 32 additions & 1 deletion src/crawlee/_utils/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,42 @@
import inspect
from collections.abc import Callable
from functools import wraps
from typing import Any, TypeVar, cast
from typing import TYPE_CHECKING, Any, Protocol, TypeVar, cast

if TYPE_CHECKING:
from types import TracebackType

T = TypeVar('T', bound=Callable[..., Any])


class Collaborator(Protocol):
"""An async context manager that reports through `active` whether it's entered."""

@property
def active(self) -> bool:
"""Whether the object is entered and not exited yet."""

async def __aenter__(self) -> Any:
"""Prepare the object for use.

Raises:
RuntimeError: If the object is already active.
"""

async def __aexit__(
self,
exc_type: type[BaseException] | None,
exc_value: BaseException | None,
exc_traceback: TracebackType | None,
/,
) -> Any:
"""Release what the object holds.

Raises:
RuntimeError: If the object is not active.
"""


def ensure_context(method: T) -> T:
"""Ensure the (async) context manager is initialized before executing the method.

Expand Down
10 changes: 4 additions & 6 deletions src/crawlee/autoscaling/_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from pydantic import Field
from pydantic.dataclasses import dataclass as pydantic_dataclass

from crawlee._utils.context import Collaborator
from crawlee._utils.docs import docs_group


Expand Down Expand Up @@ -104,12 +105,13 @@ class ConcurrencyConsumer:


@docs_group('Autoscaling')
class ConcurrencySystemInterface(Protocol):
class ConcurrencySystemInterface(Collaborator, Protocol):
"""The contract between an `AutoscaledPool` and the component that governs its concurrency.

The system answers whether there is free compute for one more task and tracks the budget that tasks are booked
against. `ConcurrencySystem` is the default implementation. Anything matching this contract can take its place,
without depending on its internals.
without depending on its internals. An implementation is an async context manager, and `AutoscaledPool.run` refuses
to run on one that isn't entered.

Every allocation method is told which `ConcurrencyConsumer` is asking, so an implementation may allocate per
consumer. `ConcurrencySystem` does not, and serves whoever asks first, which can starve a consumer that joins
Expand All @@ -127,10 +129,6 @@ def desired_concurrency(self) -> int:
def current_concurrency(self) -> int:
"""The number of parallel tasks currently booked against this system."""

@property
def active(self) -> bool:
"""Whether the system is ready to be booked against. `AutoscaledPool.run` refuses to run when it is not."""

def has_capacity_for_task(self, consumer: ConcurrencyConsumer) -> bool:
"""Report whether one more task may start right now on behalf of `consumer`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,8 @@ def __init__(

Args:
rendering_type_predictor: Object that implements RenderingTypePredictor and is capable of predicting which
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used.
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used. Keep a
predictor passed in open with `async with` until every crawler using it has finished.
result_checker: Function that evaluates whether crawling result is valid or not.
result_comparator: Function that compares two crawling results and decides whether they are equivalent.
static_parser: Implementation of `AbstractHttpParser`. Parser that will be used for static crawling.
Expand All @@ -144,11 +145,10 @@ def __init__(
save_response_cookies: Whether to save the response cookies to the session, in both the static and the
browser sub crawler.
statistics: A custom `Statistics[AdaptivePlaywrightCrawlerStatisticState]` instance, allowing the use of
non-default configuration.
non-default configuration. The crawler enters it on every run, so don't enter it yourself.
kwargs: Additional keyword arguments to pass to the underlying `BasicCrawler`.
"""
# Adaptive crawling related.
self.rendering_type_predictor = rendering_type_predictor or DefaultRenderingTypePredictor()
self.result_checker = result_checker or (lambda _: True)
self.result_comparator = result_comparator or create_default_comparator(result_checker)

Expand All @@ -164,6 +164,8 @@ def __init__(

super().__init__(statistics=adaptive_statistics, **kwargs)

self._rendering_type_predictor = self._own_or_borrow(rendering_type_predictor, DefaultRenderingTypePredictor)

# Sub crawlers related.
playwright_crawler_specific_kwargs = playwright_crawler_specific_kwargs or _PlaywrightCrawlerAdditionalOptions()

Expand All @@ -180,15 +182,16 @@ def __init__(
# Initialize sub crawlers to create their pipelines.
static_crawler_class = AbstractHttpCrawler.create_parsed_http_crawler_class(static_parser=static_parser)

# The sub crawlers never run themselves, so this crawler enters and exits their statistics.
static_crawler = static_crawler_class(
parser=static_parser,
save_response_cookies=save_response_cookies,
statistics=_NonPersistentStatistics(),
statistics=self._own_or_borrow(None, _NonPersistentStatistics),
**basic_crawler_kwargs_for_static_crawler,
)
playwright_crawler = PlaywrightCrawler(
save_response_cookies=save_response_cookies,
statistics=_NonPersistentStatistics(),
statistics=self._own_or_borrow(None, _NonPersistentStatistics),
**playwright_crawler_specific_kwargs,
**basic_crawler_kwargs_for_pw_crawler,
)
Expand Down Expand Up @@ -229,19 +232,22 @@ async def adaptive_post_navigation_hook_pw(context: PlaywrightPostNavCrawlingCon
static_crawler.post_navigation_hook(adaptive_post_navigation_hook_static)
playwright_crawler.post_navigation_hook(adaptive_post_navigation_hook_pw)

self._additional_context_managers = [
*self._additional_context_managers,
self.rendering_type_predictor,
static_crawler.statistics,
playwright_crawler.statistics,
playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
]
# The default pool is the one the browser sub crawler built.
self._own_or_borrow(
playwright_crawler_specific_kwargs.get('browser_pool'),
lambda: playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
)

# Sub crawler pipeline related
self._pw_context_pipeline = playwright_crawler._context_pipeline # noqa:SLF001 # Intentional access to private member.
self._static_context_pipeline = static_crawler._context_pipeline # noqa:SLF001 # Intentional access to private member.
self._static_parser = static_parser

@property
def rendering_type_predictor(self) -> RenderingTypePredictor:
"""The predictor that decides which sub crawler processes a request."""
return self._rendering_type_predictor

@classmethod
def with_beautifulsoup_static_parser(
cls,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@ def __init__(self) -> None:
# Flag to indicate the state.
self._active = False

@property
def active(self) -> bool:
"""Whether the predictor is initialized and not cleared since."""
return self._active

@abstractmethod
def predict(self, request: Request) -> RenderingTypePrediction:
"""Get `RenderingTypePrediction` based on the input request.
Expand Down Expand Up @@ -164,15 +169,20 @@ async def initialize(self) -> None:
await super().initialize()

if not self._state.is_initialized:
await self._state.initialize()
try:
await self._state.initialize()
except BaseException:
await super().clear()
raise

@override
async def clear(self) -> None:
"""Clear the predictor state."""
await super().clear()

if self._state.is_initialized:
await self._state.teardown()
try:
if self._state.is_initialized:
await self._state.teardown()
finally:
await super().clear()

@override
def predict(self, request: Request) -> RenderingTypePrediction:
Expand Down
Loading
Loading