Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/03_concepts/12_storage_backends.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,10 @@ Data persistence in Redis depends on your [database configuration](https://redis

The backend requires either a Redis connection string or a pre-configured Redis client instance. Use a pre-configured client when you need custom Redis settings such as connection pooling, timeouts, or SSL/TLS encryption.

Use the backend as an async context manager, so that its connections are closed when you're done. It closes a client it created from the connection string, but not one you pass in. Storages opened through a backend stop working once it's closed.

Each backend opens its own storages. With `purge_on_start` enabled, it purges the unnamed storages, such as the default dataset, when it opens them. As a result, a new backend connected to the same database wipes the data an earlier one wrote to these storages.

<CodeBlock className="language-python" language="python">
{RedisStorageBackendBasicExample}
</CodeBlock>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
from crawlee.crawlers import ParselCrawler
from crawlee.storage_backends import RedisStorageBackend

# Create a new instance of storage backend using connection string.
# 'redis://localhost:6379' is the just placeholder, replace it with your actual
# connection string.
storage_backend = RedisStorageBackend(connection_string='redis://localhost:6379')

# And pass it to the crawler.
crawler = ParselCrawler(storage_backend=storage_backend)
async def main() -> None:
# Create a new instance of the storage backend using a connection string.
# 'redis://localhost:6379' is just a placeholder. Replace it with your actual
# connection string.
# Use the context manager to ensure that connections are properly cleaned up.
async with RedisStorageBackend(
connection_string='redis://localhost:6379'
) as storage_backend:
# And pass it to the crawler.
crawler = ParselCrawler(storage_backend=storage_backend)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The async with block ends right after the crawler is created, so the backend is closed before anything runs. A user who adds await crawler.run(...) after the block gets "The RedisStorageBackend is closed". Running the crawler inside the block (as the upgrading guide does) would show the right pattern. Same in redis_storage_backend_configuration_example.py.

✍️ Drafted by Claude Code


# Run the crawler inside the block, while the storage backend is open.
await crawler.run(['https://crawlee.dev'])
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,29 @@
from crawlee.crawlers import ParselCrawler
from crawlee.storage_backends import RedisStorageBackend

# Create a new instance of storage backend using a Redis client with custom settings.
# Replace host and port with your actual Redis server configuration.
# Other Redis client settings can be adjusted as needed.
storage_backend = RedisStorageBackend(
redis=Redis(

async def main() -> None:
# Create a Redis client with custom settings.
# Replace host and port with your actual Redis server configuration.
# Other Redis client settings can be adjusted as needed.
redis = Redis(
host='localhost',
port=6379,
retry_on_timeout=True,
socket_keepalive=True,
socket_connect_timeout=10,
)
)

# Create a configuration with custom settings.
configuration = Configuration(purge_on_start=False)
# The storage backend doesn't close a client you pass in, so close it yourself.
async with redis, RedisStorageBackend(redis=redis) as storage_backend:
# Create a configuration with custom settings.
configuration = Configuration(purge_on_start=False)

# And pass them to the crawler.
crawler = ParselCrawler(
storage_backend=storage_backend,
configuration=configuration,
)

# And pass them to the crawler.
crawler = ParselCrawler(
storage_backend=storage_backend,
configuration=configuration,
)
# Run the crawler inside the block, while the storage backend is open.
await crawler.run(['https://crawlee.dev'])
28 changes: 28 additions & 0 deletions docs/06_upgrading/upgrading_to_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,34 @@ The memory backend now handles values the same way as the file-system backend. I

If processing a request can take longer than 3 minutes, call `RequestQueue.set_expected_request_processing_time()` with the longest time you expect. Every request fetched after that call stays locked at least that long. To give a single request that's still being processed more time, call `RequestQueue.extend_request_processing_time()`.

## Each `RedisStorageBackend` opens its own storages

In v1, all `RedisStorageClient` instances in a process shared the storages opened through any of them, even when they connected to different Redis servers. Now each `RedisStorageBackend` instance opens its own storages. With `purge_on_start` enabled, a new instance purges the unnamed storages, such as the default dataset, when it opens them. When it connects to the same database as an earlier instance, the data the earlier one wrote to these storages is lost. To read what a crawler stored in them, open them through the crawler's backend.

Use `RedisStorageBackend` as an async context manager. On exit, it closes the Redis client it created from the connection string. In v1, nothing closed that client. A client you pass in stays open, so close it yourself. Storages opened through a closed backend stop working.

For example:

```python
# v1
storage_client = RedisStorageClient(connection_string='redis://localhost:6379')
crawler = ParselCrawler(storage_client=storage_client)
await crawler.run(['https://crawlee.dev'])

dataset = await Dataset.open(
storage_client=RedisStorageClient(connection_string='redis://localhost:6379')
)

# v2
async with RedisStorageBackend(
connection_string='redis://localhost:6379'
) as storage_backend:
crawler = ParselCrawler(storage_backend=storage_backend)
await crawler.run(['https://crawlee.dev'])

dataset = await Dataset.open(storage_backend=storage_backend)
```

## Crawler `id` is a string

The `id` argument of `BasicCrawler` and its subclasses is now a `str`, so pass a string wherever you passed an integer. `StatisticsState.stats_id` and `EventCrawlerStatusData.crawler_id` are strings too, and `crawler_id` holds the crawler's `id`. In v1, it held the Python object ID of the crawler.
Expand Down
50 changes: 35 additions & 15 deletions src/crawlee/storage_backends/_redis/_backend_mixin.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

from crawlee.storage_backends.models import DatasetMetadata, KeyValueStoreMetadata, RequestQueueMetadata

from ._storage_backend import RedisStorageBackend


logger = getLogger(__name__)

Expand All @@ -46,17 +48,17 @@ class RedisBackendMixin:
_STORAGE_TYPE: ClassVar[str]
"""Human-readable storage type for error messages."""

def __init__(self, storage_name: str, storage_id: str, redis: Redis) -> None:
def __init__(self, storage_name: str, storage_id: str, storage_backend: RedisStorageBackend) -> None:
self._storage_name = storage_name
self._storage_id = storage_id
self._redis = redis
self._storage_backend = storage_backend

self._scripts_loaded = False

@property
def redis(self) -> Redis:
"""Return the Redis client instance."""
return self._redis
return self._storage_backend.redis

@property
def metadata_key(self) -> str:
Expand Down Expand Up @@ -107,7 +109,7 @@ async def _open(
name: str | None,
alias: str | None,
metadata_model: type[DatasetMetadata | KeyValueStoreMetadata | RequestQueueMetadata],
redis: Redis,
storage_backend: RedisStorageBackend,
extra_metadata_fields: dict[str, Any],
instance_kwargs: dict[str, Any],
) -> Self:
Expand All @@ -117,7 +119,7 @@ async def _open(
id: The ID of the storage. If not provided, a random ID will be generated.
name: The name of the storage for named (global scope) storages.
alias: The alias of the storage for unnamed (run scope) storages.
redis: Redis client instance.
storage_backend: The storage backend providing the Redis client.
metadata_model: Pydantic model for metadata validation.
extra_metadata_fields: Storage-specific metadata fields.
instance_kwargs: Additional arguments for the backend constructor.
Expand All @@ -129,17 +131,19 @@ async def _open(
storage_id: str | None = None
# Determine if storage exists by ID or name
if id:
storage_name = await cls._get_metadata_name_by_id(id=id, redis=redis)
storage_name = await cls._get_metadata_name_by_id(id=id, redis=storage_backend.redis)
storage_id = id
if storage_name is None:
raise ValueError(f'{cls._STORAGE_TYPE} with ID "{id}" does not exist.')
else:
metadata_data = await cls._get_metadata_by_name(name=internal_name, redis=redis)
metadata_data = await cls._get_metadata_by_name(name=internal_name, redis=storage_backend.redis)
storage_name = internal_name if metadata_data is not None else None
storage_id = metadata_data['id'] if metadata_data is not None else None
# If both storage_name and storage_id are found, open existing storage
if storage_name and storage_id:
backend = cls(storage_name=storage_name, storage_id=storage_id, redis=redis, **instance_kwargs)
backend = cls(
storage_name=storage_name, storage_id=storage_id, storage_backend=storage_backend, **instance_kwargs
)
async with backend._get_pipeline() as pipe:
await backend._update_metadata(pipe, update_accessed_at=True)
# Otherwise, create a new storage
Expand All @@ -153,12 +157,28 @@ async def _open(
modified_at=now,
**extra_metadata_fields,
)
backend = cls(storage_name=internal_name, storage_id=metadata.id, redis=redis, **instance_kwargs)
backend = cls(
storage_name=internal_name, storage_id=metadata.id, storage_backend=storage_backend, **instance_kwargs
)
created = await backend._create_metadata_and_storage(internal_name, metadata.model_dump())
# The storage was probably not created due to a race condition. Let's try to open it using the name.
if not created:
metadata_data = await cls._get_metadata_by_name(name=internal_name, redis=redis, with_wait=True)
backend = cls(storage_name=internal_name, storage_id=metadata.id, redis=redis, **instance_kwargs)
metadata_data = await cls._get_metadata_by_name(
name=internal_name, redis=storage_backend.redis, with_wait=True
)
if metadata_data is None:
raise RuntimeError(
f'{cls._STORAGE_TYPE} "{internal_name}" is being created by another client, but its metadata '
'did not appear in time.'
)

# The storage that won the race has its own ID, which the backend must use to drop it.
backend = cls(
storage_name=internal_name,
storage_id=metadata_data['id'],
storage_backend=storage_backend,
**instance_kwargs,
)

# Ensure Lua scripts are loaded
await backend._ensure_scripts_loaded()
Expand All @@ -177,7 +197,7 @@ async def _ensure_scripts_loaded(self) -> None:
@asynccontextmanager
async def _get_pipeline(self, *, with_execute: bool = True) -> AsyncGenerator[Pipeline]:
"""Create a new Redis pipeline."""
async with self._redis.pipeline() as pipe:
async with self.redis.pipeline() as pipe:
try:
pipe.multi()
yield pipe
Expand All @@ -192,7 +212,7 @@ async def _create_script(self, script_name: str) -> AsyncScript:
"""Load a Lua script from a file and return a Script object."""
script_content = await asyncio.to_thread(read_lua_script, script_name)

return self._redis.register_script(script_content)
return self.redis.register_script(script_content)

async def _create_metadata_and_storage(self, storage_name: str, metadata: dict) -> bool:
index_id_to_name = f'{self._MAIN_KEY}:id_to_name'
Expand All @@ -202,7 +222,7 @@ async def _create_metadata_and_storage(self, storage_name: str, metadata: dict)
metadata['modified_at'] = metadata['modified_at'].isoformat()

# Try to create name_to_id index entry, if it already exists, return False.
name_to_id = await await_redis_response(self._redis.hsetnx(index_name_to_id, storage_name, metadata['id']))
name_to_id = await await_redis_response(self.redis.hsetnx(index_name_to_id, storage_name, metadata['id']))
# If name already exists, return False. Probably an attempt at parallel creation.
if not name_to_id:
return False
Expand Down Expand Up @@ -244,7 +264,7 @@ async def _get_metadata(
self, metadata_model: type[DatasetMetadata | KeyValueStoreMetadata | RequestQueueMetadata]
) -> DatasetMetadata | KeyValueStoreMetadata | RequestQueueMetadata:
"""Retrieve the storage metadata."""
metadata_dict = await self._get_metadata_by_name(name=self._storage_name, redis=self._redis)
metadata_dict = await self._get_metadata_by_name(name=self._storage_name, redis=self.redis)
if metadata_dict is None:
raise ValueError(f'{self._STORAGE_TYPE} with name "{self._storage_name}" does not exist.')
async with self._get_pipeline() as pipe:
Expand Down
19 changes: 10 additions & 9 deletions src/crawlee/storage_backends/_redis/_dataset_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,13 @@
if TYPE_CHECKING:
from collections.abc import AsyncIterator, Mapping

from redis.asyncio import Redis
from redis.asyncio.client import Pipeline
from redis.commands.json._util import JsonType

from crawlee._types import JsonSerializable

from ._storage_backend import RedisStorageBackend

logger = getLogger(__name__)


Expand Down Expand Up @@ -57,17 +58,17 @@ class RedisDatasetBackend(DatasetBackend, RedisBackendMixin):
_STORAGE_TYPE = 'Dataset'
"""Human-readable storage type for error messages."""

def __init__(self, storage_name: str, storage_id: str, redis: Redis) -> None:
def __init__(self, storage_name: str, storage_id: str, storage_backend: RedisStorageBackend) -> None:
"""Initialize a new instance.

Preferably use the `RedisDatasetBackend.open` class method to create a new instance.

Args:
storage_name: Internal storage name used for Redis keys.
storage_id: Unique identifier for the dataset.
redis: Redis client instance.
storage_backend: The storage backend providing the Redis client.
"""
super().__init__(storage_name=storage_name, storage_id=storage_id, redis=redis)
super().__init__(storage_name=storage_name, storage_id=storage_id, storage_backend=storage_backend)

@property
def _items_key(self) -> str:
Expand All @@ -81,7 +82,7 @@ async def open(
id: str | None,
name: str | None,
alias: str | None,
redis: Redis,
storage_backend: RedisStorageBackend,
) -> RedisDatasetBackend:
"""Open or create a new Redis dataset backend.

Expand All @@ -93,7 +94,7 @@ async def open(
id: The ID of the dataset. If not provided, a random ID will be generated.
name: The name of the dataset for named (global scope) storages.
alias: The alias of the dataset for unnamed (run scope) storages.
redis: Redis client instance.
storage_backend: The storage backend providing the Redis client.

Returns:
An instance for the opened or created storage backend.
Expand All @@ -102,7 +103,7 @@ async def open(
id=id,
name=name,
alias=alias,
redis=redis,
storage_backend=storage_backend,
metadata_model=DatasetMetadata,
extra_metadata_fields={'item_count': 0},
instance_kwargs={},
Expand Down Expand Up @@ -207,7 +208,7 @@ async def get_data(
if json_path == '$':
json_path = '$[*]'

data = await await_redis_response(self._redis.json().get(self._items_key, json_path))
data = await await_redis_response(self.redis.json().get(self._items_key, json_path))

if not isinstance(data, list):
data = []
Expand Down Expand Up @@ -295,7 +296,7 @@ async def iterate_items(
json_path = f'$[{batch_start}:{batch_end}]'

# Get batch of items
batch_items = await await_redis_response(self._redis.json().get(self._items_key, json_path))
batch_items = await await_redis_response(self.redis.json().get(self._items_key, json_path))

# Handle case where batch_items might be None or not a list
if batch_items is None or not isinstance(batch_items, list):
Expand Down
Loading
Loading