fix: scope downloaded_files per crawl to prevent race condition (#1889)

[hafezparast]

Summary

• Fixes a race condition where downloaded_files from URL A leak into URL B's CrawlResult when downloads are slow
• Root cause: self._downloaded_files was shared mutable state with fire-and-forget asyncio.create_task download handlers — no synchronization between concurrent crawls
• Fix: replace the shared list with a dict[crawl_id, list] keyed by a unique ID per _crawl_web invocation, passed into _handle_download via closure

Fixes #1889

The Race (before)

URL A navigation → browser triggers download
URL A _crawl_web returns → downloaded_files = [] (download still in progress)
URL B _crawl_web starts → self._downloaded_files = [] (reset)
URL A's download completes → appends to self._downloaded_files
URL B _crawl_web returns → downloaded_files = [URL_A's_file] ← WRONG

After Fix

URL A gets crawl_id="abc" → downloads scoped to _downloads_by_crawl["abc"]
URL B gets crawl_id="xyz" → downloads scoped to _downloads_by_crawl["xyz"]
Late download for "abc" → goes to "abc"'s list (or logged if already popped)
No cross-contamination possible

Changes

• crawl4ai/async_crawler_strategy.py: Replace self._downloaded_files (list) with self._downloads_by_crawl (dict keyed by crawl_id). _handle_download now takes a crawl_id parameter. Late downloads that complete after their crawl finished are logged with a warning instead of silently contaminating the next crawl.
• tests/test_download_race_condition_1889.py: 9 tests covering isolation, late downloads, concurrent race simulation

Test plan

• 9 unit tests passing (pytest tests/test_download_race_condition_1889.py -v)
• Two crawls with isolated downloads verified
• Late download after crawl finishes — logged, not attached to wrong result
• Concurrent race simulation — slow download on A, fast B, no cross-contamination
• Integration test with real browser downloads (needs Playwright)

🤖 Generated with Claude Code

[discopops]

Approved. The regression harness change is minimal, the lockfile churn was cleaned, and the race-condition fix validates with pytest -q tests/test_download_race_condition_1889.py.