Skip to content

fix(browsers): Prevent a hang when closing a page never finishes - #2441

Merged
vdusek merged 3 commits into
apify:masterfrom
PrashanthLande:fix/page-close-timeout
Nov 17, 2026
Merged

vdusek merged 3 commits into
apify:masterfrom
PrashanthLande:fix/page-close-timeout

Conversation

@PrashanthLande

@PrashanthLande PrashanthLande commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • PlaywrightCrawler hangs forever when Chromium never finishes a page.close(). This happens when a page navigates itself with JavaScript (e.g. location.reload()) right as the crawler closes it. crawler.run() never returns, even when wrapped in asyncio.wait_for.
  • Ports fix(browser-pool): survive a page.close() that never settles crawlee#4116 (v3: fix(browser-pool): survive a page.close() that never settles crawlee#4108), with one difference: only the page close itself is time limited, not the close hooks. BrowserPool now gives page.close() 5 seconds. If it does not finish, it logs a warning, releases the page from its browser and retires the browser, so it gets no new pages and is closed once its remaining pages are closed.
  • Pre and post page close hooks run without a time limit, as before, so slow user hooks are not cancelled.
  • The close wrapper is now always installed, not only when close hooks are registered.
  • Adds BrowserController.register_page_closed(), so the page is released even when its close event never arrives. It has a no-op default, so existing custom controllers keep working.
  • Uses a fixed 5 s limit like JS (PAGE_CLOSE_TIMEOUT_MILLIS) rather than operation_timeout, since a healthy close takes milliseconds. Happy to make it configurable if you prefer.
  • Note: when the close times out, the post page close hooks still run, since the page has been released from the pool. Before, a hanging close meant they never ran.
  • Known limitation: if closing a retired browser itself hangs, the inactive-browser cleanup task blocks too. Same as JS; left out of scope.

Issues

  • None. Found in production on a WordPress site using LiteSpeed Cache guest mode, which reloads the page for new visitors. After all requests were handled, the log kept repeating Crawled 3/3 pages, 0 failed requests for over 20 minutes, and a coroutine dump of the stuck process showed the worker task waiting on Page.close inside the crawler's deferred cleanup.

Testing

  • Added tests in tests/integration/browsers/test_browser_pool.py:
    • a close that never finishes releases the page and retires the browser (fails without the fix);
    • pre and post close hooks slower than the time limit run to completion and do not retire the browser.
  • The repro below simulates the hang by patching Playwright's internal Page.close to never finish (the real trigger, a page reloading itself during close, is timing-dependent and hard to reproduce on demand). Before the fix crawler.run() hung forever; now it logs the warning and returns after about 6 s.
  • Lint and type checks pass.
Repro (simulated hang)
import asyncio

from playwright._impl._page import Page as ImplPage

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def never_finishing_close(self, *args, **kwargs):
    # Simulates Chromium never finishing the close, as it does when the page is navigating at that moment.
    await asyncio.Event().wait()


ImplPage.close = never_finishing_close


async def main() -> None:
    crawler = PlaywrightCrawler()

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Handled {context.request.url}')

    await crawler.run(['https://crawlee.dev'])
    print('crawler.run() returned')


asyncio.run(main())

Checklist

  • CI passed

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 13, 2026
@vdusek
vdusek requested a review from Mantisus November 13, 2026 12:42

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, but in its current form it introduces a breaking change. Could you update it so the timeout applies only to the page close?

Comment thread src/crawlee/browsers/_browser_pool.py Resolved

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for update! LGTM.

@vdusek vdusek changed the title fix(browsers): Release a page whose close never finishes fix(browsers): Prevent a hang when closing a page never finishes Nov 16, 2026
f'Closing the page {crawlee_page.id} did not finish within {timeout} seconds, releasing it from '
'the pool anyway.'
)
browser_controller.register_page_closed(crawlee_page.page)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: If the stuck page also blocks context.close(), BrowserPool.__aexit__ hangs on browser.close(force=True), and crawler.run() does not return. The repro patches only Page.close, so it can't show this. Did the production run exit cleanly with this fix?

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw this once in a batch of ~600 sites: after an "Execution context was destroyed" retry, the deferred cleanup's page.close() never returned, so crawler.run() hung after all requests were handled and never reached shutdown. Wrapping run() in an outer asyncio.wait_for didn't help either, since cancelling it still waited on the stuck task. This PR fixes that part.

I haven't been able to confirm your scenario. The hang wasn't reproducible on demand, the run never got as far as BrowserPool.__aexit__, and my repro patches only Page.close. You're right that browser.close(force=True) awaits self._browser_context.close() with no timeout, so if the stuck page also blocks the context close, run() could still hang at exit. I'd prefer to keep this PR to the page-close hang and add a timeout on the shutdown path as a follow-up. I'm happy to open an issue for it, or add it here if you'd prefer.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, thank you, let's keep this PR to the page close hang only then, and let's see

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit 8670f7b into apify:master Nov 17, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants