Repository navigation
fix(browsers): Prevent a hang when closing a page never finishes - #2441
Conversation
Mantisus
left a comment
There was a problem hiding this comment.
Thanks for the PR, but in its current form it introduces a breaking change. Could you update it so the timeout applies only to the page close?
Mantisus
left a comment
There was a problem hiding this comment.
Thanks for update! LGTM.
| f'Closing the page {crawlee_page.id} did not finish within {timeout} seconds, releasing it from ' | ||
| 'the pool anyway.' | ||
| ) | ||
| browser_controller.register_page_closed(crawlee_page.page) |
There was a problem hiding this comment.
Question: If the stuck page also blocks context.close(), BrowserPool.__aexit__ hangs on browser.close(force=True), and crawler.run() does not return. The repro patches only Page.close, so it can't show this. Did the production run exit cleanly with this fix?
✍️ Drafted by Claude Code
There was a problem hiding this comment.
I saw this once in a batch of ~600 sites: after an "Execution context was destroyed" retry, the deferred cleanup's page.close() never returned, so crawler.run() hung after all requests were handled and never reached shutdown. Wrapping run() in an outer asyncio.wait_for didn't help either, since cancelling it still waited on the stuck task. This PR fixes that part.
I haven't been able to confirm your scenario. The hang wasn't reproducible on demand, the run never got as far as BrowserPool.__aexit__, and my repro patches only Page.close. You're right that browser.close(force=True) awaits self._browser_context.close() with no timeout, so if the stuck page also blocks the context close, run() could still hang at exit. I'd prefer to keep this PR to the page-close hang and add a timeout on the shutdown path as a follow-up. I'm happy to open an issue for it, or add it here if you'd prefer.
There was a problem hiding this comment.
ok, thank you, let's keep this PR to the page close hang only then, and let's see
Description
PlaywrightCrawlerhangs forever when Chromium never finishes apage.close(). This happens when a page navigates itself with JavaScript (e.g.location.reload()) right as the crawler closes it.crawler.run()never returns, even when wrapped inasyncio.wait_for.BrowserPoolnow givespage.close()5 seconds. If it does not finish, it logs a warning, releases the page from its browser and retires the browser, so it gets no new pages and is closed once its remaining pages are closed.BrowserController.register_page_closed(), so the page is released even when itscloseevent never arrives. It has a no-op default, so existing custom controllers keep working.PAGE_CLOSE_TIMEOUT_MILLIS) rather thanoperation_timeout, since a healthy close takes milliseconds. Happy to make it configurable if you prefer.Issues
Crawled 3/3 pages, 0 failed requestsfor over 20 minutes, and a coroutine dump of the stuck process showed the worker task waiting onPage.closeinside the crawler's deferred cleanup.Testing
tests/integration/browsers/test_browser_pool.py:Page.closeto never finish (the real trigger, a page reloading itself during close, is timing-dependent and hard to reproduce on demand). Before the fixcrawler.run()hung forever; now it logs the warning and returns after about 6 s.Repro (simulated hang)
Checklist