Skip to content

fix: don't fail the request when a page has an unparsable link URL - #2444

Merged
vdusek merged 3 commits into
apify:masterfrom
PrashanthLande:fix/unparsable-link-urls
Nov 17, 2026
Merged

vdusek merged 3 commits into
apify:masterfrom
PrashanthLande:fix/unparsable-link-urls

Conversation

@PrashanthLande

@PrashanthLande PrashanthLande commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • enqueue_links() and extract_links() failed the whole request when any link on the page had a URL that yarl cannot parse, e.g. a phone number written as http://tel:+91... (ValueError: Port out of range 0-65535), or a host that cannot be decoded, e.g. http://xn--a.com/ (UnicodeError when the host is read). Such links appear on real sites, since browsers tolerate them.
  • These links are now skipped with a debug message that includes the parse error. This affects all crawlers that extract links, and matches crawlee JS, which drops links that new URL() rejects (tryAbsoluteURL).

Issues

  • N/A

Testing

  • Added an unparsable link to the shared start_enqueue test page, like the invalid-URL link crawlee JS keeps in its enqueueLinks test fixture. Without the fix, the enqueue_links and extract_links tests of all crawlers fail with ValueError.
  • test_basic of BeautifulSoupCrawler and ParselCrawler counts the <a> tags on that page, so its expected count goes from 3 to 4.
  • Added a parametrized unit test for to_absolute_url_iterator covering absolute, relative, mailto:, out-of-range port, unclosed IPv6 bracket and invalid IDNA host URLs.

Checklist

  • CI passed

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 13, 2026
@vdusek
vdusek requested a review from Mantisus November 13, 2026 12:42

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for PR! LGTM

Comment thread src/crawlee/_utils/urls.py Resolved
Comment thread src/crawlee/_utils/urls.py Resolved
Comment thread tests/integration/server_endpoints.py Resolved
Comment thread src/crawlee/_utils/urls.py Resolved
@vdusek vdusek changed the title fix: skip links that cannot be parsed when extracting links fix: don't fail the request when a page has an unparsable link URL Nov 16, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit 45f219e into apify:master Nov 17, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants