You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Block requests to private network addresses by default #2481
Crawlee sends requests to loopback, private, and link-local addresses, and it follows redirects to them. _check_url_after_redirects checks only the final URL, after the whole redirect chain was requested. The Security of web scraping guide leaves the filtering to the user, but user code can't see redirect hops, sitemap and robots.txt requests, or the address a hostname resolves to.
Proposal for v2: block requests to non-public addresses by default, including every redirect hop, with an opt-out option such as allow_private_networks=True for local servers in tests and development.
Block 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 0.0.0.0/8, 100.64.0.0/10, ::1, fe80::/10, fc00::/7, and IPv4-mapped addresses. ipaddress.ip_address(...).is_global covers most of them.
Open question: how to check redirect hops and resolved addresses in each HTTP client and in Playwright.
changed the title [-]Add an option to block requests to private network addresses[/-][+]Block requests to private network addresses by default[/+]on Nov 18, 2026
Activity
[-]Add an option to block requests to private network addresses[/-][+]Block requests to private network addresses by default[/+]