Skip to content

Block requests to private network addresses by default #2481

Open

Description

@vdusek

Crawlee sends requests to loopback, private, and link-local addresses, and it follows redirects to them. _check_url_after_redirects checks only the final URL, after the whole redirect chain was requested. The Security of web scraping guide leaves the filtering to the user, but user code can't see redirect hops, sitemap and robots.txt requests, or the address a hostname resolves to.

Proposal for v2: block requests to non-public addresses by default, including every redirect hop, with an opt-out option such as allow_private_networks=True for local servers in tests and development.

  • Block 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 0.0.0.0/8, 100.64.0.0/10, ::1, fe80::/10, fc00::/7, and IPv4-mapped addresses. ipaddress.ip_address(...).is_global covers most of them.
  • Open question: how to check redirect hops and resolved addresses in each HTTP client and in Playwright.

Crawlee JS: apify/crawlee#4426.

✍️ Drafted by Claude Code

Activity

  1. added this to the 2.0 milestone on Nov 18, 2026
  2. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 18, 2026
  3. changed the title [-]Add an option to block requests to private network addresses[/-] [+]Block requests to private network addresses by default[/+] on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions