Skip to content

www.php.net blocks "fetch libfetch/2.0" user-agent #483

Closed as not planned
Closed as not planned

Description

@CyberCr33p

Description

This is what I get when I try to upgrade PHP from 7.4.26 to 7.4.27 using FreeBSD ports:

=> php-7.4.27.tar.xz doesn't seem to exist in /usr/ports/distfiles//.
=> Attempting to fetch https://www.php.net/distributions/php-7.4.27.tar.xz
fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz
fetch: http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles// and try again.


FreeBSD ports uses "fetch" to download https://www.php.net/distributions/php-7.4.27.tar.xz but looks like www.php.net web-server blocks user agent "fetch libfetch/2.0" and returns "Service Unavailable".

Can you whitelist this user agent?

PHP Version

PHP 7.4

Operating System

FreeBSD

Activity

  1. transferred this issue fromphp/php-srcon Dec 26, 2021
  2. cmb69 commented on Dec 26, 2021

    @cmb69
    Member

    curl -A "fetch libfetch/2.0" --output php-7.4.27.tar.xz https://www.php.net/distributions/php-7.4.27.tar.xz works for me. What am I missing?

  3. cmb69 commented on Dec 26, 2021

    @cmb69
    Member

    Is this possibly certificate related?

  4. dsh2dsh commented on Dec 26, 2021

    @dsh2dsh

    I can confirm:

    ❯ fetch https://www.php.net/distributions/php-7.4.27.tar.xz
    fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable
    
  5. olegsidorkin commented on Dec 26, 2021

    @olegsidorkin

    wget https://www.php.net/distributions/php-7.4.27.tar.xz - does work
    fetch -vvvv https://www.php.net/distributions/php-7.4.27.tar.xz - doesn't. It successfully establishes connection and then fails:

    Security CheckOur systems have detected unusual traffic from your computer network. This page checks to see if it is really you sending the requests and not a bot.Please click here to verify that you are not a bot. Browser Working Myra Working Host WorkingWhat happened?You ran into a security check to verify the validity of your request.What can I do?If you are a visitor of this website:You must confirm that you are human.If you are the owner of this website:Please check your security settings.

    Headers are:
    `resolving server address: www.php.net:443
    SSL options: 82004854
    Peer verification enabled
    Using CA cert file: /usr/local/etc/ssl/cert.pem
    Verify hostname
    TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
    Certificate subject: /CN=*.php.net
    Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
    requesting https://www.php.net/distributions/php-7.4.27.tar.xz

    GET /distributions/php-7.4.27.tar.xz HTTP/1.1
    Host: www.php.net
    Accept: /
    User-Agent: fetch libfetch/2.0
    Connection: close

    <<< HTTP/1.1 503 Service Temporarily Unavailable
    <<< Server: myracloud
    <<< Date: Sun, 26 Dec 2021 16:05:06 GMT
    <<< Content-Type: text/html
    <<< Transfer-Encoding: chunked
    <<< Connection: close
    <<< cache-control: no-cache, no-store, max-age=0
    <<< X-Frame-Options: SAMEORIGIN
    <<< X-Content-Type-Options: nosniff
    <<< X-XSS-Protection: 1; mode=block`

  6. cmb69 commented on Dec 26, 2021

    @cmb69
    Member

    Ah, so this would be a systems issue. Maybe @saschaschumann can have a look?

  7. 5u623l20 commented on Dec 26, 2021

    @5u623l20

    I have tried changing user agent of fetch too with no results:
    fetch -vvvvv --user-agent="curl/7.80.0" https://www.php.net/distributions/php-8.1.1.tar.xz
    scheme: "https"
    user: ""
    password: ""
    host: "www.php.net"
    port: "0"
    document: "/distributions/php-8.1.1.tar.xz"
    ---> www.php.net:443
    resolving server address: www.php.net:443
    SSL options: 82004854
    Peer verification enabled
    Using CA cert file: /usr/local/etc/ssl/cert.pem
    Verify hostname
    TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
    Certificate subject: /CN=*.php.net
    Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
    requesting https://www.php.net/distributions/php-8.1.1.tar.xz

    GET /distributions/php-8.1.1.tar.xz HTTP/1.1
    Host: www.php.net
    Accept: /
    User-Agent: curl/7.80.0
    Connection: close

    <<< HTTP/1.1 503 Service Temporarily Unavailable
    <<< Server: myracloud
    <<< Date: Sun, 26 Dec 2021 16:30:08 GMT
    <<< Content-Type: text/html
    <<< Transfer-Encoding: chunked
    <<< Connection: close
    <<< cache-control: no-cache, no-store, max-age=0
    <<< X-Frame-Options: SAMEORIGIN
    <<< X-Content-Type-Options: nosniff
    <<< X-XSS-Protection: 1; mode=block
    <<<
    offset 0, length -1, size -1, clength -1
    <<< 7738

  8. added a commit that references this issue on Dec 26, 2021
  9. 5u623l20 commented on Dec 26, 2021

    @5u623l20

    In case someone is already investigating I believe the problem is with http1.1 vs http2. In case we try curl with --http1.1 it also fails with the same errors.

  10. added a commit that references this issue on Jan 4, 2022
  11. amezin commented on Jan 17, 2025

    @amezin

    It's not only libfetch

    Sometimes, node.js also gets blocked nodejs/help#4516 - seemingly by TLS fingerprints. It gets blocked, then, after a few days, unblocked, then some day blocked again.

    If blocking by TLS fingerprint, switching to a different HTTP version likely won't help (although I can't recall if I tried it last time).

    Is it really not possible to tweak the configuration not to block PHP downloads (both tarballs and the https://www.php.net/downloads.php)?

    What's the point of having things like https://www.php.net/releases/index.php?json if automatic fetches get blocked anyway

  12. derickr commented on Aug 18, 2025

    @derickr
    Member

    Our CDN sometimes realises big spikes from networks, and then puts this up. I can turn this off, but only for all urls, not specific ones. And I don't really want to do that, so I'm going to have to close this ticket.

  13. damiankeska-intellistack commented on Mar 3, 2026

    @damiankeska-intellistack

    AWS is offering a PHP environment in CodeBuild for years but... today we got blocked.

    [Container] 2026/03/03 09:02:12.510741 Running command phpenv install 8.4.6 && rm -rf /tmp/*
    [Info]: Loaded extension plugin
    [Info]: Loaded apc Plugin.
    [Info]: Loaded composer Plugin.
    [Info]: Loaded github Plugin.
    [Info]: Loaded uprofiler Plugin.
    [Info]: Loaded xdebug Plugin.
    [Info]: Loaded xhprof Plugin.
    [Info]: Loaded zendopcache Plugin.
    [Info]: php.ini-production gets used as php.ini
    [Info]: Building 8.4.6 into /root/.phpenv/versions/8.4.6
    [Downloading]: https://www.php.net/distributions/php-8.4.6.tar.bz2
    
    -----------------
    |  BUILD ERROR  |
    -----------------
    
    Here are the last 10 lines from the log:
    
    -----------------------------------------
    curl: (22) The requested URL returned error: 503
    -----------------------------------------
    
    The full Log is available at '/tmp/php-build.8.4.6.20260303090213.log'.
    [Warn]: Aborting build.
    
    [Container] 2026/03/03 09:02:13.773156 Command did not exit successfully phpenv install 8.4.6 && rm -rf /tmp/* exit status 2
    [Container] 2026/03/03 09:02:13.773773 Phase complete: DOWNLOAD_SOURCE State: FAILED
    [Container] 2026/03/03 09:02:13.773795 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while installing runtimes.
    Error: Build status: FAILED
    *****CODEBUILD COMPLETE*****
    
    <!DOCTYPE html><html><head><title>Security Check</title><meta charset="UTF-8"><meta name="viewport" content="width=device-width">
    

    So, any CI builds are stuck at this point.
    I'm not sure if tarballs shouldn't be behind a CDN instead of something blocking robotic access? I think there are many ways to secure even tarball downloads from excessive usage.

  14. added a commit that references this issue on Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions