gh-51540: Let proxy auth handlers answer a 407 to CONNECT for https URLs (#161596)

urllib.request could not authenticate with a proxy for an https URL
using ProxyBasicAuthHandler or ProxyDigestAuthHandler: http.client
raised a plain OSError when the proxy answered the CONNECT request
with 407, so the http_error_407 handlers never ran.

* http.client: add ProxyTunnelError, an OSError subclass carrying the
  code, reason and headers of the proxy's response.  It holds no
  socket and can be pickled.
* urllib.request: do_open() passes a 407 to CONNECT on to the error
  handlers, so the request is retried with Proxy-Authorization on the
  CONNECT.  Other CONNECT failures still raise URLError.
* Request.set_proxy() keeps the tunnel when a request is re-opened, so
  the retried request still sends the path, not the absolute URL.
* ProxyDigestAuthHandler computes the digest over the CONNECT request
  (CONNECT host:port) when tunnelling.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: orsenthil

Doc/library/http.client.rst

@@ -230,6 +230,29 @@ The following exceptions are raised as appropriate:
       Previously, :exc:`BadStatusLine`\ ``('')`` was raised.
 
 
+.. exception:: ProxyTunnelError
+
+   A subclass of :exc:`OSError`.  Raised when the proxy server answers the
+   CONNECT request sent for :meth:`HTTPConnection.set_tunnel` with a status
+   other than 200.
+
+   .. attribute:: code
+
+      The HTTP status code of the proxy's response, such as ``407``.
+
+   .. attribute:: reason
+
+      The reason phrase of the proxy's response.
+
+   .. attribute:: headers
+
+      The headers of the proxy's response, as an
+      :class:`http.client.HTTPMessage` instance.
+
+   .. versionadded:: next
+      Previously, :exc:`OSError` was raised.
+
+
 The constants defined in this module are:
 
 .. data:: HTTP_PORT
@@ -398,6 +421,9 @@ HTTPConnection Objects
       one will be automatically generated and transmitted if not provided in
       the headers argument.
 
+   .. versionchanged:: next
+      :exc:`ProxyTunnelError` is raised if the proxy rejects the CONNECT request.
+
 
 .. method:: HTTPConnection.get_proxy_response_headers()
 

Doc/library/urllib.request.rst

@@ -445,6 +445,10 @@ The following classes are provided:
    :ref:`http-password-mgr` for information on the interface that must be
    supported.
 
+   .. versionchanged:: next
+      Also handles a proxy that requires authentication for the CONNECT
+      request used to tunnel ``https`` URLs.
+
 
 .. class:: AbstractDigestAuthHandler(password_mgr=None)
 
@@ -482,6 +486,10 @@ The following classes are provided:
    :ref:`http-password-mgr` for information on the interface that must be
    supported.
 
+   .. versionchanged:: next
+      Also handles a proxy that requires authentication for the CONNECT
+      request used to tunnel ``https`` URLs.
+
 
 .. class:: HTTPHandler()
 
@@ -1102,6 +1110,14 @@ ProxyBasicAuthHandler Objects
 
    Retry the request with authentication information, if available.
 
+   For an ``https`` URL, the request goes through a tunnel that the proxy
+   sets up in reply to a CONNECT request.  A 407 response can then come
+   either from the proxy, rejecting the CONNECT, or from the origin server,
+   through the tunnel.  Both are handled the same way: the request is retried
+   with the credentials for the proxy, and those credentials are sent only in
+   the CONNECT request, never to the origin server.  If the origin server
+   keeps answering 407, :exc:`~urllib.error.HTTPError` is raised.
+
 
 .. _abstract-digest-auth-handler:
 
@@ -1141,6 +1157,14 @@ ProxyDigestAuthHandler Objects
 
    Retry the request with authentication information, if available.
 
+   For an ``https`` URL, the request goes through a tunnel that the proxy
+   sets up in reply to a CONNECT request.  A 407 response can then come
+   either from the proxy, rejecting the CONNECT, or from the origin server,
+   through the tunnel.  Both are handled the same way: the request is retried
+   with the credentials for the proxy, and those credentials are sent only in
+   the CONNECT request, never to the origin server.  If the origin server
+   keeps answering 407, :exc:`~urllib.error.HTTPError` is raised.
+
 
 .. _http-handler-objects:
 

Doc/whatsnew/3.16.rst

@@ -508,6 +508,16 @@ gzip
   (Contributed by Marin Misur in :gh:`91372`.)
 
 
+http.client
+-----------
+
+* Add the :exc:`~http.client.ProxyTunnelError` exception, a subclass of
+  :exc:`OSError` raised when a proxy rejects the CONNECT request sent for
+  :meth:`~http.client.HTTPConnection.set_tunnel`.  It exposes the status
+  code, reason and headers of the proxy's response.
+  (Contributed by Senthil Kumaran in :gh:`51540`.)
+
+
 imaplib
 -------
 
@@ -956,6 +966,18 @@ unicodedata
 * The Unicode database has been updated to Unicode 18.0.0.
 
 
+urllib.request
+--------------
+
+* :class:`~urllib.request.ProxyBasicAuthHandler` and
+  :class:`~urllib.request.ProxyDigestAuthHandler` now authenticate with a
+  proxy that answers the CONNECT request for an ``https`` URL with
+  ``407 Proxy Authentication Required``.  When no credentials are available,
+  :exc:`~urllib.error.HTTPError` with code 407 is raised instead of
+  :exc:`~urllib.error.URLError`.
+  (Contributed by Senthil Kumaran in :gh:`51540`.)
+
+
 uuid
 ----
 

Lib/http/client.py

@@ -87,8 +87,8 @@
            "UnknownTransferEncoding", "UnimplementedFileMode",
            "IncompleteRead", "InvalidURL", "ImproperConnectionState",
            "CannotSendRequest", "CannotSendHeader", "ResponseNotReady",
-           "BadStatusLine", "LineTooLong", "RemoteDisconnected", "error",
-           "responses"]
+           "BadStatusLine", "LineTooLong", "RemoteDisconnected",
+           "ProxyTunnelError", "error", "responses"]
 
 HTTP_PORT = 80
 HTTPS_PORT = 443
@@ -1045,7 +1045,8 @@ def _tunnel(self):
 
             if code != http.HTTPStatus.OK:
                 self.close()
-                raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
+                raise ProxyTunnelError(code, message.strip(),
+                                       _parse_header_lines(self._raw_proxy_headers))
 
         finally:
             response.close()
@@ -1617,5 +1618,15 @@ def __init__(self, *pos, **kw):
         BadStatusLine.__init__(self, "")
         ConnectionResetError.__init__(self, *pos, **kw)
 
+class ProxyTunnelError(OSError):
+    def __init__(self, code, reason, headers):
+        OSError.__init__(self, f"Tunnel connection failed: {code} {reason}")
+        self.code = code
+        self.reason = reason
+        self.headers = headers
+
+    def __reduce__(self):
+        return self.__class__, (self.code, self.reason, self.headers)
+
 # for backwards compatibility
 error = HTTPException

Lib/test/test_httplib.py

@@ -4,6 +4,7 @@
 import io
 import itertools
 import os
+import pickle
 import array
 import re
 import socket
@@ -2786,6 +2787,36 @@ def test_no_proxy_response_headers(self):
         headers = self.conn.get_proxy_response_headers()
         self.assertIsNone(headers)
 
+    def test_tunnel_error(self):
+        response_text = (
+            'HTTP/1.1 407 Proxy Authentication Required\r\n'
+            'Proxy-Authenticate: Basic realm="proxy"\r\n\r\n'
+        )
+        self.conn._create_connection = self._create_connection(response_text)
+        self.conn.set_tunnel('destination.com')
+
+        with self.assertRaises(client.ProxyTunnelError) as cm:
+            self.conn.request('HEAD', '/', '')
+        exc = cm.exception
+        self.assertIsInstance(exc, OSError)
+        self.assertEqual(exc.code, 407)
+        self.assertEqual(exc.reason, 'Proxy Authentication Required')
+        self.assertEqual(exc.headers['Proxy-Authenticate'],
+                         'Basic realm="proxy"')
+        self.assertEqual(str(exc), 'Tunnel connection failed: '
+                                   '407 Proxy Authentication Required')
+        self.assertIsNone(self.conn.sock)
+
+        for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+            with self.subTest(proto=proto):
+                copy = pickle.loads(pickle.dumps(exc, proto))
+                self.assertIs(type(copy), client.ProxyTunnelError)
+                self.assertEqual(copy.code, 407)
+                self.assertEqual(copy.reason, exc.reason)
+                self.assertEqual(copy.headers['Proxy-Authenticate'],
+                                 'Basic realm="proxy"')
+                self.assertEqual(str(copy), str(exc))
+
     def test_tunnel_leak(self):
         sock = None