Catapult291 · Nov 12, 2026
diff --git a/README.md b/README.md
@@ -75,6 +75,7 @@ commandcode/
 | `HOST` | `0.0.0.0` | Listen address → `host` |
 | `CC_API_BASE` | `https://api.commandcode.ai` | Upstream base URL → `apiBase` |
 | `CC_UPSTREAM_PROXY` | *(unset)* | Route requests **to the CC upstream** through an HTTP proxy (`http://` CONNECT only); see "Upstream proxy" below → `upstreamProxy` |
+| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | Delay before the single tunnel retry when the tunnel fails at the transport level; a second failure falls back to a direct call for that request; see [Proxy tunnel failure → direct fallback](#proxy-tunnel-failure--direct-fallback) |
 | `PROJECT_SLUG` | `cc-proxy` | `x-project-slug` → `projectSlug` |
 | `LOG_FILE` | empty | Log file → `logFile` (**synchronous writes**, see [Other notes](#other-notes)) |
 | `CC_USE_PROVIDER_MODELS` | `true` | Fetch the model list dynamically → `useProviderModels` |
@@ -178,6 +179,9 @@ CC_UPSTREAM_PROXY=http://127.0.0.1:7890 npm start
 - Each upstream request opens its own tunnel connection. TLS is end-to-end: the certificate is validated against the **target hostname**, never against the proxy.
 - Routing the fingerprint/lifecycle pre-requests through the same proxy matters: if they went out direct while generation went through the proxy, one account would register from two different IPs — exactly the inconsistency you are trying to avoid.
 - Credentials in the proxy URL (`http://user:pass@host:port`) are never logged: only `host:port` shows up.
+- If the tunnel itself is unavailable (proxy down, flapping line), a transport-level failure retries the tunnel
+  once after a short delay, then falls back to a direct connection for that request — see
+  [Proxy tunnel failure → direct fallback](#proxy-tunnel-failure--direct-fallback).
 
 > Node's built-in `fetch` does **not** read `HTTPS_PROXY`/`HTTP_PROXY`. The official env-var route requires Node ≥ 22.21 / 24.5 plus `NODE_USE_ENV_PROXY=1`; this option works without either.
 
@@ -691,6 +695,37 @@ CC_UPSTREAM_RETRY_BASE_MS=800 npm start  # wider backoff (default 400ms)
 > `/v1/messages` and `/v1/responses` have a different structure and are not covered by this change (drops are still
 > reported as before).
 
+## Proxy tunnel failure → direct fallback
+
+`CC_UPSTREAM_PROXY` routes upstream traffic through a CONNECT tunnel. The retry above cannot help when the
+**tunnel itself** is unavailable — it retries along the same route, so a dead proxy fails every attempt.
+
+For a **transport-level** tunnel failure (`ECONNRESET` / `ECONNREFUSED` / `ETIMEDOUT` / `socket hang up` /
+`terminated` … — proxy process restarting, flapping line: TLS handshake cut off, RST right after the first
+packet) the proxy therefore retries the tunnel once after a short delay, and if that fails too, **falls back
+to a direct connection for that request**: a dead proxy degrades to "slower but working" instead of a wall
+of 502s.
+
+- Non-transport failures — a CONNECT answered with `403`/`407` (config or auth problem) — are surfaced as-is:
+  no retry, no silent rerouting.
+- A client abort cancels the request: no retry, no direct fallback, no extra upstream call.
+- With no proxy configured, `upstreamFetch` is a plain `fetch` — default behaviour is unchanged.
+
+| Env var | Default | Description |
+|---|---|---|
+| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | Delay before the single tunnel retry |
+
+Logs: `Upstream proxy attempt failed, retrying` (first tunnel attempt failed) and
+`Upstream proxy retry failed, falling back to direct` (this request continues over a direct connection).
+The startup banner's `upstreamProxyFallback` field shows the effective setting.
+
+```bash
+CC_UPSTREAM_PROXY_RETRY_DELAY_MS=0 npm start    # retry the tunnel immediately before falling back
+```
+
+> **Note**: while the proxy stays down, every tunneled request pays one failed tunnel attempt plus the retry
+> delay before the direct call — this is a per-request fallback, not a circuit breaker.
+
 ## Memory & Deployment
 
 > Measurements reproduced from [issue #20](https://github.com/MAXeaglet/commandcode-proxy/issues/20) (Node v24, loopback mock upstream).

diff --git a/README_zh.md b/README_zh.md
@@ -75,6 +75,7 @@ commandcode/
 | `HOST` | `0.0.0.0` | 监听地址 → `host` |
 | `CC_API_BASE` | `https://api.commandcode.ai` | 上游地址 → `apiBase` |
 | `CC_UPSTREAM_PROXY` | 空 | 让**发往 CC 上游**的请求走 HTTP 代理（仅 `http://` CONNECT），见下文「上游代理」→ `upstreamProxy` |
+| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | 隧道传输层失败时单次重试的延迟，仍失败则该条请求回退直连；见[隧道故障 → 直连回退](#隧道故障--直连回退) |
 | `PROJECT_SLUG` | `cc-proxy` | `x-project-slug` → `projectSlug` |
 | `LOG_FILE` | 空 | 日志文件 → `logFile`（**同步写**，见[其它注意事项](#其它注意事项)）|
 | `CC_USE_PROVIDER_MODELS` | `true` | 动态拉取模型列表 → `useProviderModels` |
@@ -173,6 +174,8 @@ CC_UPSTREAM_PROXY=http://127.0.0.1:7890 npm start
 - 每个上游请求各自建立一条隧道连接。TLS 为端到端：证书按**目标主机名**校验，绝不针对代理降级。
 - **指纹/lifecycle 预请求也走代理**是刻意的：若它们直连而上游生成走代理，同一账号会从两个不同 IP 注册 —— 正是你想避免的那种矛盾。
 - 代理地址里带账号密码（`http://user:pass@host:port`）时，日志只保留 `host:port`，**不打印口令**。
+- 隧道本身不可用（代理进程重启、线路抖动）时，传输层失败会退避后重试一次隧道，仍失败则该条请求
+  改走直连 —— 见[隧道故障 → 直连回退](#隧道故障--直连回退)。
 
 > Node 原生 `fetch` **不读** `HTTPS_PROXY`/`HTTP_PROXY`。官方环境变量路线需要 Node ≥ 22.21 / 24.5 且设 `NODE_USE_ENV_PROXY=1`；本选项两者都不需要。
 
@@ -687,6 +690,34 @@ CC_UPSTREAM_RETRY_BASE_MS=800 npm start  # 退避拉长（默认 400ms）
 > **覆盖范围**：目前 `/v1/chat/completions` 的流式与非流式两条路径都接了重试循环；
 > `/v1/messages` 与 `/v1/responses` 结构不同，未在本次改动中覆盖（闪断仍按原样报错）。
 
+## 隧道故障 → 直连回退
+
+`CC_UPSTREAM_PROXY` 把发往上游的流量装进 CONNECT 隧道。上面的重试救不了**隧道本身**不可用的情况 ——
+它沿原路重试，代理挂了就每次都失败。
+
+因此对**传输层**隧道失败（`ECONNRESET` / `ECONNREFUSED` / `ETIMEDOUT` / `socket hang up` / `terminated`
+等 —— 对应代理进程重启、线路抖动：TLS 握手被掐断、首包后 RST），代理会退避后重试一次隧道，
+仍失败则**该条请求改走直连**：代理不可用时退化为「慢但可用」，而不是把 502 墙抛给客户端。
+
+- 非传输层失败 —— CONNECT 被回 `403`/`407`（配置或认证问题）—— 原样上抛：不重试、不悄悄换路。
+- 客户端主动取消（signal 已 abort）即终止：不重试、不回退直连、不多打一次上游。
+- 未配代理时 `upstreamFetch` 就是原生 `fetch` —— 默认路径行为不变。
+
+| 环境变量 | 默认 | 说明 |
+|---|---|---|
+| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | 单次隧道重试的延迟（毫秒） |
+
+日志：`Upstream proxy attempt failed, retrying`（第一次隧道尝试失败）、
+`Upstream proxy retry failed, falling back to direct`（本条请求改为直连继续）；
+启动横幅的 `upstreamProxyFallback` 字段可直接确认生效值。
+
+```bash
+CC_UPSTREAM_PROXY_RETRY_DELAY_MS=0 npm start    # 隧道失败立即重试一次，再决定是否回退直连
+```
+
+> **注意**：代理长时间不可用时，每条隧道请求都要先白付一次失败的隧道尝试加退避延迟，
+> 才轮到直连 —— 这是按请求逐条回退，不是熔断器。
+
 ## 内存与部署
 
 > 数据来自 [issue #20](https://github.com/MAXeaglet/commandcode-proxy/issues/20) 的实测复现（Node v24，loopback mock 上游）。

diff --git a/proxy.mjs b/proxy.mjs
@@ -1476,9 +1476,51 @@ async function proxyFetch(urlStr, options = {}) {
   });
 }
 
-/** 上游请求入口：配了代理走隧道，否则用原生 fetch（默认路径行为完全不变）。 */
-function upstreamFetch(urlStr, options) {
-  return UPSTREAM_PROXY ? proxyFetch(urlStr, options) : fetch(urlStr, options);
+// ── 隧道传输层重试 + 直连回退 ───────────────────────
+// 已配 CC_UPSTREAM_PROXY 时，隧道本身可能不可用（代理进程重启 / 线路抖动：
+// TLS 握手被对端掐断、首包后 RST）。#50 的流级重试救不了这一层 —— 它重试时
+// 仍走原路（隧道），代理不可用则每次都失败，客户端只会看到连续 502。
+// 传输层失败不是请求本身的问题：等一小段退避再走一次隧道，仍失败则这一条请求
+// 改走直连 —— 代理不可用时退化为「慢但可用」，而不是把 502 抛给客户端。
+// 边界：客户端主动取消（AbortError / signal 已 abort）不算传输失败，
+// 已 abort 就不再发起新尝试，也不会在退避后改走直连。
+const PROXY_RETRY_DELAY_MS = (() => {
+  const ms = Number.parseInt(process.env.CC_UPSTREAM_PROXY_RETRY_DELAY_MS ?? '', 10);
+  return Number.isFinite(ms) && ms >= 0 ? ms : 200;
+})();
+
+const TRANSPORT_CODES = new Set([
+  'ECONNRESET', 'EPIPE', 'ETIMEDOUT', 'ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN',
+  'EHOSTUNREACH', 'ENETUNREACH', 'UND_ERR_SOCKET', 'UND_ERR_CLOSED',
+]);
+
+function isTransportFailure(e) {
+  if (!e || e.name === 'AbortError') return false;
+  const code = e.code || e.cause?.code;
+  if (code && TRANSPORT_CODES.has(code)) return true;
+  const message = String(e.message || '');
+  return /aborted|socket hang up|other side closed|Client network socket disconnected|secure TLS connection|fetch failed|terminated/i.test(message);
+}
+
+/** 上游请求入口：配了代理走隧道（传输层失败重试一次后回退直连），否则用原生 fetch（行为不变）。 */
+async function upstreamFetch(urlStr, options) {
+  if (!UPSTREAM_PROXY) return fetch(urlStr, options);
+  const signal = options?.signal;
+  const retryable = () => !(signal && signal.aborted);
+  try {
+    return await proxyFetch(urlStr, options);
+  } catch (e) {
+    if (!retryable() || !isTransportFailure(e)) throw e;
+    log('warn', 'Upstream proxy attempt failed, retrying', { error: e.message, code: e.code ?? null });
+    await new Promise((r) => setTimeout(r, PROXY_RETRY_DELAY_MS));
+    try {
+      return await proxyFetch(urlStr, options);
+    } catch (e2) {
+      if (!retryable() || !isTransportFailure(e2)) throw e2;
+      log('warn', 'Upstream proxy retry failed, falling back to direct', { error: e2.message, code: e2.code ?? null });
+      return fetch(urlStr, options);
+    }
+  }
 }
 
 // ── 流式转发 ────────────────────────────────────────
@@ -4025,6 +4067,9 @@ server.listen(CFG.port, CFG.host, () => {
     idleTimeouts: `stream ${STREAM_IDLE_TIMEOUT_MS}ms / nonstream ${NONSTREAM_IDLE_TIMEOUT_MS}ms`,
     maxInflight: MAX_INFLIGHT > 0 ? `${MAX_INFLIGHT} (global, /health exempt)` : 'unlimited (CC_MAX_INFLIGHT=0)',
     upstreamProxy: redactProxyUrl(UPSTREAM_PROXY),
+    upstreamProxyFallback: UPSTREAM_PROXY
+      ? `on: 1 transport retry then direct fallback (delay ${PROXY_RETRY_DELAY_MS}ms)`
+      : 'n/a (no proxy configured)',
     upstreamRetry: UPSTREAM_RETRY_MAX > 0
       ? `${UPSTREAM_RETRY_MAX} retries, base ${UPSTREAM_RETRY_BASE_MS}ms (only before first byte)`
       : 'disabled (CC_UPSTREAM_RETRY_MAX=0)',

diff --git a/test/upstream-proxy-fallback.test.mjs b/test/upstream-proxy-fallback.test.mjs
@@ -0,0 +1,151 @@
+// 隧道传输层重试 + 直连回退：已配 CC_UPSTREAM_PROXY 时，隧道本身可能不可用
+// （代理进程重启、线路抖动：TLS 握手被掐断、首包后 RST）。#50 的流级重试救不了
+// 这一层 —— 它重试时仍走原路（隧道），代理不可用则每次都失败，客户端只看到连续 502。
+//
+// 本组用例守住四条边界：① 隧道可用时行为不变（sanity）；
+// ② 传输层失败 → 退避后重试一次隧道，仍失败则该条请求回退直连，下游只看到一次 200；
+// ③ 非传输层失败（CONNECT 被拒绝）不重试、不回退 —— 那是配置问题，不该悄悄换路；
+// ④ 退避期间客户端断连 → 不再发起新尝试（下游已经走了，换路只是白打上游）。
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import http from 'node:http';
+import net from 'node:net';
+import { setTimeout as sleep } from 'node:timers/promises';
+import { allocPort, closeServer, startProxy, startMockUpstream } from './helpers.mjs';
+
+const AUTH = { Authorization: 'Bearer user_test' };
+const CHAT = { model: 'm', messages: [{ role: 'user', content: 'hi' }] };
+
+/**
+ * 一个可编排行为的 HTTP CONNECT 代理 mock。behavior 超出 script 长度则重复最后一项：
+ *   'ok'        正常建立隧道并转发到真 mock 上游（sanity 用）
+ *   'rst'       接受 TCP 连接后直接掐断 —— 隧道层传输失败（ECONNRESET / socket hang up）
+ *   'reject'    回 HTTP 403 拒绝 CONNECT —— 非传输层失败，upstreamFetch 必须原样上抛
+ *
+ * 注意：首个对话请求前会先发 fingerprint / lifecycle 两个 init 预请求，它们同样走
+ * upstreamFetch → 同样占 CONNECT。所以统一用 ['ok','ok',...] 先把 init 走完
+ * （成功后 8h 内不再刷新），之后的 connect 才对应对话请求本身，计数才可断言。
+ */
+async function startConnectProxy(script) {
+  const port = await allocPort();
+  const state = { connects: 0 };
+  let behavior;
+  const server = http.createServer((req, res) => {
+    res.writeHead(405);
+    res.end();
+  });
+  server.on('connect', (req, clientSocket, head) => {
+    state.connects++;
+    behavior = script[Math.min(state.connects - 1, script.length - 1)];
+    if (behavior === 'rst') { clientSocket.destroy(); return; }
+    if (behavior === 'reject') {
+      clientSocket.write('HTTP/1.1 403 Forbidden\r\nProxy-Authenticate: Basic realm="mock"\r\n\r\n');
+      clientSocket.destroy();
+      return;
+    }
+    // 'ok'：把隧道另一端接到 mock 上游，纯字节转发
+    const [host, portStr] = req.url.split(':');
+    const upstream = net.connect(Number(portStr), host, () => {
+      clientSocket.write('HTTP/1.1 200 Connection Established\r\n\r\n');
+      if (head?.length) upstream.write(head);
+      clientSocket.pipe(upstream);
+      upstream.pipe(clientSocket);
+    });
+    upstream.on('error', () => clientSocket.destroy());
+    clientSocket.on('error', () => upstream.destroy());
+  });
+  await new Promise(r => server.listen(port, '127.0.0.1', r));
+  return { port, state, close: () => closeServer(server) };
+}
+
+async function setupFallback(script, env = {}) {
+  const mock = await startMockUpstream();
+  const tunnel = await startConnectProxy(script);
+  const proxy = await startProxy({
+    upstreamPort: mock.port,
+    env: { CC_UPSTREAM_PROXY: `http://127.0.0.1:${tunnel.port}`, ...env },
+  });
+  return {
+    mock, tunnel, proxy,
+    chat: (stream = true) => proxy.post('/v1/chat/completions', { ...CHAT, stream }, AUTH),
+    async close() { await proxy.kill(); await tunnel.close(); await mock.close(); },
+  };
+}
+
+/** 日志经 stdout 管道异步写出，断言前要等它落盘（而不是和刷盘赛跑） */
+async function waitForLog(proxy, pattern, count = 1, timeoutMs = 3000) {
+  const deadline = Date.now() + timeoutMs;
+  while (Date.now() < deadline) {
+    if ((proxy.logs().match(pattern) || []).length >= count) return;
+    await sleep(25);
+  }
+  assert.fail(`日志未出现 ${count} 次 ${pattern}\n实际日志:\n${proxy.logs()}`);
+}
+
+// ── ① sanity：隧道可用时行为与改动前完全一致 ────────────────
+
+test('隧道可用 → 请求照常经隧道到达上游，行为不变', async () => {
+  const s = await setupFallback(['ok']);
+  try {
+    const r = await s.chat(true);
+    const text = await r.text();
+    assert.equal(r.status, 200);
+    assert.match(text, /hello/, '正文应来自 mock 上游，经隧道送达');
+    assert.equal(s.tunnel.state.connects, 3, '2 次 init + 1 次对话，全部走隧道');
+    assert.equal(s.mock.generateCount(), 1);
+  } finally { await s.close(); }
+});
+
+// ── ② 传输层失败 → 重试一次隧道 → 仍失败则回退直连 ───────────
+
+test('隧道 RST 两次 → 回退直连成功，下游无感', async () => {
+  const s = await setupFallback(['ok', 'ok', 'rst']);   // 前两次是 init 预请求
+  try {
+    const r = await s.chat(true);
+    const text = await r.text();
+    assert.equal(r.status, 200, '代理不可用时该退化为慢但可用，而不是 502');
+    assert.match(text, /hello/, '正文应来自回退后的直连请求');
+    assert.equal(s.tunnel.state.connects, 4, '对话请求先重试一次隧道（共 2 次），然后才换路');
+    assert.equal(s.mock.generateCount(), 1, '上游只会被直连请求命中一次');
+    await waitForLog(s.proxy, /Upstream proxy attempt failed, retrying/);
+    await waitForLog(s.proxy, /Upstream proxy retry failed, falling back to direct/);
+  } finally { await s.close(); }
+});
+
+// ── ③ 非传输层失败：原样上抛，不重试、不回退 ─────────────────
+
+test('CONNECT 被 403 拒绝 → 不重试也不换路，如实回 502', async () => {
+  const s = await setupFallback(['ok', 'ok', 'reject']);   // 前两次是 init 预请求
+  try {
+    const r = await s.chat(true);
+    const text = await r.text();
+    assert.equal(r.status, 502, '代理拒绝 CONNECT 是配置/认证问题，必须原样暴露');
+    assert.match(text, /proxy_error/);
+    assert.equal(s.tunnel.state.connects, 3, '非传输层失败不得触发重试');
+    assert.equal(s.mock.generateCount(), 0, '不得悄悄改走直连');
+    assert.doesNotMatch(s.proxy.logs(), /retrying|falling back to direct/);
+  } finally { await s.close(); }
+});
+
+// ── ④ 退避期间客户端断连 → 不再发起新尝试 ────────────────────
+
+test('首次隧道失败后退避期间客户端断连 → 不再重试隧道，也不回退直连', async () => {
+  const s = await setupFallback(['ok', 'ok', 'rst'], { CC_UPSTREAM_PROXY_RETRY_DELAY_MS: '2000' });
+  try {
+    const ac = new AbortController();
+    const pending = fetch(`${s.proxy.base}/v1/chat/completions`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json', ...AUTH },
+      body: JSON.stringify({ ...CHAT, stream: true }),
+      signal: ac.signal,
+    }).catch(() => null);
+    await waitForLog(s.proxy, /Upstream proxy attempt failed, retrying/);   // 已进入退避
+    ac.abort();
+    await pending;
+    await sleep(2600);                                                      // 退避 2s 早已过去
+    assert.equal(s.mock.generateCount(), 0, '断连后不得回退直连打上游');
+    assert.ok(s.tunnel.state.connects <= 4,
+      `断连后的第 2 次隧道尝试只能被 abort 立即打断（connects=${s.tunnel.state.connects}）`);
+    assert.doesNotMatch(s.proxy.logs(), /falling back to direct/);
+  } finally { await s.close(); }
+});