From 2e43d65e8b6903f58c502ca2695dabf26d9bf51e Mon Sep 17 00:00:00 2001 From: Catapult291 Date: Thu, 12 Nov 2026 20:38:06 +0800 Subject: [PATCH] =?UTF-8?q?feat:=20=E9=9A=A7=E9=81=93=E4=BC=A0=E8=BE=93?= =?UTF-8?q?=E5=B1=82=E5=A4=B1=E8=B4=A5=E9=87=8D=E8=AF=95=E4=B8=80=E6=AC=A1?= =?UTF-8?q?=E5=90=8E=E8=AF=A5=E6=9D=A1=E8=AF=B7=E6=B1=82=E5=9B=9E=E9=80=80?= =?UTF-8?q?=E7=9B=B4=E8=BF=9E=EF=BC=88#45=20=E8=A1=A5=E5=85=A8=20#50=20?= =?UTF-8?q?=E7=9A=84=E5=8F=A6=E4=B8=80=E5=8D=8A=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #50 的流级重试沿原路重试：走隧道（CC_UPSTREAM_PROXY）的请求重试时仍走隧道， 代理本身不可用则每次都失败，客户端只会看到连续 502。 本 PR 在上游请求入口 upstreamFetch 补上另一半： - 配了代理走隧道；未配代理仍是原生 fetch，默认路径行为不变。 - 隧道发生传输层失败（ECONNRESET / ECONNREFUSED / ETIMEDOUT / EPIPE / ENOTFOUND / EAI_AGAIN / EHOSTUNREACH / ENETUNREACH / UND_ERR_SOCKET / UND_ERR_CLOSED，或 socket hang up / terminated 等消息）→ 退避 CC_UPSTREAM_PROXY_RETRY_DELAY_MS（默认 200ms）后重试一次隧道。 - 第二次仍为传输层失败 → 该条请求改走直连：代理不可用时退化为 「慢但可用」，而不是把 502 抛给客户端。 - 边界：CONNECT 被回 403/407（配置/认证问题）属非传输层失败，原样上抛， 不重试、不悄悄换路；客户端已 abort 则不再发起新尝试（退避期间断连 也不会在退避后改走直连多打一次上游）。 - 新增启动横幅字段 upstreamProxyFallback 回显生效配置。 - 日志：Upstream proxy attempt failed, retrying / Upstream proxy retry failed, falling back to direct。 背景动机见 #45：维护者建议走 CC_UPSTREAM_PROXY（#32）规避直链闪断， 但代理路径自身缺少故障回退 —— 代理一挂，所有请求直接 502。 测试：新增 test/upstream-proxy-fallback.test.mjs（4 项，复用 helpers.mjs）： ① 隧道可用时行为不变；② 隧道两次 RST → 回退直连成功、下游无感； ③ 403 拒绝 CONNECT 不重试不换路；④ 退避期间客户端断连不再发起新尝试。 npm test 全绿（150 项）。README.md / README_zh.md 补了章节与变量表。 --- README.md | 35 ++++++ README_zh.md | 31 ++++++ proxy.mjs | 51 ++++++++- test/upstream-proxy-fallback.test.mjs | 151 ++++++++++++++++++++++++++ 4 files changed, 265 insertions(+), 3 deletions(-) create mode 100644 test/upstream-proxy-fallback.test.mjs diff --git a/README.md b/README.md index 2bfcd35..680ad9c 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,7 @@ commandcode/ | `HOST` | `0.0.0.0` | Listen address → `host` | | `CC_API_BASE` | `https://api.commandcode.ai` | Upstream base URL → `apiBase` | | `CC_UPSTREAM_PROXY` | *(unset)* | Route requests **to the CC upstream** through an HTTP proxy (`http://` CONNECT only); see "Upstream proxy" below → `upstreamProxy` | +| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | Delay before the single tunnel retry when the tunnel fails at the transport level; a second failure falls back to a direct call for that request; see [Proxy tunnel failure → direct fallback](#proxy-tunnel-failure--direct-fallback) | | `PROJECT_SLUG` | `cc-proxy` | `x-project-slug` → `projectSlug` | | `LOG_FILE` | empty | Log file → `logFile` (**synchronous writes**, see [Other notes](#other-notes)) | | `CC_USE_PROVIDER_MODELS` | `true` | Fetch the model list dynamically → `useProviderModels` | @@ -178,6 +179,9 @@ CC_UPSTREAM_PROXY=http://127.0.0.1:7890 npm start - Each upstream request opens its own tunnel connection. TLS is end-to-end: the certificate is validated against the **target hostname**, never against the proxy. - Routing the fingerprint/lifecycle pre-requests through the same proxy matters: if they went out direct while generation went through the proxy, one account would register from two different IPs — exactly the inconsistency you are trying to avoid. - Credentials in the proxy URL (`http://user:pass@host:port`) are never logged: only `host:port` shows up. +- If the tunnel itself is unavailable (proxy down, flapping line), a transport-level failure retries the tunnel + once after a short delay, then falls back to a direct connection for that request — see + [Proxy tunnel failure → direct fallback](#proxy-tunnel-failure--direct-fallback). > Node's built-in `fetch` does **not** read `HTTPS_PROXY`/`HTTP_PROXY`. The official env-var route requires Node ≥ 22.21 / 24.5 plus `NODE_USE_ENV_PROXY=1`; this option works without either. @@ -691,6 +695,37 @@ CC_UPSTREAM_RETRY_BASE_MS=800 npm start # wider backoff (default 400ms) > `/v1/messages` and `/v1/responses` have a different structure and are not covered by this change (drops are still > reported as before). +## Proxy tunnel failure → direct fallback + +`CC_UPSTREAM_PROXY` routes upstream traffic through a CONNECT tunnel. The retry above cannot help when the +**tunnel itself** is unavailable — it retries along the same route, so a dead proxy fails every attempt. + +For a **transport-level** tunnel failure (`ECONNRESET` / `ECONNREFUSED` / `ETIMEDOUT` / `socket hang up` / +`terminated` … — proxy process restarting, flapping line: TLS handshake cut off, RST right after the first +packet) the proxy therefore retries the tunnel once after a short delay, and if that fails too, **falls back +to a direct connection for that request**: a dead proxy degrades to "slower but working" instead of a wall +of 502s. + +- Non-transport failures — a CONNECT answered with `403`/`407` (config or auth problem) — are surfaced as-is: + no retry, no silent rerouting. +- A client abort cancels the request: no retry, no direct fallback, no extra upstream call. +- With no proxy configured, `upstreamFetch` is a plain `fetch` — default behaviour is unchanged. + +| Env var | Default | Description | +|---|---|---| +| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | Delay before the single tunnel retry | + +Logs: `Upstream proxy attempt failed, retrying` (first tunnel attempt failed) and +`Upstream proxy retry failed, falling back to direct` (this request continues over a direct connection). +The startup banner's `upstreamProxyFallback` field shows the effective setting. + +```bash +CC_UPSTREAM_PROXY_RETRY_DELAY_MS=0 npm start # retry the tunnel immediately before falling back +``` + +> **Note**: while the proxy stays down, every tunneled request pays one failed tunnel attempt plus the retry +> delay before the direct call — this is a per-request fallback, not a circuit breaker. + ## Memory & Deployment > Measurements reproduced from [issue #20](https://github.com/MAXeaglet/commandcode-proxy/issues/20) (Node v24, loopback mock upstream). diff --git a/README_zh.md b/README_zh.md index e1b72d3..dce0196 100644 --- a/README_zh.md +++ b/README_zh.md @@ -75,6 +75,7 @@ commandcode/ | `HOST` | `0.0.0.0` | 监听地址 → `host` | | `CC_API_BASE` | `https://api.commandcode.ai` | 上游地址 → `apiBase` | | `CC_UPSTREAM_PROXY` | 空 | 让**发往 CC 上游**的请求走 HTTP 代理（仅 `http://` CONNECT），见下文「上游代理」→ `upstreamProxy` | +| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | 隧道传输层失败时单次重试的延迟，仍失败则该条请求回退直连；见[隧道故障 → 直连回退](#隧道故障--直连回退) | | `PROJECT_SLUG` | `cc-proxy` | `x-project-slug` → `projectSlug` | | `LOG_FILE` | 空 | 日志文件 → `logFile`（**同步写**，见[其它注意事项](#其它注意事项)）| | `CC_USE_PROVIDER_MODELS` | `true` | 动态拉取模型列表 → `useProviderModels` | @@ -173,6 +174,8 @@ CC_UPSTREAM_PROXY=http://127.0.0.1:7890 npm start - 每个上游请求各自建立一条隧道连接。TLS 为端到端：证书按**目标主机名**校验，绝不针对代理降级。 - **指纹/lifecycle 预请求也走代理**是刻意的：若它们直连而上游生成走代理，同一账号会从两个不同 IP 注册 —— 正是你想避免的那种矛盾。 - 代理地址里带账号密码（`http://user:pass@host:port`）时，日志只保留 `host:port`，**不打印口令**。 +- 隧道本身不可用（代理进程重启、线路抖动）时，传输层失败会退避后重试一次隧道，仍失败则该条请求 + 改走直连 —— 见[隧道故障 → 直连回退](#隧道故障--直连回退)。 > Node 原生 `fetch` **不读** `HTTPS_PROXY`/`HTTP_PROXY`。官方环境变量路线需要 Node ≥ 22.21 / 24.5 且设 `NODE_USE_ENV_PROXY=1`；本选项两者都不需要。 @@ -687,6 +690,34 @@ CC_UPSTREAM_RETRY_BASE_MS=800 npm start # 退避拉长（默认 400ms） > **覆盖范围**：目前 `/v1/chat/completions` 的流式与非流式两条路径都接了重试循环； > `/v1/messages` 与 `/v1/responses` 结构不同，未在本次改动中覆盖（闪断仍按原样报错）。 +## 隧道故障 → 直连回退 + +`CC_UPSTREAM_PROXY` 把发往上游的流量装进 CONNECT 隧道。上面的重试救不了**隧道本身**不可用的情况 —— +它沿原路重试，代理挂了就每次都失败。 + +因此对**传输层**隧道失败（`ECONNRESET` / `ECONNREFUSED` / `ETIMEDOUT` / `socket hang up` / `terminated` +等 —— 对应代理进程重启、线路抖动：TLS 握手被掐断、首包后 RST），代理会退避后重试一次隧道， +仍失败则**该条请求改走直连**：代理不可用时退化为「慢但可用」，而不是把 502 墙抛给客户端。 + +- 非传输层失败 —— CONNECT 被回 `403`/`407`（配置或认证问题）—— 原样上抛：不重试、不悄悄换路。 +- 客户端主动取消（signal 已 abort）即终止：不重试、不回退直连、不多打一次上游。 +- 未配代理时 `upstreamFetch` 就是原生 `fetch` —— 默认路径行为不变。 + +| 环境变量 | 默认 | 说明 | +|---|---|---| +| `CC_UPSTREAM_PROXY_RETRY_DELAY_MS` | `200` | 单次隧道重试的延迟（毫秒） | + +日志：`Upstream proxy attempt failed, retrying`（第一次隧道尝试失败）、 +`Upstream proxy retry failed, falling back to direct`（本条请求改为直连继续）； +启动横幅的 `upstreamProxyFallback` 字段可直接确认生效值。 + +```bash +CC_UPSTREAM_PROXY_RETRY_DELAY_MS=0 npm start # 隧道失败立即重试一次，再决定是否回退直连 +``` + +> **注意**：代理长时间不可用时，每条隧道请求都要先白付一次失败的隧道尝试加退避延迟， +> 才轮到直连 —— 这是按请求逐条回退，不是熔断器。 + ## 内存与部署 > 数据来自 [issue #20](https://github.com/MAXeaglet/commandcode-proxy/issues/20) 的实测复现（Node v24，loopback mock 上游）。 diff --git a/proxy.mjs b/proxy.mjs index 8ec2def..4f532f0 100644 --- a/proxy.mjs +++ b/proxy.mjs @@ -1476,9 +1476,51 @@ async function proxyFetch(urlStr, options = {}) { }); } -/** 上游请求入口：配了代理走隧道，否则用原生 fetch（默认路径行为完全不变）。 */ -function upstreamFetch(urlStr, options) { - return UPSTREAM_PROXY ? proxyFetch(urlStr, options) : fetch(urlStr, options); +// ── 隧道传输层重试 + 直连回退 ─────────────────────── +// 已配 CC_UPSTREAM_PROXY 时，隧道本身可能不可用（代理进程重启 / 线路抖动： +// TLS 握手被对端掐断、首包后 RST）。#50 的流级重试救不了这一层 —— 它重试时 +// 仍走原路（隧道），代理不可用则每次都失败，客户端只会看到连续 502。 +// 传输层失败不是请求本身的问题：等一小段退避再走一次隧道，仍失败则这一条请求 +// 改走直连 —— 代理不可用时退化为「慢但可用」，而不是把 502 抛给客户端。 +// 边界：客户端主动取消（AbortError / signal 已 abort）不算传输失败， +// 已 abort 就不再发起新尝试，也不会在退避后改走直连。 +const PROXY_RETRY_DELAY_MS = (() => { + const ms = Number.parseInt(process.env.CC_UPSTREAM_PROXY_RETRY_DELAY_MS ?? '', 10); + return Number.isFinite(ms) && ms >= 0 ? ms : 200; +})(); + +const TRANSPORT_CODES = new Set([ + 'ECONNRESET', 'EPIPE', 'ETIMEDOUT', 'ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN', + 'EHOSTUNREACH', 'ENETUNREACH', 'UND_ERR_SOCKET', 'UND_ERR_CLOSED', +]); + +function isTransportFailure(e) { + if (!e || e.name === 'AbortError') return false; + const code = e.code || e.cause?.code; + if (code && TRANSPORT_CODES.has(code)) return true; + const message = String(e.message || ''); + return /aborted|socket hang up|other side closed|Client network socket disconnected|secure TLS connection|fetch failed|terminated/i.test(message); +} + +/** 上游请求入口：配了代理走隧道（传输层失败重试一次后回退直连），否则用原生 fetch（行为不变）。 */ +async function upstreamFetch(urlStr, options) { + if (!UPSTREAM_PROXY) return fetch(urlStr, options); + const signal = options?.signal; + const retryable = () => !(signal && signal.aborted); + try { + return await proxyFetch(urlStr, options); + } catch (e) { + if (!retryable() || !isTransportFailure(e)) throw e; + log('warn', 'Upstream proxy attempt failed, retrying', { error: e.message, code: e.code ?? null }); + await new Promise((r) => setTimeout(r, PROXY_RETRY_DELAY_MS)); + try { + return await proxyFetch(urlStr, options); + } catch (e2) { + if (!retryable() || !isTransportFailure(e2)) throw e2; + log('warn', 'Upstream proxy retry failed, falling back to direct', { error: e2.message, code: e2.code ?? null }); + return fetch(urlStr, options); + } + } } // ── 流式转发 ──────────────────────────────────────── @@ -4025,6 +4067,9 @@ server.listen(CFG.port, CFG.host, () => { idleTimeouts: `stream ${STREAM_IDLE_TIMEOUT_MS}ms / nonstream ${NONSTREAM_IDLE_TIMEOUT_MS}ms`, maxInflight: MAX_INFLIGHT > 0 ? `${MAX_INFLIGHT} (global, /health exempt)` : 'unlimited (CC_MAX_INFLIGHT=0)', upstreamProxy: redactProxyUrl(UPSTREAM_PROXY), + upstreamProxyFallback: UPSTREAM_PROXY + ? `on: 1 transport retry then direct fallback (delay ${PROXY_RETRY_DELAY_MS}ms)` + : 'n/a (no proxy configured)', upstreamRetry: UPSTREAM_RETRY_MAX > 0 ? `${UPSTREAM_RETRY_MAX} retries, base ${UPSTREAM_RETRY_BASE_MS}ms (only before first byte)` : 'disabled (CC_UPSTREAM_RETRY_MAX=0)', diff --git a/test/upstream-proxy-fallback.test.mjs b/test/upstream-proxy-fallback.test.mjs new file mode 100644 index 0000000..c81cc8f --- /dev/null +++ b/test/upstream-proxy-fallback.test.mjs @@ -0,0 +1,151 @@ +// 隧道传输层重试 + 直连回退：已配 CC_UPSTREAM_PROXY 时，隧道本身可能不可用 +// （代理进程重启、线路抖动：TLS 握手被掐断、首包后 RST）。#50 的流级重试救不了 +// 这一层 —— 它重试时仍走原路（隧道），代理不可用则每次都失败，客户端只看到连续 502。 +// +// 本组用例守住四条边界：① 隧道可用时行为不变（sanity）； +// ② 传输层失败 → 退避后重试一次隧道，仍失败则该条请求回退直连，下游只看到一次 200； +// ③ 非传输层失败（CONNECT 被拒绝）不重试、不回退 —— 那是配置问题，不该悄悄换路； +// ④ 退避期间客户端断连 → 不再发起新尝试（下游已经走了，换路只是白打上游）。 +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import http from 'node:http'; +import net from 'node:net'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { allocPort, closeServer, startProxy, startMockUpstream } from './helpers.mjs'; + +const AUTH = { Authorization: 'Bearer user_test' }; +const CHAT = { model: 'm', messages: [{ role: 'user', content: 'hi' }] }; + +/** + * 一个可编排行为的 HTTP CONNECT 代理 mock。behavior 超出 script 长度则重复最后一项： + * 'ok' 正常建立隧道并转发到真 mock 上游（sanity 用） + * 'rst' 接受 TCP 连接后直接掐断 —— 隧道层传输失败（ECONNRESET / socket hang up） + * 'reject' 回 HTTP 403 拒绝 CONNECT —— 非传输层失败，upstreamFetch 必须原样上抛 + * + * 注意：首个对话请求前会先发 fingerprint / lifecycle 两个 init 预请求，它们同样走 + * upstreamFetch → 同样占 CONNECT。所以统一用 ['ok','ok',...] 先把 init 走完 + * （成功后 8h 内不再刷新），之后的 connect 才对应对话请求本身，计数才可断言。 + */ +async function startConnectProxy(script) { + const port = await allocPort(); + const state = { connects: 0 }; + let behavior; + const server = http.createServer((req, res) => { + res.writeHead(405); + res.end(); + }); + server.on('connect', (req, clientSocket, head) => { + state.connects++; + behavior = script[Math.min(state.connects - 1, script.length - 1)]; + if (behavior === 'rst') { clientSocket.destroy(); return; } + if (behavior === 'reject') { + clientSocket.write('HTTP/1.1 403 Forbidden\r\nProxy-Authenticate: Basic realm="mock"\r\n\r\n'); + clientSocket.destroy(); + return; + } + // 'ok'：把隧道另一端接到 mock 上游，纯字节转发 + const [host, portStr] = req.url.split(':'); + const upstream = net.connect(Number(portStr), host, () => { + clientSocket.write('HTTP/1.1 200 Connection Established\r\n\r\n'); + if (head?.length) upstream.write(head); + clientSocket.pipe(upstream); + upstream.pipe(clientSocket); + }); + upstream.on('error', () => clientSocket.destroy()); + clientSocket.on('error', () => upstream.destroy()); + }); + await new Promise(r => server.listen(port, '127.0.0.1', r)); + return { port, state, close: () => closeServer(server) }; +} + +async function setupFallback(script, env = {}) { + const mock = await startMockUpstream(); + const tunnel = await startConnectProxy(script); + const proxy = await startProxy({ + upstreamPort: mock.port, + env: { CC_UPSTREAM_PROXY: `http://127.0.0.1:${tunnel.port}`, ...env }, + }); + return { + mock, tunnel, proxy, + chat: (stream = true) => proxy.post('/v1/chat/completions', { ...CHAT, stream }, AUTH), + async close() { await proxy.kill(); await tunnel.close(); await mock.close(); }, + }; +} + +/** 日志经 stdout 管道异步写出，断言前要等它落盘（而不是和刷盘赛跑） */ +async function waitForLog(proxy, pattern, count = 1, timeoutMs = 3000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if ((proxy.logs().match(pattern) || []).length >= count) return; + await sleep(25); + } + assert.fail(`日志未出现 ${count} 次 ${pattern}\n实际日志:\n${proxy.logs()}`); +} + +// ── ① sanity：隧道可用时行为与改动前完全一致 ──────────────── + +test('隧道可用 → 请求照常经隧道到达上游，行为不变', async () => { + const s = await setupFallback(['ok']); + try { + const r = await s.chat(true); + const text = await r.text(); + assert.equal(r.status, 200); + assert.match(text, /hello/, '正文应来自 mock 上游，经隧道送达'); + assert.equal(s.tunnel.state.connects, 3, '2 次 init + 1 次对话，全部走隧道'); + assert.equal(s.mock.generateCount(), 1); + } finally { await s.close(); } +}); + +// ── ② 传输层失败 → 重试一次隧道 → 仍失败则回退直连 ─────────── + +test('隧道 RST 两次 → 回退直连成功，下游无感', async () => { + const s = await setupFallback(['ok', 'ok', 'rst']); // 前两次是 init 预请求 + try { + const r = await s.chat(true); + const text = await r.text(); + assert.equal(r.status, 200, '代理不可用时该退化为慢但可用，而不是 502'); + assert.match(text, /hello/, '正文应来自回退后的直连请求'); + assert.equal(s.tunnel.state.connects, 4, '对话请求先重试一次隧道（共 2 次），然后才换路'); + assert.equal(s.mock.generateCount(), 1, '上游只会被直连请求命中一次'); + await waitForLog(s.proxy, /Upstream proxy attempt failed, retrying/); + await waitForLog(s.proxy, /Upstream proxy retry failed, falling back to direct/); + } finally { await s.close(); } +}); + +// ── ③ 非传输层失败：原样上抛，不重试、不回退 ───────────────── + +test('CONNECT 被 403 拒绝 → 不重试也不换路，如实回 502', async () => { + const s = await setupFallback(['ok', 'ok', 'reject']); // 前两次是 init 预请求 + try { + const r = await s.chat(true); + const text = await r.text(); + assert.equal(r.status, 502, '代理拒绝 CONNECT 是配置/认证问题，必须原样暴露'); + assert.match(text, /proxy_error/); + assert.equal(s.tunnel.state.connects, 3, '非传输层失败不得触发重试'); + assert.equal(s.mock.generateCount(), 0, '不得悄悄改走直连'); + assert.doesNotMatch(s.proxy.logs(), /retrying|falling back to direct/); + } finally { await s.close(); } +}); + +// ── ④ 退避期间客户端断连 → 不再发起新尝试 ──────────────────── + +test('首次隧道失败后退避期间客户端断连 → 不再重试隧道，也不回退直连', async () => { + const s = await setupFallback(['ok', 'ok', 'rst'], { CC_UPSTREAM_PROXY_RETRY_DELAY_MS: '2000' }); + try { + const ac = new AbortController(); + const pending = fetch(`${s.proxy.base}/v1/chat/completions`, { + method: 'POST', + headers: { 'Content-Type': 'application/json', ...AUTH }, + body: JSON.stringify({ ...CHAT, stream: true }), + signal: ac.signal, + }).catch(() => null); + await waitForLog(s.proxy, /Upstream proxy attempt failed, retrying/); // 已进入退避 + ac.abort(); + await pending; + await sleep(2600); // 退避 2s 早已过去 + assert.equal(s.mock.generateCount(), 0, '断连后不得回退直连打上游'); + assert.ok(s.tunnel.state.connects <= 4, + `断连后的第 2 次隧道尝试只能被 abort 立即打断（connects=${s.tunnel.state.connects}）`); + assert.doesNotMatch(s.proxy.logs(), /falling back to direct/); + } finally { await s.close(); } +});