Repository navigation
fix(cli): pin website capture connections to public peers - #5668
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Edit accuracy: accurate 271 (base branch 271), smooth 173 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (3)
|
terencecho
left a comment
There was a problem hiding this comment.
Source review at exact HEAD 5f35f0250af835bb647aa7cef1da08a3edff01c0, API/effective base 0d6bdd453bbcfae2280cf2b66f0b6b5afccc10ba.
[P2] Preserve callers' explicit transfer deadlines
packages/cli/src/capture/publicNetwork.ts:86–89
The unconditional AbortSignal.timeout(30_000) is combined with, rather than replaced by, the caller's signal, so the new transport terminates healthy response bodies at 30 seconds even when their real consumer explicitly allows 120 seconds. commands/capture/video.ts:13–18 passes that 120-second deadline through safeFetch; its actual CLI route is capture --video … --index … → runVideoMode → streamToFile. Before this change, safeFetch passed that signal directly to fetch; now it calls publicRequest and inherits the shorter deadline.
Concrete source-valid witness: an existing, uniquely named manifest video on public HTTPS, valid TLS, prompt 200/video headers, and a 40 MiB MP4 streaming steadily at 1 MiB/s. This is below the standalone 250 MiB cap and completes at 40 seconds, well within the caller's 120 seconds. BASE can save it; HEAD aborts the body at 30 seconds, streamToFile deletes the partial output, and runVideoMode reports download failure/exit 1. The main capture's mediaCapture.ts:225–287,547–572 is also affected when its remaining budget exceeds 30 seconds: it returns no saved body/localPath. This is a static before/after source witness, not a claimed executed network reproduction.
Honor an explicit caller signal/deadline and apply a default only when needed, or separate bounded connection/header establishment from the caller's body-transfer budget. Add a regression that exercises the real transport with a longer caller deadline; existing consumer mocks do not cover the new timeout.
[P2] Make the new dotenv isolation/opt-out promise cover the shipped entrypoint
docs/packages/cli.mdx:223–227
The new documentation promises that ancestor dotenv cannot change the browser path/runtime settings and that inherited HYPERFRAMES_NO_CAPTURE_DOTENV=1 skips those files. That is not true for normal CLI capture. The actual packaged route (package.json bin → bin/hyperframes.mjs → the tsup src/cli.ts entry) first runs cli.ts:63–74: it loads CWD .env with unrestricted applyDotEnv, before the lazy capture command and before the new allowlist/opt-out in capture/scaffolding.ts:35–41.
With the default ./capture output, CWD is an output ancestor. Start with the opt-out inherited and no inherited browser override, and let that CWD .env name an existing browser using HYPERFRAMES_BROWSER_PATH. The early loader still imports it; browser/manager.ts:337–343 selects it, and captureAttempt.ts:121–124 passes that exact executable path to the launcher. No malicious browser behavior is needed to demonstrate the configuration-selection mismatch. The newly restricted helper is correct in isolation, but the earlier entrypoint bypasses the advertised boundary. Make startup capture-aware or explicitly narrow the documentation so callers are not promised an isolation/skip guarantee they do not receive.
Baseline distinction: the broad CLI loader and browser selection are unchanged from BASE. I am flagging the newly introduced false end-to-end security/opt-out contract, not calling the preexisting loader a newly introduced arbitrary-execution vulnerability.
Public concern and unresolved question
The public CodeQL inline reports file data reaching an outbound request. I traced the real sink: hostname is the checked numeric IPv4 returned by resolvePublicAddress, while Host/SNI and TLS certificate verification retain the original identity; redirect targets are revalidated. I did not establish a concrete introduced private-peer bypass from that public taint report. This is not a runtime security guarantee or a claim to have inspected the private alert trace.
The raw-browser signal shutdown does not join the capture wrapper's profile-removal cleanup while postprocessing is pending. Whether the supported Puppeteer/browser itself removes an explicitly supplied profile remains unverified, so I have not promoted actual profile retention to a confirmed blocker. Normal finally/watchdog paths do close the wrapper.
Audited: all 25 changed files across four completed bounded source lanes (11 transport, 5 consumers, 8 configuration/lifecycle, 1 documentation), each BASE-first within its assigned scope; complete HEAD semantics include exact unchanged-byte maps plus all changed blocks where applicable. Real CLI/bootstrap/browser-selection, download consumers, redirects, proxy/session ownership, retry/watchdog/cleanup and cross-scope fix interactions were traced. Parent independently reconstructed both immutable trees, rebound all 41 changed source versions, checked the 25-path coverage union and rebound 10 unchanged real entry/consumer dependencies. This is scoped aggregate coverage, not a claim of two complete whole-PR passes or whole-repository semantic completeness.
Trusting/not exercised: repository tests/compiler/build/CI, browser/socket/native execution, external Puppeteer/Chromium/platform semantics, providers, rendered/packet evidence and unreviewed transitive dependency ranges. The author's local/fixture/CI statements are attributed reports, not my executions. No live network/provider/credential access, source edits or workflow triggers were performed.
Verdict: REQUEST_CHANGES
Reasoning: the new transport shortens existing supported video transfer deadlines, and the new dotenv isolation/opt-out promise is false at the actual shipped CLI entrypoint.
— tai
|
Current successor: This head removes one redundant comment to satisfy the comment-share gate; runtime and test sources are identical to tested implementation Finding-by-finding reconciliation:
Validation: 487 unique passing tests (483 capture/bootstrap tests plus four separately executed actual-Chrome proxy/WebRTC cases), CLI build/typecheck, process-ownership audit, lint/format and normal signed-commit hooks. Cached Vitest 3.2.4 differs from declared 4.1.11. No dependency installation or bypass. Linux signal probes do not certify macOS/Windows or SIGKILL cleanup; no provider/credential calls. Current-head hosted CI and eligible review remain required. The public CodeQL taint path is not dismissed; no introduced private-peer bypass has been established by the source audit. |
terencecho
left a comment
There was a problem hiding this comment.
Reviewed exact HEAD 1679bd8e68e32b728284ccd4f5dc3b8e39e588c9 against API/effective merge-base 0d6bdd453bbcfae2280cf2b66f0b6b5afccc10ba.
Verdict: REQUEST_CHANGES — the explicit deadline fix is resolved, but capture dotenv isolation still misses supported dispatch, and signal shutdown bypasses explicit-profile cleanup.
[P2] Classify capture using the same command selection as real dispatch
packages/cli/src/cli.ts:64–75
captureEnv checks only argv[0], whereas the bundled Citty 0.2.2 dispatch selects the first non-option subcommand (src/command.ts, findSubCommandIndex). Consequently, a supported invocation such as hyperframes --json capture https://www.wikipedia.org --skip-vision takes the unrestricted early dotenv loader and then actually dispatches capture.
Concrete source-valid schedule: inherit HYPERFRAMES_NO_CAPTURE_DOTENV=1, leave the inherited browser override unset, and run that invocation from a project whose .env supplies an existing legitimate Chrome path through HYPERFRAMES_BROWSER_PATH. The bootstrap sees --json, sets captureEnv=false, and imports the file without an allowlist despite the opt-out. The real browser/manager.ts:337–343 then selects the imported path, which captureAttempt passes to the launcher. The later output-ancestor loader's opt-out cannot undo that environment mutation. This also bypasses the allowlist without the opt-out.
The ordinary capture-first case was fixed in f0d32ac6768c5a9fa9b8f72b83e7989c65a45da2; the retained cli.dotEnv.test.ts cases all place the command first, so they do not cover this real dispatch shape. Share the actual subcommand selection, or otherwise enforce the capture policy before any unrestricted import for every supported capture route, and add a leading-option dispatch regression.
Baseline distinction: BASE already loaded CWD dotenv broadly. This is an incomplete implementation of this PR's new isolation/opt-out contract, not a newly introduced arbitrary-execution vulnerability or a claim that an existing Chrome binary is malicious.
[P2] Join explicit capture-profile cleanup into the signal owner
packages/cli/src/capture/launchCaptureBrowser.ts:30–53
The new launcher creates hf-capture-profile-* and passes it as explicit userDataDir. Its disconnected callback closes only the proxy; profile removal exists in the wrapper's close() and launch-error catch. However, the actual unchanged signal owner in browser/launch.ts:21–31,73–105 registers and closes the raw Puppeteer Browser, not this wrapper. After raw-browser shutdown it calls requestCliExit; cli.ts:346–351 finalizes telemetry and calls process.exit without joining the running capture's wrapper cleanup.
A valid user schedule is Ctrl-C after coreExtractionPhase.ts:428 closes its page while postExtractionPhase.ts:307–336 is awaiting optional vision work. Closing the raw browser need not settle that non-browser promise. The root can therefore exit before captureAttempt reaches its finally, leaving the new profile behind. This is not an impossible within-one-turn race and does not require a browser crash, malicious input, or uncatchable SIGKILL.
The uncertainty recorded in my earlier review is now resolved by the pinned dependency source: the lock selects puppeteer-core@25.11.0 / @puppeteer/browsers@3.2.2; at upstream commit 043cd46e594b39ea77343e221a59724183448059, ChromeLauncher sets isTempUserDataDir only when it creates the profile itself, and cleanUserDataDir removes only that temporary case. BASE omits userDataDir and uses this dependency-owned cleanup; HEAD's explicit directory is not covered by it. Register an idempotent capture-session cleanup with the same signal shutdown owner and await profile removal before root exit. The new disconnected test explicitly calls session.close() afterward, so it does not cover this production shutdown path.
This is a source-proven ownership/retention defect, not a claim that I executed a signal or inspected any profile contents.
Prior review reconciliation
- Explicit transfer deadline: resolved by
f0d32ac6768c5a9fa9b8f72b83e7989c65a45da2. CurrentpublicRequestusesinit.signal ?? AbortSignal.timeout(30_000)through DNS/request/body, preserving the real standalone video's 120-second caller deadline. The retained HTTPS fixture streams 35 MiB over 35 seconds; separate cases distinguish default expiry from caller cancellation. Regression sources read, not run. - Early dotenv boundary: partially resolved, with the supported leading-option bypass above remaining.
- Public CodeQL concern: numeric IPv4 dialing, original Host/SNI, TLS verification and redirect revalidation were traced. No concrete introduced private-peer bypass was established from the public taint report. I did not inspect or dismiss the private scanning alert.
Audited: Two independent complete changed-source passes cover all 27 changed paths, 44 present BASE/HEAD versions, and tree-proven absences, BASE-first with explicit byte-equality transfers where applicable. Actual CLI packaging/bootstrap/dispatch/browser selection, transport/download consumers, proxy/session ownership, retry/watchdog/postprocessing/cleanup and bundled cross-feature pairs were traced. Parent separately reconstructed both current trees, rehashed all 44 changed versions, rebound real dependencies and the pinned Citty/Puppeteer sources, and independently checked the two findings and historical fixes. This is not a third full semantic pass or a whole-repository audit.
Trusting / not exercised: Repository tests, compiler/build/CI, browser/socket/native execution, rendered/packet evidence, providers and unreviewed transitive source ranges. The dependency source contract above is pinned; arbitrary custom browser binaries and platform runtime behavior are not certified. Author test/CI/help-probe receipts remain attributed reports. No live service/credential access, process signals, source edits, or workflow triggers were performed.
— tai
terencecho
left a comment
There was a problem hiding this comment.
Reviewed exact HEAD b0c90870a8d69e486de301c7d99955ac5dc8ff2d against API/effective merge-base 0d6bdd453bbcfae2280cf2b66f0b6b5afccc10ba.
My earlier change requests are resolved in this revision:
- Explicit transfer deadlines:
f0d32ac6768c5a9fa9b8f72b83e7989c65a45da2preserves the caller signal instead of imposing the default 30-second limit on it. The real standalone video route supplies 120 seconds. Retained HTTP/TLS regression sources cover a 35 MiB/35-second response, default versus explicit expiry, and cancellation after headers. - Leading-option dotenv isolation:
c7ca51969b6c75addc4ae355cf5bd5756483a9caselects the first non-option command before--, matching the pinned Citty 0.2.2 dispatcher for the actual root, which declares no value-taking options. The early CWD and ancestor loaders share the allowlist and inherited opt-out. Actual CLI-dispatch test sources cover direct, one-leading-option and multiple-leading-option capture, inherited browser settings, opt-out and non-capture compatibility. - Explicit-profile signal cleanup: the same fixing commit registers the capture-session cleanup with the shared signal owner before native launch. Shutdown joins pending launches, raw-browser closure and registered session cleanup before requesting root exit; the wrapper removes its explicitly owned profile. The SIGINT/SIGTERM regression sources invoke the real signal owner without first calling
session.close()and inspect profile absence at the root-exit request. Pinned Puppeteer source confirms why explicit profiles require this ownership.
I checked these fixes together with real bootstrap/browser selection, download consumers, redirects, stream cancellation/backpressure, proxy teardown, watchdog/retry and pending postprocessing. The current final commit is a redundant-comment removal after that implementation, not a substitute for current integration review.
Nonblocking compatibility note: capture/publicNetwork.ts:18 intentionally excludes all 192.0.0.0/16, broader than the old shared classifier's special-use /24 exclusions. Consequently some otherwise global IPv4 peers are now refused by both transports. This is a deliberate conservative policy narrowing, not evidence of a private-peer bypass; keep it visible as a compatibility limit.
Public CodeQL concern: the checked numeric IPv4 is the actual dial target; original Host/SNI, certificate verification and redirect revalidation remain in place. No concrete introduced private-peer bypass was established from the public taint report. I did not inspect or dismiss the private scanning alert.
Audited: the aggregate source review covers all 29 changed leaves and 47 present BASE/HEAD versions, including 11 tree-proven additions. The independent adversarial pass covered all changed-source semantics BASE-first, with exact unchanged-span transfers where applicable. The primary pass fully read production and current tests but was partial for unchanged documentation and separately reread BASE-test ranges. Parent independently reconstructed both trees, rehashed every changed version, read the complete generated diff, and checked the key dispatch, deadline and cleanup integration. This is scoped aggregate coverage, not two unqualified full passes or a whole-repository audit.
Trusting / not exercised: repository tests, compiler/build/CI, browser/socket/native execution, platform/packet evidence, providers and unreviewed transitive internals. Regression sources were read, not run. Author execution receipts remain attributed reports; macOS/Windows, arbitrary custom browsers and SIGKILL cleanup are not certified. No credentials/live services, process signals, source edits or workflow triggers were used.
Verdict: APPROVE
Reasoning: the historical deadline, leading-option isolation and profile-cleanup defects are repaired at this exact head; the current source audit established no remaining introduced blocker.
— tai
What changed
Website capture now connects only to checked public IPv4 peers. Browser navigation, redirects, subresources and secondary pages use a per-capture proxy that dials the validated numeric address. Asset, font, CSS, GIF, Lottie and standalone download requests use the same address policy with verified HTTPS and manual redirects.
The browser uses a temporary profile that disables direct WebRTC UDP, removes the implicit loopback proxy bypass and disables QUIC. Capture retains the Chromium sandbox on macOS and Windows; Linux keeps its documented compatibility exception. Capture ancestor dotenv files can supply only supported vision settings; inherited
HYPERFRAMES_NO_CAPTURE_DOTENV=1skips loading them entirely.What I measured
fails without the fix: the baseline browser reaches the loopback redirect fixture; dotenv/sandbox tests fail against the prior policy; removing the WebRTC profile preference reproduces local UDP packets.
What I did NOT exercise
No production/private services, paid providers, or live credentials. Browser fixture coverage uses the available Chrome 147 on Linux; it does not establish macOS/Windows execution or support for arbitrary custom browser binaries. Controlled socket tests substitute the final public peer with an owned local fixture and assert the numeric dial argument.
Compatibility
Capture is deliberately IPv4-only, rejects credentialed/non-HTTP(S) initial URLs, refuses plaintext WebSocket upgrades and rejects asset responses that ignore identity content encoding. Unsupported resources may be absent from a capture. The proxy is a request-routing boundary, not an operating-system sandbox against a compromised browser binary.
Windows test correction
The first Windows run passed 4,926 CLI tests and failed one new assertion that treated uppercase and lowercase environment names as separate keys. Windows aliases those names. The successor separates the wrong-case dotenv rejection test from the uppercase allowed-setting control; production behavior is unchanged. All 35 focused scaffolding/dotenv tests, changed-file lint/format and normal commit hooks pass locally. Fresh hosted Windows CI remains required.
Review follow-up
Current head
1679bd8e6addresses both findings from review at5f35f025:publicRequestnow uses the caller signal when supplied and creates its 30-second default only without one. A real HTTPS fixture streams 35 MiB over 35 seconds under a 120-second caller deadline: reviewed source aborts at 30.01 seconds; the fix completes at 35.09 seconds. Additional tests exercise default expiry and caller cancellation independently.HYPERFRAMES_NO_CAPTURE_DOTENV=1skips that early file too; inherited browser settings are preserved. Actual CLI dispatch tests cover capture with/without opt-out, inherited browser settings and unchanged non-capture behavior.bin/hyperframes.mjsentrypoint exercised from a temporary project with an existing Chrome path, a runtime setting and a synthetic provider key in.env: capture help ignores browser/runtime values, opt-out ignores the provider key too, and non-capture help retains its existing loading behavior. These are startup/help probes, not a capture or provider call.No new Mac/Windows runtime, paid provider, customer content or private-service verification is claimed. The previously documented browser shutdown/profile cleanup limit remains unverified; no new cleanup guarantee is added.
Latest review follow-up (
b0c90870a)This head removes one redundant comment after tested implementation
c7ca51969; runtime and test sources are otherwise identical.--boundary as the current root Citty dispatch (the root declares no value-taking options). Both--json captureand--json --verbose capturenow apply the allowlist and inherited opt-out before command loading. Existing inherited browser settings and non-capture dotenv behavior remain covered. The real CLI-dispatch matrix has fifteen cases; six added cases fail on1679bd8eand pass with this fix. Three rebuiltbin/hyperframes.mjsstartup probes also pass, including leading--jsonwith opt-out.session.close()first: SIGINT and SIGTERM both fail on1679bd8ebecause the explicit profile still exists at the root-exit request, and pass with the fix. Repeated manual close remains idempotent.