Skip to content

test(agent): answer the credential store's keyring prompts in the harness - #8577

Merged
senamakel merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:test/agent-harness-keyring-consent
Nov 6, 2026
Merged

senamakel merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:test/agent-harness-keyring-consent

Conversation

@M3gA-Mind

@M3gA-Mind M3gA-Mind commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Run directly against a staged TinySecurity, the agent harness E2E (tests/agent_harness_e2e.rs) hung for 120 s at sign-in in every test. This PR makes the harness answer the credential store's keyring prompts itself, so its tests get past sign-in.

Problem

Since #8536, signing in stores the session token in the OS keyring, and TinySecurity asks the user to approve that write. Tests answer it through the test-only keyring responder (test-keyring-consent, compiled into the CLI test targets). That responder acts only when OPENHUMAN_TEST_KEYRING_CONSENT is set, and only scripts/test-rust-e2e.sh and scripts/ci/rust-coverage.sh set it. Run directly with cargo test --test agent_harness_e2e, with security-module and a staged TinySecurity, the boot's auth_store_session waits on a prompt nobody answers. Each test fails after its 120 s timeout (error sending request, agent_harness_e2e.rs:636) (finding TS-E2E-018).

Fix

boot_stack_with_options sets OPENHUMAN_TEST_KEYRING_CONSENT=approve for its own test process, unless the run already set a value. The responder still answers only the credential store's prompts, and it still exists only in test builds.

Scope

This removes the sign-in hang only. Past sign-in, the harness hits TinySecurity refusals it doesn't account for yet: the prompt screen has no model under the scripted mock, the temporary action directory is a protected location, and a LocalOnly refusal of the scripted model. So against a staged TinySecurity most of its tests still fail, approval_gate_approve_flow among them. Those gaps are tracked as TS-E2E-034. The official runner (scripts/test-rust-e2e.sh) doesn't stage TinySecurity and already sets the variable, so its runs are unchanged.

Tests

  • Run directly with security-module, a staged TinySecurity and OPENHUMAN_TEST_KEYRING_CONSENT unset, no test in agent_harness_e2e times out at sign-in.
  • Red control: on main the same run fails at sign-in (:636, error sending request after 120 s).

Validation

Cuttlefish job CF-W4-h18 on b0f4d8bd41 (base 6043171902), with a staged TinySecurity, security-module, and OPENHUMAN_TEST_KEYRING_CONSENT unset:

Check Result
fmt pass
clippy (cli --tests, -D warnings) red only on main's type_complexity error at core/observability_reporting.rs:535, which stops the build before the CLI tests compile; it's in every clippy run of main 6043171902 this session
whole agent_harness_e2e, head 0 sign-in timeouts, 0 error sending request. 2 passed, 25 failed and 1 ignored, every failure after sign-in from the TinySecurity refusals above (TS-E2E-034)
red: approval_gate_approve_flow on main fails at sign-in: :636, error sending request, after 121 s

Related: TS-E2E-018, TS-E2E-034.

Submission Checklist

  • Tests added or updated (happy path + at least one failure / edge case) per Testing Strategy: the harness change is proven by agent_harness_e2e itself; the red control (main's harness) times out at sign-in (:636, "error sending request" after 121 s).
  • N/A: diff coverage. Test-only change.
  • N/A: no feature row added, removed or renamed in docs/TEST-COVERAGE-MATRIX.md.
  • N/A: no matrix feature IDs affected; the review finding is listed under Related.
  • No new external network dependencies introduced.
  • N/A: no release-cut surface changes (docs/RELEASE-MANUAL-SMOKE.md).
  • N/A: no GitHub issue tracks this. It is TinySecurity E2E review finding TS-E2E-018 (follow-up gaps: TS-E2E-034).

AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

  • Key: N/A
  • URL: N/A

Commit & Branch

  • Branch: test/agent-harness-keyring-consent
  • Commit SHA: b0f4d8bd41a9dfc08251f5e2aa4283f1db57f780

Validation Run

  • N/A: pnpm --filter openhuman-app format:check, no frontend change
  • N/A: pnpm typecheck, no TypeScript change
  • Focused tests: on cuttlefish (CF-W4-h18)
  • Rust fmt/check (if changed): on cuttlefish (CF-W4-h18)
  • N/A: Tauri fmt/check, no shell change

Validation Blocked

  • command: N/A
  • error: N/A
  • impact: N/A

Behavior Changes

  • Intended behavior change: none in product code; the test harness answers keyring prompts.
  • User-visible effect: none.

Parity Contract

  • Legacy behavior preserved: all product paths; test-only.
  • Guard/fallback/dispatch parity checks: N/A

Duplicate / Superseded PR Handling

  • Duplicate PR(s): none
  • Canonical PR: this
  • Resolution (closed/superseded/updated): N/A

…ness

Since the session moved into the OS keyring, signing in asks the user to
approve storing it. Run without the test scripts' environment, the agent
harness E2E stalled at that prompt in builds with TinySecurity: its
sign-in timed out and the approval tests never ran. boot_stack now opts
the test build's keyring responder in unless the run already chose.
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 1 billable file and costs up to $0.25.

Or wait 48 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c9a7d906-3f8e-4689-8379-6c5de8b3159c
Commits

Reviewing files that changed from the base of the PR and between 6043171 and b0f4d8b.

Files selected for processing: 1

  • tests/agent_harness_e2e.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T06:09:25.711964Z b0f4d8b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tinysweeper

tinysweeper Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

Reviewed head: b0f4d8bd41a9dfc08251f5e2aa4283f1db57f780

Verdict: Ready for maintainer review · Priority: none
Coverage: Reviewed 1 of 1 files.
Pull request purpose (title): test(agent): answer…

Diagnostics and supporting details

Full summary

Pull request purpose (title): test(agent): answer the credential store's keyring prompts in the harness. The test-only responder still answers only the credential store's prompts and remains absent from product builds, so product code and user-visible behavior are unchanged. Latest-push delta: unavailable — no trusted base-to-head diff comparison was available, so no push-delta behavior is asserted.

Review snapshot

Change surface Files Review signal Count
Production 0 Active findings 0
Tests 1 Noted findings 0
Documentation 0 Resolved findings 0
Configuration 0 Pending checks/questions 0

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

What changed

No product code or user-visible behavior changes. Latest-push delta: unavailable — the prior reviewed head exists, but there was no trusted base-to-head diff comparison, and cached prompt differences are not a proven push delta.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

No active actionable findings.

Before merge

None.

Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The change sets OPENHUMAN_TEST_KEYRING_CONSENT to "approve" before the stack boots, matching what scripts/test-rust-e2e.sh already exports and only when the variable is unset. I could not see the agent-harness binary's process model or the test-responder build gating, but the same defaulting is already done by the repository's own E2E scripts, and the change only affects this test binary. No defect identified. _The code index is behind this pull request (indexed at `3aa1ac215243`), so retrieved context may be out of date._

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: Security lane answered with no finding: the change is test-build plumbing backed by the existing test_responder, does not affect product builds, and introduces no attacker-controlled path. It flagged an unconfirmed pre-existing caveat that std::env::set_var on a non-main thread is unsound on newer Rust/Cargo, and this adds a second such call site in the same binary; the lane did not confirm the toolchain and treated it as a caveat rather than a finding.
  • Lane summary: This change sets OPENHUMAN_TEST_KEYRING_CONSENT to "approve" in the test harness boot only when unset, which is test-build plumbing backed by the existing test_responder (which answers only when the variable says "approve" and only for the credential-store caller). It does not affect product builds and introduces no attacker-controlled path. A minor pre-existing caveat is that where an inline env assertion already ran, std::env::set_var on a non-main thread is unsound on newer Rust/Cargo, and this new call adds a second such call site in the same binary; I did not confirm the toolchain from the tree, so I flag it as a caveat rather than a finding. _The code index is behind this pull request (indexed at `3aa1ac215243`), so retrieved context may be out of date._

tests

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: No behavioural change: nothing outside documentation, configuration and tests.

commits

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: Commits lane answered: no sensitive content in committed files, with a stale-index caveat.
  • Lane summary: No sensitive content in committed files. _The code index is behind this pull request (indexed at `3aa1ac215243`), so retrieved context may be out of date._

description

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The diff sets `OPENHUMAN_TEST_KEYRING_CONSENT=approve` in the harness's own boot helper when the variable is unset, which matches the described fix exactly and does nothing in product code; the description accurately covers the change, the test-only scope, and the remaining TS-E2E-034 gaps. No defects found. _The code index is behind this pull request (indexed at `3aa1ac215243`), so retrieved context may be out of date._
Evidence and run details
  • Models: ladder/vectors-oai3, deepseek-v4.1-flash
  • Spend: $0.000551
  • Tokens: 54421 input · 4123 output · 11264 cached · 73 embedding
  • Updated: 2026-11-06T06:15:38Z
Head State Pass summary
b0f4d8bd41a9 ready for maintainer review 0 active finding(s), 0 resolved finding(s) (at 2026-11-06T06:15:38Z)

tinysweeper 0.1.2

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

Diagnostics

Total spend: $0.0006

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Nov 6, 2026
@senamakel
senamakel merged commit 8750138 into tinyhumansai:main Nov 6, 2026
20 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants