Skip to content

fix: distinguish incomplete cognitive assessments from clean verdicts - #36

Closed
sriaradhyula wants to merge 1 commit into
mainfrom
codex/cognitive-evidence-coverage
Closed

sriaradhyula wants to merge 1 commit into
mainfrom
codex/cognitive-evidence-coverage

Conversation

@sriaradhyula

Copy link
Copy Markdown

Description

An empty cognitive-failure result currently appears as a successful clean assessment even when the reviewer never inspected decisive evidence. This change records review completion separately from findings and makes incomplete, failed, running, and legacy assessments visible in the session UI.

The reviewer receives a bounded packet of captured user requests, terminal responses per agent invocation, and tool inputs/results, with failed results prioritized. It tracks full-window delivery, requires an explicit claim/evidence comparison for each terminal response, and accepts findings only with actual delivered call/span citations. Missing evidence, invalid submissions, missing comparisons, and exhausted review budgets produce incomplete. Retrieval/model/persistence errors produce error. The existing taxonomy and agent instructions are unchanged.

The API adds cognitiveAssessment; the UI shows a qualified no-findings message only for complete. Older sessions default to unknown and need reprocessing. Deploy API, worker, and UI together. Existing failure reports remain retained as before; this does not add versioned assessment history.

Validation and remaining blocker

  • 114 worker tests, 25 targeted API tests, 6 cognition-protocol API tests, and 7 UI regressions pass.
  • TypeScript/Vite production build, changed-file ESLint/Prettier, Ruff, and diff checks pass. Local ESLint required supplying the repository's undeclared Storybook plugin outside tracked dependencies.
  • Correct the existing cognitive-metric fixture to use the API's numeric result contract; the previous JSON-string fixture fails on unmodified main too.
  • Broader database integration tests could not run successfully without local Neo4j/ClickHouse test credentials.

Read-only regression used frozen real normal-agent recordings, the same model, failure definitions, and trace hashes; no agent/tool outputs were invented, and no production reports or queues were written. Final candidate module hashes were verified against this commit's sources.

Recorded case Deployed baseline Final candidate
Known erroneous absence claim after failed search 0 findings in 5 judgments 0 findings; 3/3 incomplete
Capacity investigation acknowledging uncertainty 0 findings in 5 judgments 0 findings; 3/3 complete
Additional real follow-up Not assessed in the baseline corpus 0 findings; incomplete

This is a coverage/status safeguard, not a validated accuracy fix. The judge still misinterprets evidence and fails to produce a verified positive detection. Labels still need independent human confirmation, and repeated judgments of one recording are not independent accuracy samples. Final six-case regression used 3,002,453 prompt and 22,567 completion tokens; the three incomplete cases exhausted 15 turns. Review cost and latency remain material. Keeping this PR draft pending the detector-quality follow-up; no rollout was performed.

Private recordings and request/response audit logs are intentionally excluded from this public repository.

Type of Change

  • Bugfix
  • New Feature
  • Documentation

Checklist

  • I have read the contributing guidelines
  • I have verified this change is not present in other open pull requests
  • Functionality is documented
  • Changed code style checks pass
  • New behavior is covered by automated tests
  • All database integration tests pass (test credentials unavailable)
  • Detector-quality acceptance passes

Prepared with Codex. Commits include the requested Signed-off-by.

Signed-off-by: Sri Aradhyula <sraradhy@cisco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant