Repository navigation
fix: distinguish incomplete cognitive assessments from clean verdicts - #36
Closed
sriaradhyula wants to merge 1 commit into
Closed
sriaradhyula wants to merge 1 commit into
sriaradhyula wants to merge 1 commit into
Conversation
Signed-off-by: Sri Aradhyula <sraradhy@cisco.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An empty cognitive-failure result currently appears as a successful clean assessment even when the reviewer never inspected decisive evidence. This change records review completion separately from findings and makes incomplete, failed, running, and legacy assessments visible in the session UI.
The reviewer receives a bounded packet of captured user requests, terminal responses per agent invocation, and tool inputs/results, with failed results prioritized. It tracks full-window delivery, requires an explicit claim/evidence comparison for each terminal response, and accepts findings only with actual delivered call/span citations. Missing evidence, invalid submissions, missing comparisons, and exhausted review budgets produce
incomplete. Retrieval/model/persistence errors produceerror. The existing taxonomy and agent instructions are unchanged.The API adds
cognitiveAssessment; the UI shows a qualified no-findings message only forcomplete. Older sessions default tounknownand need reprocessing. Deploy API, worker, and UI together. Existing failure reports remain retained as before; this does not add versioned assessment history.Validation and remaining blocker
Read-only regression used frozen real normal-agent recordings, the same model, failure definitions, and trace hashes; no agent/tool outputs were invented, and no production reports or queues were written. Final candidate module hashes were verified against this commit's sources.
This is a coverage/status safeguard, not a validated accuracy fix. The judge still misinterprets evidence and fails to produce a verified positive detection. Labels still need independent human confirmation, and repeated judgments of one recording are not independent accuracy samples. Final six-case regression used 3,002,453 prompt and 22,567 completion tokens; the three incomplete cases exhausted 15 turns. Review cost and latency remain material. Keeping this PR draft pending the detector-quality follow-up; no rollout was performed.
Private recordings and request/response audit logs are intentionally excluded from this public repository.
Type of Change
Checklist
Prepared with Codex. Commits include the requested Signed-off-by.