Skip to content

Add viewport vision to subscription chat and research - #83

Merged
TerminallyLazy merged 1 commit into
mainfrom
codex/subscription-viewer-vision
Sep 23, 2026
Merged

TerminallyLazy merged 1 commit into
mainfrom
codex/subscription-viewer-vision

Conversation

@TerminallyLazy

@TerminallyLazy TerminallyLazy commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Codex sidebar requests previously sent only neutral metadata, so an image-capable model could not see the current viewport or its measurement overlays. Add an explicit Attach current view action for Chat and Research: confirm synthetic/deidentified data, inspect a removable preview, then submit one JPEG through the selected subscription model's native image input. Voice remains optional.

The backend verifies exact-model image support and bounds JPEG size, dimensions, age and owned context. Pixels remain transient; saved image receipts make submission visible in chat and research history. Stale previews, unsupported models and uncertain retries preserve the draft and avoid a second dispatch. Existing filesystem/computer tools remain disabled. One viewport is not whole-series review or clinical validation.

Validation:

  • 233 focused backend regressions, 52 viewer tests, 20 desktop protocol tests, root TypeScript checks and viewer build passed.
  • Isolated Electron capture smoke passed for preview, Chat/Research transport, receipts and removal with zero voice connections or persisted image bytes.
  • Real signed-in gpt-6-astra correctly described a generated colored-shape image and its visible label; a second image-plus-research turn completed with a backend PubMed call and citation. No user case image was sent.

Updated owning guidance and the subscription runbook. Updating the user's currently open desktop window is tracked separately to preserve their loaded study/draft.

High-level PR Summary

This PR adds explicit viewport image attachment to Codex Chat and Research requests in the RadSysX AI sidebar. Users can preview and submit one JPEG snapshot of the active viewport (including measurement overlays) through an Attach current view action. The backend validates image support, bounds JPEG size/dimensions/age, and saves only submission receipts—never pixel data—in history. The feature works without Realtime connections and includes comprehensive validation: 233 backend tests, 52 viewer tests, 20 desktop protocol tests, plus isolated Electron smoke tests and real gpt-6-astra acceptance with a synthetic colored-shape image.

⏱️ Estimated Review Time: 15-30 minutes

💡 Review Order Suggestion
Order File Path
1 README.md
2 roadmap/ai-backend/CODEX_SUBSCRIPTION.md
3 backend/clinical/AGENTS.md
4 backend/clinical/ai_view_image.py
5 backend/clinical/ai_codex.py
6 backend/clinical/ai_text.py
7 backend/clinical/ai_text_routes.py
8 packages/clinical-web/src/contracts.ts
9 packages/clinical-web/AGENTS.md
10 viewer/assets/live/protocol.ts
11 viewer/assets/live/controller.ts
12 viewer/assets/live/panel.ts
13 viewer/assets/radsysx-viewer.css
14 desktop/AGENTS.md
15 backend/clinical/ai_evidence_routes.py
16 backend/clinical/ai_fixture_server.py
17 backend/tests/AGENTS.md
18 backend/tests/test_ai_codex.py
19 desktop/scripts/ui-import-smoke.mjs
20 viewer/scripts/AGENTS.md
21 viewer/scripts/live-contracts.ts
22 viewer/scripts/test-live.mjs
23 AGENTS.md
24 CLAUDE.md
25 WARP.md

Need help? Join our Discord

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Sep 23, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 3b99f769-8ed0-49db-b917-eff8882ee2ae


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@TerminallyLazy
TerminallyLazy merged commit 6b49bd4 into main Sep 23, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant