Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/security-regressions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ jobs:
python-version: '3.12'
- run: python -m pip install -r backend/requirements-ai.txt pillow numpy uv pip-audit
- run: python -m pip check
- run: python -m pytest backend/tests/test_ai_radiology.py -q
- run: python -m pytest backend/tests/test_security_regressions.py backend/tests/test_biomedparse_demo.py backend/tests/test_clinical_platform.py backend/tests/test_ai_live.py backend/tests/test_ai_text.py backend/tests/test_ai_codex.py backend/tests/test_ai_codex_exploration.py backend/tests/test_ai_actions.py backend/tests/test_ai_exploration_contracts.py backend/tests/test_ai_exploration_coverage.py backend/tests/test_ai_exploration_routes.py backend/tests/test_ai_exploration_lifecycle.py backend/tests/test_ai_exploration_tools.py backend/tests/test_ai_research.py backend/tests/test_ai_providers.py backend/tests/test_ai_openai.py backend/tests/test_ai_connection_races.py backend/tests/test_ai_screen_awareness.py backend/tests/test_ai_credentials.py backend/tests/test_ai_research_settings.py backend/tests/test_ai_evidence_review.py backend/tests/test_ai_evidence_routes.py backend/tests/test_ai_evidence_provenance.py backend/tests/evidence_review -q
- name: Resolve and audit research dependencies
run: |
Expand Down
12 changes: 1 addition & 11 deletions .impeccable/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,6 @@
"detector": {
"ignoreRules": [],
"ignoreFiles": [],
"ignoreValues": [
{
"rule": "broken-image",
"value": "*",
"files": [
"viewer/assets/live/panel.ts"
],
"createdAt": "2026-09-23T05:15:16.354Z",
"reason": "Agent: the transient viewport preview starts inside a hidden container; render assigns its validated captured JPEG data URL before paint and hides it before clearing src. Native Electron vision acceptance verified a decoded preview. No placeholder image is visible."
}
]
"ignoreValues": []
}
}
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ The standalone [evidence-review runbook](backend/evidence_review/README.md) docu
- See `roadmap/ai-backend/NIM_IMPLEMENTATION.md` for configuration, commands and dated acceptance. Keep provider keys separate and never silently fall back when the chosen model fails.

- The sidebar **Settings → Research models** saves Gemini/NVIDIA NIM provider and exact model per signed account. Environment supplies the default until a saved choice exists. The dropdown includes the entire hosted NVIDIA catalog; catalog inclusion does not verify tool support or entitlement. Saving closes that account's sessions/jobs and requires reconnection; live voice selection and offline evidence CLI model flags remain separate.
- Keep the AI sidebar compact: show the text/research model near the header and separate Chat, Research and Jev review views. Optional voice setup is collapsed within Chat; data attestation and disclosure live by the composer. Keep media state visible while connected and preserve separate microphone/image consent.
- Keep the AI sidebar compact: show the text/research model near the header and separate Chat, Research and Jev review views. Optional voice setup is behind the header Voice button; data attestation and disclosure live by the composer. Images default to None; explicit scope plus Send captures fresh pixels automatically. Keep acknowledged delivery near the composer, collapse completed native activity, and preserve separate microphone/image consent.

- Sidebar Jev reviews run through backend-owned `/api/ai/sidebar/*/evidence-reviews` contracts in research/pilot only, using backend-only `RADSYSX_TYPESAFE_AI_API_KEY`. They retain original answers, require independent text confirmation, and persist POSIX-private artifacts in `.ai-evidence/` beside the actual database (or absolute `RADSYSX_AI_EVIDENCE_DIR`). Ending voice preserves review; explicit cancel/account changes/logout stop it; source-history deletion cancels jobs and removes their artifacts. Clinical use and the qualified human evidence-quality study remain unapproved/pending.

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,4 @@ Typed **Send** and explicit **Research** work without Gemini Live/OpenAI Realtim

ChatGPT/Codex subscription sign-in is available under desktop AI Settings for typed chat and public PubMed research. It uses isolated backend-owned Codex App Server and the OS keyring, not OpenAI API credentials or Realtime entitlement. Read `roadmap/ai-backend/CODEX_SUBSCRIPTION.md`; never copy the user's existing Codex auth.

The AI sidebar separates Chat, Research and Jev review. Optional voice setup stays in Chat; the review workspace hides the composer and collapses abstracts/technical receipts. Sidebar evidence uses versioned intact cited passages and explicitly reports empty previews as unavailable. See `roadmap/ai-backend/JEV_SIDEBAR_IMPLEMENTATION.md` for the corrected workflow and live acceptance.
The AI sidebar separates Chat, Research and Jev review. Optional voice setup is behind the header Voice button. Explicit Images selection plus Send captures fresh pixels automatically, with acknowledged delivery beside the composer. The review workspace hides the composer and collapses abstracts/technical receipts. Sidebar evidence uses versioned intact cited passages and explicitly reports empty previews as unavailable. See `roadmap/ai-backend/JEV_SIDEBAR_IMPLEMENTATION.md` for the corrected workflow and live acceptance.
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ The OHIF sidebar offers `gemini-3.8-live-extended-thinking` and `gpt-realtime-2.

**ChatGPT subscription:** in **Settings**, choose **Sign in with ChatGPT**, continue in the official browser flow, then select **ChatGPT / Codex subscription** and an account-available model under Text & research models. Sign-in preserves your existing model until you save. Uses your plan’s Codex allowance for text/public PubMed research; Realtime voice remains separately API-key billed. Credentials stay in an isolated OS-keyring account. See the [subscription runbook](roadmap/ai-backend/CODEX_SUBSCRIPTION.md) for setup, limits and validation.

The sidebar separates **Chat**, **Research**, and **Jev review**. Chat discusses the case; Research shows literature answers and collapsed source lists. Optional voice setup is inside Chat. From a completed PubMed result, choose **Review evidence with Jev** to enter a dedicated review view. Select the unchanged cited passages, inspect their abstracts, confirm public/synthetic text, then explicitly start Jev. Results show abstract-support judgments with execution receipts under details. Empty previews explain why Jev has not run and offer **Prepare review again**. It does not analyze image pixels.
The sidebar separates **Chat**, **Research**, and **Jev review**. Chat discusses the case; Research shows literature answers and collapsed source lists. Optional voice setup is behind the header **Voice** button. From a completed PubMed result, choose **Review evidence with Jev** to enter a dedicated review view. Select the unchanged cited passages, inspect their abstracts, confirm public/synthetic text, then explicitly start Jev. Results show abstract-support judgments with execution receipts under details. Empty previews explain why Jev has not run and offer **Prepare review again**. It does not analyze image pixels.

**Literature research** cards show the recorded provider/model and worker steps, including waiting for the model, searching PubMed and preparing the answer. Completed, timed-out and cancelled jobs stay visible in history. Model configuration alone does not mean a job is running. [Desktop activation evidence](roadmap/ai-backend/DESKTOP_AI_ACTIVATION.md) separates local checks from hosted-provider results.

Expand Down Expand Up @@ -561,10 +561,12 @@ NVIDIA NIM is available for explicit evidence evaluation and opt-in PubMed resea

### Attach an image to Codex Chat or Research

With a ChatGPT/Codex model selected, confirm synthetic/deidentified data and choose **Current image** under **Share images with AI** (or **Attach current view** in a client without study sharing). Expand **Preview image** to inspect the exact snapshot and visible measurement overlays, then type your question and choose **Send** or **Research**. The selected model must advertise image input; single-image input on `gpt-6-astra` was verified on 2026-09-23. This works without Realtime. This option shares only that captured viewport. Saved image receipts record what was submitted; pixels are not saved in RadSysX history. See the [subscription runbook](roadmap/ai-backend/CODEX_SUBSCRIPTION.md).
With a ChatGPT/Codex model selected, confirm synthetic/deidentified data and choose **Active viewport** in the composer’s **Images** selector, then **Send with images** or **Research**. A fresh snapshot includes visible measurement overlays. Clients without study sharing retain **Attach current view** and its removable preview. The selected model must advertise image input. This works without Realtime. Saved image receipts record what was submitted; pixels are not saved in RadSysX history. See the [subscription runbook](roadmap/ai-backend/CODEX_SUBSCRIPTION.md).

### Share a study with your Codex model

In the AI sidebar, select your ChatGPT/Codex model in Settings, confirm synthetic/deidentified data, then open **Share images with AI**. Choose the active image, the reading view, or the entire active series. **Allow viewer tools** separately permits native navigation and reversible edits. Prepare the scope and send your question with Send or Research; a voice connection is optional.
In the AI sidebar, select your ChatGPT/Codex model in Settings, confirm synthetic/deidentified data, then choose **Whole reading view** or **Active series · all frames** in **Images**. **Let AI use viewer tools** separately permits native navigation and reversible edits. **Send with images** or **Research** inventories and captures fresh pixels automatically; no Prepare step or voice connection is needed. Delivery counts and Stop/Take over remain beside the composer. Real `gpt-6-astra` acceptance on 2026-09-23 delivered all 34 synthetic frames and correctly identified a random marker visible only in the pixels; see the [study runbook](roadmap/ai-backend/CODEX_STUDY_EXPLORATION.md).

The [native OpenMed adaptations](roadmap/ai-backend/OPENMED_ADAPTATIONS.md) add scoped technical metadata, literal report structuring, and richer PubMed abstracts/search receipts to the existing tool paths. They do not install NER models or provide clinical validation.

The study card shows acknowledged frame delivery, actions, Stop and Take over. Partial series coverage remains visible and requires explicit Continue review. Saving a report still requires review. See the [implementation and acceptance record](roadmap/ai-backend/CODEX_STUDY_EXPLORATION.md).
2 changes: 1 addition & 1 deletion WARP.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,4 +98,4 @@ Typed **Send** and explicit **Research** work without Gemini Live/OpenAI Realtim

ChatGPT/Codex subscription sign-in is available under desktop AI Settings for typed chat and public PubMed research through isolated, pinned Codex App Server. Subscription credentials stay in the OS keyring; Realtime remains API-key billed. Read `roadmap/ai-backend/CODEX_SUBSCRIPTION.md`.

The AI sidebar separates Chat, Research and Jev review. Optional voice setup stays in Chat; the review workspace hides the composer and collapses abstracts/technical receipts. Sidebar evidence uses versioned intact cited passages and explicitly reports empty previews as unavailable. See `roadmap/ai-backend/JEV_SIDEBAR_IMPLEMENTATION.md` for the corrected workflow and live acceptance.
The AI sidebar separates Chat, Research and Jev review. Optional voice setup is behind the header Voice button. Explicit Images selection plus Send captures fresh pixels automatically, with acknowledged delivery beside the composer. The review workspace hides the composer and collapses abstracts/technical receipts. Sidebar evidence uses versioned intact cited passages and explicitly reports empty previews as unavailable. See `roadmap/ai-backend/JEV_SIDEBAR_IMPLEMENTATION.md` for the corrected workflow and live acceptance.
4 changes: 4 additions & 0 deletions backend/clinical/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,10 @@

## Voice-independent text sessions

- Scoped Codex study turns capture an initial observation before `turn/start` (up to eight remaining series frames, or reading-grid overview/panes). Submit actual inline image inputs; matching turn acceptance acknowledges that initial batch, while later dynamic images require their exact tool completion acknowledgment. Capture failure must never downgrade to text-only inference. Clear transient input buffers after acceptance. Exploration instructions explicitly authorize declared viewer tools and public PubMed queries when requested; never inherit the text-only instruction forbidding tools. Initial delivery and saved coverage are not diagnostic validation.
- `ai_radiology.py` adapts the OpenMed report-structuring and DICOM-metadata workflows into bounded native tools. `series_get_metadata` uses a technical allowlist from validated scoped manifests, excluding patient fields, source UIDs and free-text headers; it is not a full-header PHI audit or SR parser. `structure_radiology_report` preserves literal sections/measurements and exact Unicode source offsets, including negation/uncertainty in original prose. It never infers clinical labels, recommendations or billing codes. Use the existing reviewed `report_draft`/`report_save` path for effects. No OpenMed model dependency or NER inference is implied.
- The native PubMed tool adapts the OpenMed mining workflow: bounded ESearch/EFetch, per-worker paced requests and one bounded retry for 429/503, structured abstract labels, publication metadata/MeSH terms and exact query/translation/PMID receipts. The same implementation runs under native Codex and DeepAgents. Persist only bounded public search receipts; preserve source IDs and original abstracts for existing Jev preparation. MeSH-only queries can miss unindexed articles; include title/abstract variants when appropriate. PMC full-text retrieval, corpus harvesting and entity NER remain outside this adaptation.

- Own `ai_text.py` and `ai_text_routes.py`. POST `/text-sessions` allocates a synthetic/deidentified session using the owner's selected research provider/model, without Realtime setup or voice credentials. Additive `ai_text_sessions` marks transport/provider; existing sessions retain their voice identity. Text session DTOs have `mode=text`, null audio rates/live URL, and cannot attach a WebSocket.
- POST `/sessions/{id}/text-turns` requires signed unexpired `ai.run`, enabled research/pilot, allowed Origin, strict bounded 720,000-byte turn JSON (session creation remains 16 KiB), current context and attestation. Chat accepts 2,000 characters, research 1,700. Idempotency keys bind action/text/context and image receipt; duplicates return the original task. One active typed task per actor, two globally, with the shared subprocess cap and 120-second bound. Cancellation joins the worker, including cancellation before its first instruction. Context/account/model changes, close, expiry and shutdown stop work. Late responses cannot restore cancelled/deleted results.
- Chat uses the same fixed-endpoint native ChatNVIDIA/ChatGoogle adapter in a credential-isolated subprocess, with one `ainvoke` and no tools. Only neutral allowlisted numeric/modality/series metadata and bounded completed chat pairs enter its prompt; explicitly state no pixels or records were provided. Research uses the existing bounded DeepAgents graph with the public question and neutral modality/count only, never prior chat or arbitrary viewer strings. These Gemini/NVIDIA lanes remain text-only; explicit Codex image input is described below.
Expand Down
Loading
Loading