Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions backend/clinical/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,3 +128,5 @@

- Whole-reading-view scope permits the visible overview/panes, not offscreen acquisition frames. Omit `series_read_frames` from its tool declarations and independently reject non-series frame requests in the service. Only terminal incomplete series runs offer continuation. Preserve the first terminal Stop/Take over reason during worker cleanup.
- Codex PubMed arguments allow 1–10 abstracts and default omitted/null limits to five. Invalid arguments return actionable bounded tool feedback; retain safe search-failure messages with the research result. No successful PubMed receipt means a failed research result, never completed literature research. Distinguish service, timeout, rate-limit and response-format errors without returning raw exceptions or query URLs.

- When a renderer pauses after an already recorded unknown action, the saved activity explains that a viewer action could not be confirmed. Do not label that failure as a user changing controls. Keep unknown-effect revocation and no-replay rules intact.
4 changes: 3 additions & 1 deletion backend/clinical/ai_exploration.py
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,9 @@ async def revoke(self, task_id, actor, *, status='cancelled'):
if not task or task.closing: return await self.snapshot(task_id,actor)
task.closing=True
task.snapshot.status=status; task.snapshot.grant.status='revoked'
task.snapshot.activity='Viewer control changed. Review paused.' if status=='paused' else 'Review stopped.'
uncertain_action=any(action['status']=='outcome_unknown' for action in task.snapshot.actions)
task.snapshot.activity=('A viewer action could not be confirmed. Review paused; inspect viewer activity before retrying.' if uncertain_action
else 'Viewer control changed. Review paused.') if status=='paused' else 'Review stopped.'
self.live.actions.release_viewer(actor,task.snapshot.grant.grant_id)
for key,op in task.operations.items():
if not op.future.done():
Expand Down
2 changes: 2 additions & 0 deletions desktop/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,5 @@

- Series-scoped capture can coexist with adjacent unshared panes: main captures only explicitly permitted study/series panes and rejects an unshared requested pane. Whole-view overview still requires every visible pane to match its scope. Sensitive-panel, epoch, revision, ownership and before/after surface checks remain mandatory.
- The scoped synthetic smoke drives the Research tab through an eight-frame partial answer and the visible remaining-frames action, verifies 34/34 cumulative delivery and preserved draft, then proves whole-view and active-viewport scope counts remain distinct. The real subscription smoke additionally requires a successful public PubMed receipt and source alongside synthetic-pixel delivery and native actions; this is separate from clinical accuracy.

- Combining --study-inventory with the synthetic --study-exploration smoke additionally exercises a two-pane layout: select the first pane, jump the second to frame index 15, apply the native brain preset at width 80/center 40, and verify the first pane did not scroll. This tests the real OHIF adapter without hosted inference or patient images.
23 changes: 23 additions & 0 deletions desktop/scripts/ui-import-smoke.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -575,6 +575,7 @@ async function runUiImportSmoke(publicBaseUrl, debugPort) {
nativeTools:Object.keys(group?.getToolInstances?.()??{})};
})()`, 30000);
studyInventory.capabilities = await evaluateInRenderer(cdp, `(async()=>{const a=new window.__RadSysXSmokeAdapter.OHIFAdapter();a.bind(window.__RADSYSX_OHIF_MANAGERS__);return (await a.execute('viewer_get_capabilities',{})).capabilities;})()`, 30000);
if(studyExplorationSmoke) studyInventory.targetedActions = await evaluateInRenderer(cdp, `(${exerciseTargetedReading.toString()})()`, 30000);
}
let adapterState;
if (aiViewerSmoke) {
Expand Down Expand Up @@ -2792,3 +2793,25 @@ async function exerciseStudyExploration(real = false) {
}
return {framesDelivered:34,continuedResearch:!real,scopeCounts:real?undefined:{currentImage:1,readingView:2,seriesFrames:34},imagesDelivered:seriesImages,cloudCalls:real,...(real?{answer:result.result.summary}:{nativeActions:2,paneObserved:true,voiceConnections:0})};
}


async function exerciseTargetedReading() {
const managers=window.__RADSYSX_OHIF_MANAGERS__,services=managers.servicesManager.services;
const adapter=new window.__RadSysXSmokeAdapter.OHIFAdapter();adapter.bind(managers);
const wait=async(fn)=>{const end=Date.now()+8000;while(!fn()){if(Date.now()>end)throw Error('Two-pane fixture not ready');await new Promise(r=>setTimeout(r,40));}};
const displaySetInstanceUID=services.displaySetService.activeDisplaySets[0].displaySetInstanceUID;
await adapter.execute('viewer_set_layout',{rows:1,columns:2});
const ids=[...services.viewportGridService.getState().viewports.keys()];
await managers.commandsManager.runCommand('setDisplaySetsForViewports',{viewportsToUpdate:ids.map(viewportId=>({viewportId,displaySetInstanceUIDs:[displaySetInstanceUID]}))},'CORNERSTONE');
await wait(()=>ids.every(id=>services.cornerstoneViewportService.getCornerstoneViewport(id)?.getImageIds?.().length===34));
const panes=adapter.context().state.viewports;
await adapter.execute('viewer_select_viewport',{viewportId:panes[0].id});
const first=services.cornerstoneViewportService.getCornerstoneViewport(ids[0]);
const before=first.getCurrentImageIdIndex();
await adapter.execute('viewer_jump_to_slice',{viewportId:panes[1].id,index:15});
await adapter.execute('viewer_set_window_level',{viewportId:panes[1].id,preset:'brain'});
const state=adapter.context().state;
if(state.viewportId!==panes[1].id||state.index!==15||state.windowWidth!==80||state.windowCenter!==40||first.getCurrentImageIdIndex()!==before)throw Error('Named pane or native brain preset failed');
await adapter.execute('viewer_set_layout',{rows:1,columns:1});
return {targetedSecondPane:true,firstPaneUnchanged:true,index:15,brainWindow:80,brainCenter:40};
}
8 changes: 8 additions & 0 deletions roadmap/ai-backend/CODEX_STUDY_EXPLORATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,11 @@ Verification was limited to the affected paths:
- Real signed-in scoped Electron request delivered 34/34 generated frames (43 total observations), correctly returned the random pixel-only marker count of three, navigated to slice index 31, set window width 800/center 80, observed the pane and returned a cited public PubMed source in that same turn. The screenshot confirmed the actual slice/window state and the compact 280 px layout. The check required saved PubMed receipts and sources, not just the model's assertion. No patient images were used.

These checks establish transport, scope, continuation and the exercised viewer commands. They do not validate diagnostic accuracy or every native tool/modality. Original saved failures remain historical failures; the app does not replay them automatically.

## 2026-09-23 named-preset and target-pane repair

Subsequent saved task receipts identified `viewer_set_window_level` with `preset: brain` as the repeated unknown action, with earlier unknown slice jumps. The pinned OHIF customization supplies presets as an array of IDs/descriptions; the adapter incorrectly indexed it by a semantic name. Slice commands also omitted their explicit pane and depended on active-pane timing.

Preset lookup now resolves the selected pane's modality and configured native values before any effect, then sends numeric window/level to that pane. Slice commands carry the native grid viewport explicitly, and reading commands wait for active-pane selection. Known preflight rejections return failed tool results without revoking the task; uncertain effects still stop execution. Their saved pause reason now states that an action could not be confirmed instead of attributing it to the user.

Verification: viewer build, 64 focused viewer/controller tests and 14 backend lifecycle tests passed. The isolated synthetic Electron check completed series continuation and scope checks, then used a real two-pane OHIF layout to navigate the second pane to index 15 and apply the configured brain preset (80/40), leaving the first pane unchanged. No patient images or new hosted-model calls were used in this repair. The previous single-pane/numeric-window acceptance did not cover these defects.
2 changes: 2 additions & 0 deletions viewer/assets/live/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,5 @@
- Scope changes revoke the previous lease and clear its visible receipt before enabling Send. Show cumulative distinct series frames sent separately from model status; whole-view sharing has image counts, not a series-completion denominator. Retrying an unconfirmed submission reuses its exact operation identity. Research results appear newest first.
- Scope checks for read-only state/capabilities filter returned series/panes rather than rejecting an adjacent unshared localizer. Pane observations/native commands validate the targeted pane; overview capture validates every visible pane. Known preflight/read failures return a bounded unavailable result; only uncertain mutations revoke control as outcome unknown.
- Takeover tracking compares study/pane/frame/presentation/layout identity, excluding derived measurement statistics and canvas-size changes. Trusted manual reading interactions still take over immediately. Neither background measurement calculations nor sidebar resizing should cancel research.

- Native window presets come from the modality-specific OHIF customization array (matched by preset ID/description), with compatibility for keyed configurations. Resolve them before effects and execute numeric window/level on the named pane. Slice jumps pass the explicit native grid viewport; wait for active-pane selection before commands that use it. Invalid arguments, unavailable controls and unresolved presets are preflight failures, not unknown mutations; genuinely unconfirmed effects still pause the task.
6 changes: 4 additions & 2 deletions viewer/assets/live/exploration.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { ReadingToolUnavailable } from './reading-tools.js';
import { OHIFAdapter } from './ohif.js';
import { ObservationService, DesktopWorkspaceObserver } from './observations.js';
import { request, type DesktopStudyCapture, type ExplorationGrant, type TaskSnapshot, type ShareSelection, type RendererCommand, type ObservationRequest, type RendererBinding } from './protocol.js';
Expand Down Expand Up @@ -131,8 +132,9 @@ export class ExplorationController {
const raw=await this.adapter.execute(command.name,command.args,signal); current();
const revision=binding.revision+(command.name==='viewer_get_state' || command.name==='viewer_get_capabilities'?0:1);
result={operationId:command.operationId,claimId:claimed.claimId,status:'completed',beforeRevision:binding.revision,revision,state:{...raw,...(raw.state?{state:this.scopedState(raw.state as Record<string,unknown>,binding)}:this.scopedState(raw,binding))},canUndo:raw.canUndo===true};
} catch {
current(); result={operationId:command.operationId,claimId:claimed.claimId,status:!permitted||reading?'failed':'outcome_unknown',beforeRevision:binding.revision,revision:binding.revision, state:{},canUndo:false,error:!permitted||reading?'unavailable':'unknown'};
} catch (error) {
const rejected = !permitted || reading || error instanceof ReadingToolUnavailable;
current(); result={operationId:command.operationId,claimId:claimed.claimId,status:rejected?'failed':'outcome_unknown',beforeRevision:binding.revision,revision:binding.revision, state:{},canUndo:false,error:rejected?'unavailable':'unknown'};
}
}
current();
Expand Down
33 changes: 25 additions & 8 deletions viewer/assets/live/ohif.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { applyMeasurement, applyCalibration, applySegmentation, applyRegion, readMeasurements, measurementSpecs } from './measurements.js';
import { capabilities, READING_TOOLS, ANNOTATION_TOOLS, validateArguments } from './capabilities.js';
import { executeReadingTool } from './reading-tools.js';
import { executeReadingTool, ReadingToolUnavailable } from './reading-tools.js';
import { alive, abortable, CornerstoneSeriesRenderer, type SeriesSource, type PrivateFrame } from './series.js';
import type { Presentation, RendererBinding } from './protocol.js';
import { object, type CaptureRequest, type Json, type ViewerContext } from './protocol.js';
Expand Down Expand Up @@ -259,6 +259,24 @@ export class OHIFAdapter {
return { available, ...(available ? {} : { reason: 'Unavailable for the current pane, data or native tool group.' }) };
} catch { return { available: false, reason: 'Open a supported image in the current study.' }; }
}
private windowLevel(args: Json): { windowWidth: number; windowCenter: number } {
let width = args.windowWidth, center = args.windowCenter;
if (args.preset) {
const { grid } = this.viewport(args.viewportId);
const display = list(this.services.displaySetService?.activeDisplaySets).find(ds => grid.displaySetInstanceUIDs.includes(ds.displaySetInstanceUID));
const modality = String(display?.Modality ?? '');
const presets = this.services.customizationService?.getCustomization?.('cornerstone.windowLevelPresets')?.[modality];
const key = String(args.preset).replaceAll('_', '-');
const entries = Array.isArray(presets) ? presets : Object.values(presets ?? {});
const preset = presets?.[String(args.preset)] ?? entries.find((item: Host) =>
String(item.id).toLowerCase() === `${modality.toLowerCase()}-${key}` ||
String(item.description).toLowerCase().replaceAll(' ', '-') === key);
if (!preset) throw new ReadingToolUnavailable('This window preset is unavailable for the selected pane. Use explicit window width and center.');
width = Number(preset.window); center = Number(preset.level);
}
try { return { windowWidth: number(width, 0.01, 100000, 'window width'), windowCenter: number(center, -100000, 100000, 'window center') }; }
catch { throw new ReadingToolUnavailable('Choose a valid window preset or explicit window width and center.'); }
}
private assertPaneStudy(id: string, studyId: string): void {
const grid = this.services.viewportGridService.getState().viewports.get(id);
const displays = list(this.services.displaySetService?.activeDisplaySets).filter(ds => grid?.displaySetInstanceUIDs.includes(ds.displaySetInstanceUID));
Expand Down Expand Up @@ -317,6 +335,7 @@ export class OHIFAdapter {
async performReadingTool(name: string, args: Json, signal: AbortSignal): Promise<Json> {
alive(signal);
const study = this.studyBinding();
if (name === 'viewer_set_window_level') args = { ...args, ...this.windowLevel(args), preset: undefined };
const { id, viewport, grid } = this.viewport(args.viewportId);
this.assertPaneStudy(id, study.studyId);
const native = this.viewportAdapter(viewport);
Expand All @@ -329,6 +348,7 @@ export class OHIFAdapter {
let result: Json = {};
let presentationChanged = false;
this.services.viewportGridService.setActiveViewportId(id);
await this.settleReading(signal, () => this.services.viewportGridService.getActiveViewportId() === id, guard);
switch (name) {
case 'viewer_select_viewport': predicate = () => this.services.viewportGridService.getActiveViewportId() === id; break;
case 'viewer_set_orientation':
Expand Down Expand Up @@ -473,10 +493,8 @@ export class OHIFAdapter {
return ['zoom','panX','panY','rotation','invert','flipHorizontal','flipVertical'].filter(key => args[key] !== undefined).every(key => typeof args[key] === 'number' ? Math.abs(Number(state[key]) - Number(args[key])) < 0.001 : state[key] === args[key]);
};
if (name === 'viewer_set_window_level') {
const modality = this.context().state.modality;
const preset = args.preset ? this.services.customizationService?.getCustomization?.('cornerstone.windowLevelPresets')?.[String(modality)]?.[String(args.preset)] : undefined;
const width = args.windowWidth ?? preset?.window, center = args.windowCenter ?? preset?.level;
predicate = () => { const state = this.context().state; return Math.abs(Number(state.windowWidth) - Number(width)) < 0.001 && Math.abs(Number(state.windowCenter) - Number(center)) < 0.001; };
predicate = () => { const properties = viewport.getProperties?.(); const range = properties?.voiRange;
return Boolean(range && Math.abs(Math.abs(range.upper-range.lower)+1-Number(args.windowWidth)) < 0.001 && Math.abs((range.upper+range.lower+1)/2-Number(args.windowCenter)) < 0.001); };
}
if (name === 'viewer_jump_to_slice') predicate = () => this.viewport(args.viewportId).viewport.getCurrentImageIdIndex?.() === args.index;
if (name === 'viewer_open_series') predicate = () => this.viewport(args.viewportId).grid.displaySetInstanceUIDs.includes(this.resolve(args.displaySetId,'series'));
Expand Down Expand Up @@ -560,8 +578,7 @@ export class OHIFAdapter {
this.services.viewportGridService.setActiveViewportId(id);
switch (name) {
case 'viewer_set_window_level':
if (args.preset) await run('setWindowLevelPreset', { presetName: text(args.preset, 80) });
else await run('setViewportWindowLevel', { viewportId: id, windowWidth: number(args.windowWidth, 0.01, 100000, 'window width'), windowCenter: number(args.windowCenter, -100000, 100000, 'window center') });
await run('setViewportWindowLevel', { viewportId: id, ...this.windowLevel(args) });
break;
case 'viewer_set_layout': {
const rows = number(args.rows, 1, 4, 'rows'), columns = number(args.columns, 1, 4, 'columns');
Expand All @@ -577,7 +594,7 @@ export class OHIFAdapter {
const count = viewport.getImageIds?.()?.length;
const index = number(args.index, 0, typeof count === 'number' ? count - 1 : 100000, 'slice index');
if (!Number.isInteger(index)) throw new Error('Slice index must be an integer.');
await run('jumpToImage', { imageIndex: index }); break;
await run('jumpToImage', { imageIndex: index, viewport: { id } }); break;
}
case 'viewer_set_tool': {
const toolName = text(args.tool, 80);
Expand Down
6 changes: 4 additions & 2 deletions viewer/assets/live/reading-tools.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
import type { Json } from './protocol.js';
import { READING_TOOLS, validateArguments, type ReadingAdapter } from './capabilities.js';
import { alive } from './series.js';
export class ReadingToolUnavailable extends Error {}

/** Semantic registry only: callers never choose a native command or selector. */
export async function executeReadingTool(adapter: ReadingAdapter,name: string,args: Json,signal: AbortSignal): Promise<Json> {
alive(signal);const capability=READING_TOOLS[name];
if(!capability)throw new Error('Unknown reading tool.');
validateArguments(args,capability.schema);
if(!adapter.readingAvailability(name,args).available)throw new Error('This reading control is unavailable in the current view.');
try { validateArguments(args,capability.schema); } catch (error) { throw new ReadingToolUnavailable(error instanceof Error ? error.message : 'Invalid reading tool arguments.'); }
if(!adapter.readingAvailability(name,args).available)throw new ReadingToolUnavailable('This reading control is unavailable in the current view.');
const result=await adapter.performReadingTool(name,args,signal);
alive(signal);return result;
}
2 changes: 2 additions & 0 deletions viewer/scripts/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,5 @@
- The build fingerprint also includes `ohif-build/study-tools.patch`; apply it independently from the security source patch so existing caches remain reproducible. The pinned adapter export and neutral panel marker support native semantic tooling. Navigation regressions reject native no-ops, manual case switches, foreign fusion layers and cross-study cine.

- Continuation regressions submit exactly one original research request, preserve the unsent draft and cumulative coverage, and clear stale scope receipts. Scope regressions allow filtered state reads beside an unshared localizer, while presentation-change/manual takeover and native pane capture remain governed. Delivery markup must not show zero delivery alongside restored frame coverage.

- Reading-tool regressions cover OHIF array-backed brain/soft-tissue presets, unavailable presets before effects, deferred active-pane selection and explicit slice-jump targets. The renderer must preserve an active task after a known preflight rejection.
Loading
Loading