[AIR-Xie]

Summary

• add an opt-in permission guardrail that runs only after deterministic permission rules
• keep decision-provider credentials separate from normal model providers while reusing the provider settings UX
• support OpenAI Decisions, OpenRouter Decisions, OpenAI-compatible chat, and Ollama chat formats
• fail closed to ask on missing credentials, timeouts, provider errors, or malformed output
• keep explicit deny rules and bypass-immune checks ahead of all decision-model calls
• persist model-assisted permission markers, including model, preset, decision, and score, across app restarts
• route desktop /permissions to permission settings and document the project rule file and cross-platform shell names
• add a model-specific OpenAI Decisions evaluation path for SABER dev/test

Validation status

The only provider/model combination validated end to end is the hosted OpenAI Decisions API with gpt-6-luna. Manual acceptance covered provider discovery and connection testing, safety-net allow/ask behavior, invalid-key fallback, Bash and PowerShell explicit-deny precedence in bypass mode, bypass-immune writes, and transcript-marker persistence after a full DreamCoder restart.

Jev, d1-ultra, Claude, Ollama, and the other decision models have not been validated end to end in DreamCoder. Their presets and protocol adapters are implementation support, not a deployment-validation claim. Thresholds and AUROC values must not be transferred between models.

How to deploy Jev

Jev is a hosted OpenRouter Decisions model, not a local deployment. It requires an OpenRouter API key; an OpenAI API key cannot call it.

1. Obtain an OpenRouter API key and keep it out of the repository, logs, and transcripts.
2. Open Settings > Providers > Permission guardrail providers and add a provider.
3. Select the OpenRouter preset.
4. Use base URL https://openrouter.ai/api/alpha and API format OpenRouter Decisions.
5. Enter the OpenRouter key, fetch models, and select typesafe/jev-1.14.
6. Test the connection, save the provider, and make it the default decision provider if desired.
7. Open Settings > Permissions, enable the guardrail, and select the Jev provider/model.
8. Start with safetyNet or observation-only behavior. Do not allow Jev to turn ask into allow until it has a model-specific calibrated threshold.
9. Evaluate Jev on SABER dev with:

$env:OPENROUTER_API_KEY='<openrouter-key>'
bun experiments/system1-guardrail/eval/decisions.ts --model typesafe/jev-1.14 --split dev --view full --budget 1500

Use the generated dev run to select the threshold with the scoring workflow documented in experiments/system1-guardrail/eval/README.md. Freeze that threshold before running the test split, and report the test split once. This Jev deployment path is documented but still needs real-key end-to-end verification.

Safety boundaries

• disabled by default
• deterministic deny rules cannot be overridden
• bypass-immune checks cannot be overridden
• missing keys, timeouts, HTTP errors, and unreadable results return ask
• only verified loopback endpoints are classified as local/private
• API keys follow the existing provider-secret path and are not included in transcripts
• uncalibrated chat/local models cannot automatically change ask to allow

Request for feedback

1. Decision-provider UI: should decision providers remain a separate section under Providers with Permission settings referencing providerId and modelId, or should deployment and behavior be combined into one permission-specific workflow?
2. Hard guardrail rules: should DreamCoder add a structured editor for explicit Allow/Ask/Deny rules in .claude/settings.local.json, including separate Bash(...) and PowerShell(...) rules, or continue documenting manual JSON editing until rule ownership and merging semantics are settled?
3. Permission modes: should the first release consult decision models only as a safety net for bypass-mode allows, or should calibrated models also participate in normal/default permission flows through fewerPrompts? The current implementation supports both presets, but automatic ask to allow remains disabled until a model is calibrated.

Testing

• bun test --randomize --seed=1 for the permission guardrail, structured I/O, transcript history, and WebSocket test files: 67 passed
• cd desktop && bun run lint: passed
• desktop guardrail/provider/settings tests: passed
• full desktop Vitest run reached two unrelated load-sensitive timeouts; both failing files passed in isolation (3/3 and 32/32)
• Windows sidecar build: passed
• manual Windows acceptance: passed for OpenAI gpt-6-luna

Part of #146

[GoDiao]

Thanks for this, and for listing the open questions. Before going into the code, we want to talk about the direction, because it has changed on our side since we wrote #146.

#146 asks for a "safety net" preset in bypass mode and a "fewer prompts" preset in the default and accept-edits modes. You built what it says. We have since changed our minds, so this is about where we want to go now, not a mistake on your part.

Where we want to go

• The guardrail is auto mode. The deterministic rules run first, the calls they do not allow go to the decision model, and only what the model does not allow goes to the user. Turning the guardrail on means choosing auto mode, with no separate switch.
• Auto mode has to be built. It is not usable today: the CLI only accepts auto when the TRANSCRIPT_CLASSIFIER flag is on (src/types/permissions.ts), our builds never turn it on, and its judge, yoloClassifier.ts, is limited to internal users and has stub prompts. The desktop Permissions page lists four modes (Manual, Accept edits, Plan, Bypass permissions); auto would be the fifth.
• No guardrail in the other modes. Manual, Accept edits and Plan keep today's behaviour. Bypass means the user chose to skip prompts, so no model asks on top of it.
• No decision model configured: fall back to an LLM. Providers already have a fastModel setting. When no decision provider is chosen, auto mode should use the active provider's fastModel instead of a second chat-provider configuration.
• Decision providers stay a separate section under Providers, as you have them now: endpoint, key, model list, connection test.

What we would like your view on

We have some ideas for the settings, but you know this code better now, so we would rather hear how you would do it.

1. The Permissions page. One idea: when auto is selected, the page shows its settings under it (which decision provider and model, or the fastModel fallback; the preset; the privacy notice; a link to add a decision provider). How would you lay it out?
2. Rules. The desktop app has no view of the allow/ask/deny rules today, and "Always allow" in the permission dialog only adds a rule for the current session. Rules apply in every mode, including bypass for deny rules. Do you think the Permissions page should show and edit them, and if so, which file should it write to (user-level, or the project's settings.local.json), and should it be part of this PR or a separate one?
3. Auto with no usable model. If no decision provider is chosen and the active provider has no usable fastModel, should auto mode be unavailable, or selectable and asking for every call that the rules do not allow?
4. The preset. Thresholds depend on the model, as Permission guardrail: opt-in hosted decision model behind the rules #146's numbers show. Should users choose a preset, or should auto start with one fixed default per calibrated model?
5. yoloClassifier.ts. Remove it, or route it through the guardrail?
6. The fastModel fallback. How does it fit your current provider and preset code?

[AIR-Xie]

Update: model-assisted Auto permission mode (b1f4e59)

This update incorporates the latest acceptance feedback point by point.

Runtime and safety behavior

1. Auto is now a separate permission mode. The decision model runs only in Auto; Default, Accept Edits, Plan, and Bypass keep their existing semantics.
2. Hard rules always run first. Explicit workspace/user deny rules and bypass-immune checks cannot be overridden by a model. A deterministic deny ends evaluation before any decision-provider call.
3. Only unresolved confirmation requests reach the model. Auto can use either a dedicated decision provider or the active model provider's fast model. When no separate fast model is configured, it uses the active provider's selected model.
4. Failure is safe. Timeout, provider error, missing credentials, unsupported policy/model, and malformed output all fall back to interactive ask; they never become an automatic allow.
5. Transcript evidence persists. Model-assisted results preserve provider, model, source, decision, score when available, policy version, and reason.

Provider and desktop behavior

1. Manual model IDs are preserved. Fetch Models refreshes suggestions without replacing a manually entered decision model such as no-auto-decision-model with gpt-6-luna.
2. Fast-model selection is restored. Clearing a dedicated fast model means "use selected model"; it no longer creates a broken no-model state.
3. Permission prompts are serialized. Concurrent tool requests display one actionable confirmation card at a time instead of several overlapping cards.
4. Hard-deny presentation is structured. Workspace-rule denials show the blocked command in a copyable in-chat plain-text card.
5. The settings and protocol changes are covered in both English and Chinese. The changelog is updated under Unreleased.

Design decisions implemented

1. Decision-model deployment UI: connection details and secrets remain in the Providers page as independent decision-provider records. Auto settings reference a provider/model or the active provider's fast/current model; there is no second global active decision-provider state.
2. Hard guardrails: deterministic Allow/Ask/Deny configuration remains in .claude/settings.local.json, because the CLI/sidecar permission pipeline owns enforcement. Shell rules remain tool-specific, so cross-platform projects should include both Bash(...) and PowerShell(...) entries where applicable and start a new session after edits.
3. Where the model runs: model-assisted decisions are limited to Auto mode. Bypass does not silently gain model policy, and the model cannot weaken deterministic denials.

Validated and unvalidated decision models

• Validated end to end: OpenAI Decisions API with gpt-6-luna.
• Current calibrated policy: openai_decisions:gpt-6-luna, ask threshold 0.5, policy version 1.
• Not deployment-validated yet: Jev, d1-ultra, Claude/Haiku, OpenRouter-hosted alternatives, and Ollama/chat-based models. Their AUROC/thresholds must not inherit the gpt-6-luna calibration.

Jev deployment direction

1. Obtain an OpenRouter API key; an OpenAI API key cannot call Jev.
2. Add the OpenRouter decision-provider preset, using https://openrouter.ai/api/alpha, API format openrouter_decisions, and model typesafe/jev-1.14.
3. Test connectivity without placing the key in logs, transcripts, fixtures, or the repository.
4. Evaluate Jev independently on SABER: choose the threshold using the dev split only, then report the test split once.
5. Until that calibration and real deployment validation are complete, Jev remains fail-safe confirmation-only: unsupported policy/model evaluation returns ask rather than auto-allowing.

Verification

• Focused CLI/server guardrail tests: 104 passed, 0 failed.
• Touched desktop suites: 243 passed, 0 failed across 8 files.
• Stateful business-flow suite in its normal project order: 73 passed, 0 failed.
• Desktop TypeScript lint/typecheck: passed.
• Randomized touched server run: 285 passed; its 13 failures were confined to the pre-existing order-dependent shared state in business-flow, which passes 73/73 in its supported normal order.
• Manual acceptance: model-ID preservation, serialized confirmation cards, active-provider fast/current-model fallback, deterministic Bash/PowerShell hard deny in Auto, and persisted guardrail markers all passed.

[GoDiao]

Thanks for the update and for taking the direction on board.

The branch now conflicts with main, so CI only ran GitGuardian. main has moved on with #232, #235, #239 and #242; a local merge shows conflicts in:

• CHANGELOG.md
• desktop/src/components/chat/MessageList.tsx
• desktop/src/components/chat/PermissionDialog.tsx
• desktop/src/stores/chatStore.ts
• desktop/src/types/chat.ts
• src/server/session/state/types.ts
• src/server/session/translate.ts
• src/utils/sessionStorage.ts

Could you merge main into the branch and resolve them? #239 changed how messages reach a session mid-turn and how they are stored, so please check the session state and transcript changes against it rather than only taking one side. Keep the CHANGELOG entry under ## [Unreleased].

We'll review once CI is green.

[AIR-Xie]

Implemented the requested #146 update and resolved the main conflict.

1. Merged main at f7f1445 into this branch in merge commit b443fa0.
2. Resolved all eight reported conflicts:
   • CHANGELOG.md
   • desktop/src/components/chat/MessageList.tsx
   • desktop/src/components/chat/PermissionDialog.tsx
   • desktop/src/stores/chatStore.ts
   • desktop/src/types/chat.ts
   • src/server/session/state/types.ts
   • src/server/session/translate.ts
   • src/utils/sessionStorage.ts
3. Preserved the feat: sessions start and message each other #239/feat(kanban): add a project board shared by you and your sessions #242 semantics while retaining the permission guardrail:
   • a permission request can carry both guardrail and alwaysAsks;
   • the desktop still activates one permission prompt at a time, avoiding duplicate confirmation cards;
   • queued mid-turn commands remain loggable;
   • permission_guardrail_decision transcript markers remain loggable and survive history reload;
   • the translated server/session types preserve both fields.
4. Added cross-feature regression coverage for the combined request shape, desktop mirror handling, and transcript persistence.
5. Added the feature design at docs/features/permission-guardrail/tech-design.md and linked it from docs/README.md. It documents provider ownership, Auto-only execution, deny/bypass-immune precedence, fail-safe Ask fallback, privacy boundaries, and coexistence with feat: sessions start and message each other #239 session/transcript behavior.
6. Stabilized the cross-shell hard-deny regression test after Linux CI exposed the PowerShell parser cold-start exceeding Bun's default 5-second per-test limit. Bash and PowerShell are now separate parameterized cases, each still covering Auto and bypass, with a 10-second test-only timeout. Product behavior is unchanged (58e8f6a, 42c782e).
7. Verification:
   • focused server/Permission guardrail: opt-in hosted decision model behind the rules #146/feat: sessions start and message each other #239 tests: 153 passed;
   • focused desktop tests: 269 passed;
   • desktop TypeScript/Electron typecheck: passed;
   • final GitHub CI: Server tests, Desktop tests, Desktop typecheck, and GitGuardian all passed.

GitHub now reports the PR as MERGEABLE. The remaining BLOCKED state is review approval only.

[AIR-Xie]

CI annotation follow-up: the single Server tests error belonged to the superseded 58e8f6a run, where the PowerShell deny test exceeded the CI timeout. Commit 42c782e fixed that test and the latest run remains green. Commit e60011a also upgrades actions/checkout and actions/cache from v4 to v5, removing the three Node.js 20 deprecation warnings. Latest run 43140782736: Server tests 2656 passed, 2 skipped, 0 failed; Desktop tests, Desktop typecheck, and GitGuardian also passed. The only remaining annotations are three GitHub notices about the future ubuntu-latest migration; there are now 0 errors and 0 warnings.

[GoDiao]

Thanks for rebuilding this around Auto mode and for resolving the conflicts. The overall shape is what we asked for: Auto is its own mode, deny rules run before the model, every failure falls back to asking, the other four modes never call the model, and #235's secret reads stay with the user. Lint and the touched tests pass on our side too (235 server/CLI tests in random order, 311 desktop tests), and business-flow fails the same 14 tests under --randomize on main, so that is not from this PR.

We reviewed it along the seven dimensions in AGENTS.md. Below are the changes we need, sorted by tier.

Must fix

1. An explicit ask rule can be approved by the model.
In hasPermissionsToUseTool (src/utils/permissions/permissions.ts, the mode === 'auto' branch), only a safetyCheck with classifierApprovable: false and tools with requiresUserInteraction skip the model. A user's own ask rule (step 1b for a whole tool, step 1f for content rules such as Bash(git push:*)) returns ask with decisionReason.type === 'rule' and goes to the model like any other ask. With Bash(git push:*) set to ask and Auto selected, a decision of allow runs git push origin main without a prompt, while bypass mode still asks for it. The settings text (settings.permissions.rulesDescription) also says these rules take precedence over the guardrail.
Please return early for ask rules as well, and add a test that fails without it.

2. The CLI reads providers.json and the decision provider's key itself.
readDecisionProvider in permissionGuardrail.ts reads providers.json from disk on every decision, imports normalizeProvidersIndex from src/server/, and requestAskProbability sends the key from the CLI process. On main the CLI never reads providers.json: the sidecar resolves the active provider and passes its settings to the CLI as environment variables when it starts the CLI (buildProviderManagedEnv in providerRuntimeEnv.ts). Here the file format is now read in two places, and a read error is swallowed, so Auto quietly falls back to asking with only "Decision provider or model is unavailable".
Please follow the main pattern: when Auto uses a decision provider, have the sidecar put the provider's endpoint, API format, model and key into the CLI environment alongside the variables it already sets, and have the CLI read only those. Then the import from src/server/ and the file read go away. A change to the Auto settings can take effect when the CLI restarts, as a provider change does today.

Should fix

3. Long input can hide the tool call from the decision model.
renderInput joins the user request, the tool call and the working directory, and clipMiddle keeps the first 70% and last 30% of 1500 characters. When the user's message is long, the tool call in the middle is cut entirely, and the model still returns a score. Please fall back to ask when the tool call itself does not fit, and if anything is shortened, shorten only the user request.

4. Protected paths can be approved by the model, and always_asks is lost.
Writes to Claude config files and other sensitive files return safetyCheck with classifierApprovable: true (filesystem.ts), so in Auto the model can approve edits to .claude/settings.json or hooks. These paths still ask in bypass mode. When the model answers ask, { ...result, decisionReason } replaces the safetyCheck reason, so structuredIO.ts no longer sets always_asks and the dialog offers "Always allow" for a protected path. Please keep every safetyCheck with the user in Auto, and keep the original reason when the model answers ask.

5. Background agents get a prompt instead of a denial.
The Auto branch returns directly, both in the early returns and after a model ask, so it never reaches the shouldAvoidPermissionPrompts block further down (permission hooks, then deny). In Auto, a background agent that reads .env or gets a model ask sends a permission request instead of being denied. Please let these cases fall through to that block.

6. Switching to Auto is not reported to the server.
print.ts still has newMode === (feature('TRANSCRIPT_CLASSIFIER') && 'auto') in the permission-mode listener. The flag is off in our builds, so a switch to Auto never emits system/status and never calls savePermissionMode. The server's effective mode stays on the old mode, and a resumed session goes back to it. Please compare with 'auto' directly and add a test.

7. The settings page names a different model than the one Auto uses.
With no fastModel, the runtime uses the session's selected model (fastModel ?? selectedModel in buildProviderManagedEnv, as on main), but the Auto settings show fastModel ?? models[0]. If the session runs on a model other than the first, the page and the privacy confirmation name the wrong one. Please show the model the runtime uses, from one place.

8. Decision provider requests skip the network settings.
discoverDecisionModels uses a bare fetch with no timeout, and testDecisionProviderConfig and the CLI request do not use the manual proxy. The provider code already has this in fetchUpstreamModels / fetchWithTimeout with getProxyFetchOptions (src/server/upstream/client.ts). Please reuse that, so "Fetch models" and "Test" work for users behind a proxy.

9. Desktop.

• DecisionProviderSettingsSection calls useDecisionProviderStore() without a selector. Please read each field with its own selector, as the rest of the PR does.
• DecisionProviderFormModal repeats the model list (add, remove, merge with discovered models) and the connectivity line from ProviderFormModal. Please move the shared parts into one component or hook that both forms use.

10. The rule-denied card.
We're fine with keeping it in this PR, with two fixes:

• The description always points to .claude/settings.local.json, but a deny rule can also come from the user or policy settings. Please word it so it holds for any source, or name the actual source.
• It recognises a denial by matching the English message text (/Permission to use .+ has been denied\./). Please use something structured that travels with the result instead, so the card doesn't silently disappear when the message changes.

11. Tests and errors.

• Removing the two early returns in the Auto branch (the non-approvable safetyCheck and requiresUserInteraction) leaves all 14 guardrail tests passing. Replacing evaluateFastModel(...) with return null leaves those and provider-runtime-env passing. Please add tests that fail in both cases: a non-approvable safety check and an interactive tool never reach the model; and the fast-model path returns allow, falls back to ask when sideQuery throws, and makes no call when it is not configured.
• The four catch {} blocks and the !response.ok branch in permissionGuardrail.ts drop the cause. Please log the provider, model and error (logForDebugging or logError).

Notes

• PR description: it still describes the first version (safetyNet / fewerPrompts presets, enabling the guardrail, a default decision provider, the bypass question). Please rewrite it to match the current code.
• Design doc: docs/features/permission-guardrail/tech-design.md says the Auto settings include a link to provider configuration; there is none. Please add the link or drop the sentence.
• Leftovers:
  • normalizeSavedDecisionProvider repairs concatenated model IDs and strips activeDecisionProviderId. That data only came from earlier commits of this branch, which were never released, so this code can go.
  • DEFAULT_AUTO_PERMISSION_SETTINGS in src/types/permissionGuardrail.ts is not used.
  • The i18n key settings.decisionProviders.modelsPlaceholder is not used.
• yoloClassifier.ts and the TRANSCRIPT_CLASSIFIER flag: leave them as they are in this PR. We'll remove them separately.

Thanks again. This is a big piece of work, and it's close.