Skip to content

feat: call an agent from its 1:1 DM (desktop and iOS) - #252

Merged
RossHartmann merged 9 commits into
mainfrom
feat/agent-dm-calls
Oct 23, 2026
Merged

RossHartmann merged 9 commits into
mainfrom
feat/agent-dm-calls

Conversation

@RossHartmann

@RossHartmann RossHartmann commented Oct 23, 2026 •

Copy link
Copy Markdown

Summary

Item 5e (desktop and iOS): call an agent from its 1:1 DM. This is a proper voice call with the agent through its host's voice core, the same as the host's phone and web calls. It is not a huddle.

  • Call button. It sits beside the sessions picker in a 1:1 agent DM header and shows for known agents only, not for agents managed on this device (they have no voice host). It stays hidden until the local agent inventory has loaded, so it never flashes for a local agent. It starts the call with the agent key and the DM channel id. The host answers only after proving that the signer is the verified requester of that private 1:1 DM with the agent. Channels, group DMs and guests are refused server-side.
  • agent_voice_request (Rust). One native command that:
    • signs NIP-98 as the signed-in user (exact URL with query, method, SHA-256 of the exact body) and sends X-Buzz-Community;
    • reaches only the host's call routes under <guest route origin>/api/buzz-voice/desktop: allowlisted paths, https, 35 s timeout, no redirects;
    • parses the request URL once, checks its normalised path and query against the allowed route, then signs and sends that same Url. Ask ids reject percent escapes of ., / and \ (%2e, %2f, %5c) and malformed escapes;
    • caps answers at 2 MiB, on the declared Content-Length first and then while reading chunks;
    • honours BUZZ_DESKTOP_AGENT_VOICE_URL, and the shared HTTP-loopback allowance in acceptable_base_url, in debug builds only;
    • returns {status, body}.
      The key never enters the webview.
  • AgentCallController. A port of the host's web call controller:
    • mic with echo cancellation, an oai-events data channel, the time limit and a 5 s disconnect grace;
    • mute, and stop the agent's work;
    • hang-up never cancels the agent's work, which still lands in the DM;
    • it follows the host's call screen feed by long-poll (wait 20 s, backoff from 1 s to 15 s), and after hang-up for the recap (at most 90 s);
    • a call still connecting after 20 s ends as media_failed and closes the mic;
    • stale attempts are fenced: a superseded mic request stops only its own stream, and a call hung up while the offer is made never reaches the host.
  • One audio owner. A call never runs beside a huddle:
    • The call refuses while a huddle is active or starting. The audio-owning HuddleProvider registers a probe, and the native huddle state is checked as well, both before and after the mic opens. If the huddle state can't be read, the call fails closed: it refuses and says why.
    • startHuddle and joinHuddle refuse while a call is busy, with "End your call with X before you start/join a huddle." The huddle is refused and the call is never ended for it, which is the safer choice: a click elsewhere never drops a call in progress.
  • AgentCallDock. Mounted once in AppShell. It shows the timer, mute, hang up and stop, the call's asks (working, done, now line, results, https links with their host shown beside the label), the open question's option taps, per-ask cancel and archive (feed v2) and the recap.
  • Community switch. Hangs up and clears the call (resetCommunityState).
  • Mic usage string. Now mentions calls with agents. tauri.conf.json is untouched (minimum macOS 13.0 from fix(desktop): raise the macOS minimum to 13.0 #251).
  • Fork inventory. Entries for desktop and iOS, listing the hooks in HuddleContext.tsx, huddle_session.dart, huddle.dart, mobile_huddle_controller.dart and HuddleMediaPlugin.swift.

Server side (host) is live on the Kiingo side.

iOS

  • Call action. A native navigation action and a Flutter button in the 1:1 agent DM header open the call screen. They show when dmAgent is set, and on iOS only (the native media is iOS; Android has no button).
  • AgentCallPlugin.swift (buzz/agent_call). Mirrors the host's phone-app media layer:
    • stasel/WebRTC 154.0.0, pinned exactly (revision 0c0ad84d, binary checksum a2bcdda9…), as an SPM package in Runner;
    • an outgoing CallKit call from a provider created at engine start (kept out of Recents, with the system's mute and end buttons). The caller name is trimmed, stripped of control characters and capped at 60 characters;
    • WebRTC in manual-audio mode, activated by CallKit (playAndRecord, voiceChat), with an in-app fallback when CallKit refuses. The call screen then shows a banner saying the call isn't on the iPhone's call screen;
    • handling for interruption, route change and media-services reset;
    • peer-connection and data-channel events (including the 5 s disconnect timer) act only for the current peer connection;
    • VoiceAudioSessionOwner marks the call's hold on the AVAudioSession. While it is set, HuddleMediaPlugin never prepares, reactivates (on interruption end) or deactivates the session.
  • Info.plist. UIBackgroundModes now has audio and voip, and the microphone string mentions calls. Huddles still leave on background as before.
  • Dart features/agent_call.
    • A NIP-98 client for <guest route origin>/api/buzz-voice/ios. It reads answers with the hosted API's 2 MiB cap (guestAccessMaxResponseBytes), on the declared length and while streaming.
    • The feed model and a controller with the same feed long-poll, option taps, cancel, archive and recap as desktop. It refuses while a huddle is in session, at the tap and again once the native call is up. It ends a call still connecting 20 s after the native call is up as media_failed, and hangs up on a community switch.
    • While busy, the call holds agentCallAudioBusyProvider (shared/huddle/agent_call_audio.dart). HuddleSessionNotifier.join refuses on it, and so does MobileHuddleController._start, before announcing a huddle. The call also refuses while a huddle start is under way (huddleStartingProvider, mirrored from MobileHuddleController), so a huddle is never announced and then rolled back.
    • The call screen.

voip background mode without PushKit (App Review)

Buzz never receives VoIP pushes and does not register with PushKit. voip is declared because an agent call is a user-initiated outgoing CallKit call carried over WebRTC. The background mode keeps that live call's network connection and audio running while the phone is locked or another app is in front, like any outgoing VoIP call. Without it, iOS would suspend the call when the screen locks. The app runs no work in the background beyond the call the user placed, and the call ends when they hang up or the host ends it.

Testing

What has run:

  • Locally (static only): tsc --noEmit, biome, dart format, flutter analyze (changed paths, clean), cargo fmt, check-kiingo-fork-boundary.
  • Pre-push hooks: branch-skew, desktop-source, desktop-tauri-checks (clippy and check), mobile-checks and file-size.
  • CI on Kiingo/buzz (static only): Desktop JS (lint, format, build, no unit tests), Desktop Tauri core (clippy and check only), Mobile Checks and Mobile Android build, Dead Token Reference Guard, CodeQL.
  • Build gate on the dev Mac: two builds on 368e892, the fixes in this round. Both are compile and package checks only; nothing was launched.
    • iOS: unsigned device build (flutter build ios --config-only --no-codesign, then xcodebuild … -sdk iphoneos -configuration Release CODE_SIGNING_ALLOWED=NO). BUILD SUCCEEDED.
    • macOS: .app bundle at the shipped 13.0 minimum (MACOSX_DEPLOYMENT_TARGET=13.0 CARGO_PROFILE_RELEASE_STRIP=none pnpm tauri build --bundles app --no-sign). Built Buzz.app; its Info.plist has LSMinimumSystemVersion 13.0 and the new microphone string.

What has not run:

  • No unit test has executed anywhere. Kiingo/buzz CI skips them:
    • _ci-desktop.yml:60-62: desktop unit tests are if: github.repository != 'Kiingo/buzz'.
    • _ci-desktop.yml:87-88: the Tauri matrix runs clippy and check only, with no desktop-tauri-test.
    • _ci-clients.yml:74-75: the mobile matrix has no Flutter test shards.
    • _ci-clients.yml:163: the Mobile Swift job carries the same repository guard.
    • The Desktop Build (macOS) and Desktop macOS Domain jobs carry the same guard in ci.yml.
    • Per policy, nothing runs on the dev Mac.
  • Spot runs after merge, not before. Kiingo/buzz policy is that tests never gate a merge (no tests in CI or on the dev Mac). These suites run on post-deploy Spot from the first buzz-desktop release containing this PR, and anything red is repaired forward:
    • desktop node:test: agentCallApi, agentCallController and agentCallFeed .test.mjs;
    • Tauri Rust: extensions::agent_voice tests, plus commands::guest_access base_urls_must_be_https_or_loopback_http;
    • Flutter: test/features/agent_call/agent_call_test.dart.
  • Not done yet:
    • a live call check with a fixture identity;
    • phone behaviour on a device (CallKit, lock screen, background audio, refused huddles), which comes after merge via TestFlight.

🤖 Generated with Claude Code

Ross Hartmann and others added 4 commits October 22, 2026 22:54
A Call button in a 1:1 agent DM header places a proper voice call with the
agent through its host's voice core (WebRTC to the host's realtime voice
provider, the host answering the SDP offer), not a huddle. The host answers
only after proving the signer is the verified requester of that private 1:1
DM with the agent.

- agent_voice_request: one native command that signs NIP-98 as the signed-in
  user (exact URL, method, body hash) with X-Buzz-Community and reaches only
  the host's call routes (allowlisted, https, size and time caps).
- AgentCallController: a port of the host's web call controller (mic, offer,
  data channel, time limit, disconnect grace, mute, stop, hang-up that never
  cancels work) following the host's call screen feed by long-poll.
- AgentCallDock (mounted once in AppShell): timer, mute, hang up, stop, the
  call's asks, the open question's option taps, cancel/archive and the recap.
- A community switch hangs up and clears the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
A Call action in a 1:1 agent DM header places a proper voice call with the
agent through its host's voice core, the same as desktop: the host answers
the WebRTC offer only after proving the signer is the verified requester of
that private 1:1 DM with the agent.

- AgentCallPlugin (buzz/agent_call): stasel/WebRTC 154.0.0 pinned exactly,
  an outgoing CallKit call from a provider created at engine start (kept out
  of Recents, system mute and end), WebRTC in manual-audio mode activated by
  CallKit (playAndRecord, voiceChat), an in-app fallback when CallKit
  refuses, and interruption, route-change and media-reset handling.
- Info.plist: UIBackgroundModes audio and voip; the microphone string
  mentions calls.
- Dart: a NIP-98 client for <guest route origin>/api/buzz-voice/ios, the
  call screen feed model, a controller that follows the feed long-poll
  (asks, option taps, cancel and archive, recap), refuses while a huddle
  holds the audio and hangs up on a community switch, and the call screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
@RossHartmann RossHartmann changed the title feat(desktop): call an agent from its 1:1 DM feat: call an agent from its 1:1 DM (desktop and iOS) Oct 23, 2026
Ross Hartmann and others added 5 commits October 23, 2026 08:40
…caps

Review fixes for 1:1 agent calls on desktop and iOS.

- One audio owner per platform. Desktop: a call refuses while a huddle
  is active or starting (probe from the audio-owning HuddleProvider,
  then the native huddle state, before and after the mic opens);
  startHuddle/joinHuddle refuse while a call is busy (the huddle is
  refused, the call is never ended for it). iOS: the call holds
  agentCallAudioBusyProvider; HuddleSessionNotifier.join and
  MobileHuddleController._start refuse on it; the call re-checks the
  huddle after the native call is up; HuddleMediaPlugin never prepares,
  reactivates or deactivates the AVAudioSession while a call holds it.
- Desktop native request: parse the URL once, check its normalised
  path, sign and send that same URL; ask ids reject escapes of dot,
  slash and backslash; answers capped by declared length and while
  read; the voice URL override and the HTTP-loopback allowance are
  debug-only.
- Connect timeout (20 s) ends a still-connecting call as media_failed
  and closes the mic, on both platforms.
- Desktop: a superseded mic request stops only its own stream; a call
  hung up while the offer is made never reaches the host.
- iOS: peer and data-channel events act only for the current peer
  connection; CallKit name trimmed and capped; response cap shared
  with the hosted API client; in-call banner without CallKit; the Call
  button shows on iOS only.
- Feed links show their host; no desktop Call button for agents
  managed on this device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
base_urls_must_be_https_or_loopback_http now expects loopback HTTP to be
accepted exactly when debug assertions are on, so it holds in release
test runs too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
… starts

Review follow-ups for 1:1 agent calls.

- Desktop: when the native huddle state can't be read, the call refuses
  with a clear reason instead of assuming no huddle.
- iOS: a call also refuses while a huddle start is under way
  (huddleStartingProvider, mirrored from MobileHuddleController), so a
  huddle is never announced and then rolled back for a call.
- Desktop: the Call button waits for the local agent inventory, so it
  never flashes for an agent managed on this device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
MobileHuddleController.join supersedes an in-flight start by bumping the
generation, so the start's finally skipped its reset and the controller
(and huddleStartingProvider, which agent calls read) stayed "starting"
until the next leave, end, background or community switch. join now
clears it like the other superseding paths. Widget test covers a join
during an in-flight start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
Signed-off-by: Ross Hartmann <rosshartmann@RosssMacStudio.attlocal.net>
@RossHartmann
RossHartmann marked this pull request as ready for review October 23, 2026 20:26
@RossHartmann
RossHartmann merged commit eb273e3 into main Oct 23, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant