Skip to content

feat(getstream): move StreamEdge to the Rust-based stream-py 7.0 - #1195

Open
dangusev wants to merge 17 commits into
mainfrom
feat/stream-py-rust-upgrade
Open

dangusev wants to merge 17 commits into
mainfrom
feat/stream-py-rust-upgrade

Conversation

@dangusev

@dangusev dangusev commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Why

stream-py 7.0.0 replaces its aiortc transport with the Rust Video SDK (GetStream/stream-py#303). Vision Agents pinned getstream>=4.1.0,<5, so it stayed on the old transport. The new SDK changes the RTC surface that StreamEdge uses:

  • track_added gives a RemoteTrack with decoded I420 frames instead of an aiortc track on the subscriber peer connection. A RemoteTrack has no id: the SDK keeps one per session and track type, and dropping the last reference unsubscribes it.
  • The SDK attaches the participant and the track type to each track, so TrackResolver (which matched WebRTC tracks to SFU TrackPublished events) has nothing to do.
  • SFU events arrive as typed payloads (rtc.TrackPublished, rtc.ParticipantLeft, ...) instead of protobuf messages, so the generated sfu_events wrappers and the protobuf conversion in EventManager get no input.
  • Tracks and SFU events come on separate streams, so a track_added can arrive after its participant_left.
  • Camera mute gives track_unpublished, and the same RemoteTrack resumes after unmute.
  • The SDK sends participant_joined and track_added for participants already in the call, so republish_tracks() after join is not needed.
  • The SDK decodes only VP8 and VP9 video.
  • AudioStreamTrack no longer paces recv() by default. The openai and inworld Realtime plugins send it on their own aiortc connection, which needs one frame every 20 ms.
  • StreamAPIException is renamed to StreamApiException.

The branch also fixes the agent shutdown on call end. The CallEndedEvent handler started Agent.close(), and the close() from join() returned at once because a close was already running. join() then exited while components were still closing, and asyncio.run shut down in the middle of their cleanup. With a Realtime LLM this logged aclose(): asynchronous generator is already running.

User-facing breaks (install only on macOS arm64 and Linux, removed sfu_events, new participant and track id formats, no video from H.264-only publishers, VP9 publishing, AudioStreamTrack pacing) are in CHANGELOG.md.

Changes

  • Require getstream[webrtc,telemetry]>=7.0.0 in vision-agents and vision-agents-plugins-getstream.
  • StreamEdge maps the SDK events to the core edge events directly. It keeps each RemoteTrack until its participant leaves, keeps unpublished tracks for unmute, and ignores tracks of sessions that left.
  • New RemoteVideoTrack: an aiortc VideoStreamTrack over RemoteTrack.video_frames() that gives yuv420p av.VideoFrames with RTP timestamps. add_track_subscriber() returns a new one per call.
  • Remove sfu_events.py, its generator and docs, _track_resolver.py, and the protobuf branch in EventManager.
  • open_demo() asks the browser for VP8 (was H.264).
  • The openai and inworld Realtime plugins pass pace=True to AudioStreamTrack.
  • Agent.close(): a concurrent call waits for the running close. The call-end close runs in its own task, so events.shutdown() does not wait on the handler that waits for the close.
  • Tests: StreamEdge integration tests against a real call (join, leave, unpublish and republish, late track_added, video frames, two subscribers), unit tests for RemoteVideoTrack, and an Agent test for join() after a call-end close.

🤖 Generated with Claude Code

@dangusev dangusev changed the title Feat/stream py rust upgrade feat(getstream): move StreamEdge to the Rust-based stream-py 7.0 Oct 23, 2026
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 497b76c and 581fcd3.

📒 Files selected for processing: 1
  • CHANGELOG.md
🚧 Files skipped from review as they are similar to previous changes: 1
  • CHANGELOG.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Walkthrough

The GetStream dependency now requires version 7.0.0 or later. StreamEdge handles native RTC participant and track events, uses session-based IDs, and adapts remote video frames. The change removes protobuf event wrappers, raw-protobuf conversion, and TrackResolver. The OpenAI and Inworld audio tracks now enable pacing. Agent call-ended handling schedules shutdown separately, and concurrent close calls wait for the active close. Tests add transport integration and frame-conversion coverage.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 581fc

No identified issue currently blocks merging after normal checks.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments: 1
CHANGELOG.md: 1

5-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add PR references to the new changelog entries.

The three new Breaking Changes entries have no inline PR number. Add (#1195) to each heading, as the other entries do.

As per coding guidelines: "Reference PR numbers inline, e.g. (#374)."

Also applies to: 14-14, 18-18

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 902db86 and 497b76c.

⛔ Files ignored due to path filters: 1
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing: 21
  • CHANGELOG.md
  • agents-core/pyproject.toml
  • agents-core/vision_agents/core/agents/agents.py
  • agents-core/vision_agents/core/events/manager.py
  • dev.py
  • plugins/getstream/PROTOBUF_GENERATION.md
  • plugins/getstream/_generate_sfu_events.py
  • plugins/getstream/pyproject.toml
  • plugins/getstream/tests/test_remote_video_track.py
  • plugins/getstream/tests/test_sfu_events.py
  • plugins/getstream/tests/test_stream_edge_transport.py
  • plugins/getstream/tests/test_track_resolver.py
  • plugins/getstream/vision_agents/plugins/getstream/_track_resolver.py
  • plugins/getstream/vision_agents/plugins/getstream/remote_video_track.py
  • plugins/getstream/vision_agents/plugins/getstream/sfu_events.py
  • plugins/getstream/vision_agents/plugins/getstream/stream_conversation.py
  • plugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py
  • plugins/inworld/vision_agents/plugins/inworld/rtc_manager.py
  • plugins/openai/pyproject.toml
  • plugins/openai/vision_agents/plugins/openai/rtc_manager.py
  • tests/test_agents/test_agents.py
💤 Files with no reviewable changes: 6
  • plugins/getstream/PROTOBUF_GENERATION.md
  • plugins/getstream/tests/test_sfu_events.py
  • plugins/getstream/vision_agents/plugins/getstream/sfu_events.py
  • plugins/getstream/tests/test_track_resolver.py
  • plugins/getstream/_generate_sfu_events.py
  • plugins/getstream/vision_agents/plugins/getstream/_track_resolver.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread CHANGELOG.md Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant