Repository navigation
Keep the H.264 PPS id readable in E2EE keyframes - #2181
Conversation
Chrome reads pic_parameter_set_id from an H.264 IDR slice before the decrypt transform runs, and drops the keyframe if the id names an unknown PPS. Encoders such as Intel Quick Sync code slice_type in 7 bits, so the id lies past the 2 unencrypted slice bytes and gets encrypted. The sender now encrypts again with a fresh IV, up to 16 times, until the sent bytes still read as the true PPS id. The receiver takes the IV from the frame trailer, so the wire format and receivers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: fb14286 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
size-limit report 📦
|
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
writeRbsp takes about 95% of an encryption attempt on a large keyframe. The retry loop escaped the whole frame on each attempt, but the PPS id check needs only the first 16 bytes. Escape those for the check, and escape the whole frame once, after the loop. A retry then costs only the AES-GCM encryption. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1egoman
left a comment
There was a problem hiding this comment.
👍 Would it be worth adding some sort of e2e test for this?
|
Thanks for the quick fix in #2181. We tested it before release on a build of the
We have production participants on Intel laptops with E2EE, so this fix matters to us. Is there an expected date for the 2.24.0 release (#2158)? |
#327) Native senders that use the frame cryptor have the same E2EE H.264 bug as the JS SDK ([livekit/client-sdk-js#2180](livekit/client-sdk-js#2180)). With some encoders, the receiver drops about half of the keyframes, and video stalls for seconds. ## Cause `encryptFrame` leaves `payload_start_offset + 2` bytes in cleartext: the NAL header and one byte of the slice header. A receiver reads `pic_parameter_set_id` from an IDR slice before the decrypt transform runs (`VideoRtpDepacketizerH264` → `PpsParser::ParseSliceHeader`, then the PPS lookup in `H26xPacketBuffer` / `H264SpsPpsTracker`). An unknown id drops the keyframe and requests a new one. - VideoToolbox codes `slice_type` 2 in 3 bits. The PPS id is in the cleartext byte. - Intel Quick Sync codes `slice_type` 7 in 7 bits. The PPS id is the first encrypted bit, so it is random. ## Change Same approach as [livekit/client-sdk-js#2181](livekit/client-sdk-js#2181). The receiver takes the IV from the frame trailer, so the sender can choose any fresh IV: 1. For an H.264 IDR slice, parse the PPS id from the plaintext slice. 2. Encrypt the frame. 3. Escape only the first 16 bytes of the ciphertext, and parse the PPS id from the 2 cleartext slice bytes followed by those bytes. `WriteRbsp` works byte by byte, so this matches the start of the escaped frame. 4. If the ids differ, encrypt again with a new IV from `makeIv()`, at most 16 times. Each call advances the per-SSRC counter, so no IV is reused with the same key. 5. Escape the whole frame once, after the loop. A Quick Sync keyframe takes 2 encryptions on average. Delta frames and other codecs need no extra work. `ParseH264IdrPpsId()` is exported next to `IsKnownSifPayload()` so that tests can call it. It reuses the receiver's `PpsParser::ParseSliceHeader`. **Wire format is unchanged.** Existing JS and native receivers decrypt these frames as before. #319 widened the cleartext prefix instead and was closed in favor of a wire-compatible fix. This is that fix. ## Tests New `api/crypto/h264_pps_id_unittest.cc` in `rtc_unittests`: - `ParseH264IdrPpsId.*`: the same vectors as the JS PR (Quick Sync, VideoToolbox, PPS id 1, non-IDR, truncated). - `H264PpsIdTransformTest.KeepsQuickSyncPpsIdReadable`: 32 Quick Sync keyframes go through a real sender transformer and a real receiver transformer. Each test reads the PPS id from the encrypted frame the way the depacketizer does, and checks that decryption returns the original frame. - VideoToolbox keyframe and delta frame round trips. ``` gn gen out/test # rtc_include_tests = true autoninja -C out/test rtc_unittests ./out/test/rtc_unittests --gtest_filter='ParseH264IdrPpsId.*:H264PpsIdTransformTest.*:Sif*:FrameCryptor.*:DataPacketCryptor.*:KeyProvider.*' ``` Result: 22 of 22 passed (macOS arm64). With the retry disabled (`kMaxPpsIdEncryptAttempts = 1`), the Quick Sync test fails on 15 of 32 keyframes, so the test covers the bug. ## Not covered - **Receiver reads the IV before RBSP unescaping (existing bug, not fixed here).** The sender escapes the IV and trailer along with the ciphertext, but `decryptFrame` reads the IV from the escaped tail. If an emulation-prevention byte lands in the IV, decryption fails. Example: SSRC `1234` and RTP timestamp `96000` give `… 77 00 00 03 01 …`. The new tests use an SSRC and timestamps without zero bytes, with a comment that explains why. A follow-up can unescape before reading the IV. That change is backward compatible. - H.265 is unchanged. In this tree, receivers do not read `slice_pic_parameter_set_id` before the transform. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With E2EE and H.264, Chrome subscribers drop keyframes from publishers that use Intel Quick Sync, and video stalls for seconds. The e2ee worker leaves only the first 2 bytes of the slice NALU unencrypted. Chrome reads
pic_parameter_set_idfrom an IDR slice before the decrypt transform runs. Quick Sync codesslice_typein 7 bits, so the PPS id is the first encrypted bit. The receiver reads a random id and drops the frame. In #2180, 48 of 94 Quick Sync keyframes read as the correct PPS id 0, and one run stalled for 21 s. Fixes #2180.Change
The sender keeps the encrypted bits readable as the true PPS id. The receiver takes the IV from the frame trailer, so the sender can pick any fresh IV, and receivers need no change.
parseH264IdrPpsId()readsfirst_mb_in_slice,slice_typeandpic_parameter_set_idafter RBSP unescaping, the way Chrome does.makeIV, up toMAX_PPS_ID_ENCRYPT_ATTEMPTS(16).writeRbspescapes the whole frame once, after the loop. A retry costs only the AES-GCM encryption.Delta frames and VideoToolbox keyframes, where the id sits in the plaintext byte, need no extra encryption. For Quick Sync, one bit must match, so a keyframe takes 2 encryptions on average.
A longer plaintext prefix that ends after
pic_parameter_set_idalso fixes the bug, but I dropped it because it changes the wire format. Old JS receivers and the native SDKs (payload_start_offset + 2inwebrtc-sdk/webrtc) then fail to decrypt.Failure case
A slice can have more than one PPS id bit past the plaintext byte, for example with a non-zero PPS id. Each extra bit halves the chance that an attempt matches. If all 16 attempts miss, the frame goes out as the last attempt produced it. That is the behavior without this change.
Verified
npx vitest run src/e2ee/workerpasses, 80 tests. The newparseH264IdrPpsIdtests use the Quick Sync and VideoToolbox bytes from E2EE with H.264: encrypted pic_parameter_set_id makes Chrome drop keyframes from some encoders #2180, a non-zero id, a non-IDR slice and a truncated slice.FrameCryptortest encrypts 16 Quick Sync keyframes. It checks that each one reads as PPS id 0 and decrypts to the original frame. WithMAX_PPS_ID_ENCRYPT_ATTEMPTSset to 1, the test fails on the first frame that reads as a non-zero PPS id.npx tsc --noEmit, prettier and eslint pass. CItestpasses.Not done yet
slice_pic_parameter_set_idis always encrypted. It is not known whether Chrome reads it before the transform.webrtc-sdk/webrtchave the same bug. That fix is a separate change in that repo.🤖 Generated with Claude Code