Add support for packet trailer features by chenosaurus · Pull Request #1886 · livekit/client-sdk-js

chenosaurus · 2026-04-14T23:43:02Z

Add support for packet trailer features (user timestamp, frame ID).
Add new web worker for packet trailer parsing.
update example/demo to have options to enable packet trailer & render the parsed metadata.
Unit tests for packet trailer parsing.

…ryptor for performance

changeset-bot · 2026-04-14T23:43:09Z

⚠️ No Changeset found

Latest commit: 49942ef

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2026-04-15T06:22:50Z

size-limit report 📦

Path	Size
dist/livekit-client.esm.mjs	97.27 KB (+1.67% 🔺)
dist/livekit-client.umd.js	106.15 KB (+1.37% 🔺)

lukasIO · 2026-04-16T13:32:19Z

      // Disabling it for Chrome based browsers until the API has stabilized
-      !isChromiumBased()
+      !isChromiumBased() &&
+      !ptPreProcessed


I'm not sure I follow why this is being checked for here, can you elaborate on the intention behind it?

removed, it was left over.

lukasIO · 2026-04-16T13:34:02Z

@@ -0,0 +1,198 @@
+export const PACKET_TRAILER_MAGIC = Uint8Array.from([


have you considered signalling packet-trailer presence as part of the publication info instead of relying on checking every frame on every publication for the magic bytes?

There is already support on track publication to declare which packet trailer features are available on the track. It's exposed to the subscriber in TrackInfo. I have updated all place to check TrackInfo before attaching transformer.

lukasIO · 2026-04-16T13:37:06Z

+    if (typeof metadata.rtpTimestamp === 'number') {
+      return metadata.rtpTimestamp;
+    }
+    if (typeof metadata.timestamp === 'number') {


what is this referring to? I don't see a timestamp value on getMetadata here

lukasIO · 2026-04-16T13:39:47Z

is it intentional that this file lives in the /e2ee folder? 👀

no, that's a mistake. Moved.

lukasIO · 2026-04-16T13:59:42Z


 export const version = v;
-export const protocolVersion = 16;
+export const protocolVersion = 17;


The related protocol PR talks about stripping the trailer on the SFU side, but I don't think bumping protocolVersion is enough: what happens if a client with protocolVersion 17 doesn't have the trailer worker set?

^ This is probably my biggest question as well.

It sort of sounds like you want a "feature" you can communicate support of unrelated to a protocol / client version. I recently wrote a note about this in the context of the in flight rpc protocol updates here: #1832 (comment)

Keeping in mind we haven't discussed mechanically how this pattern would work yet (ie, new fields in ParticipantInfo, something in the JoinResponse, etc?), I think if you did something like that then the SFU could know if packet trailer processing is enabled on the client independent of the protocol version and opt to strip or not strip conditionally.

Thoughts on that proposal?

The packet trailer transform is lightweight enough to run on the main thread, we can as a backup attach the transform automatically (when worker is not passed in) when we see any tracks that declared packet trailer features in its TrackInfo. That would always work and avoid having to change any signaling to the SFU.

The packet trailer transform is lightweight enough to run on the main thread

If this is the case, is there a reason why it shouldn't always be run on the main thread then rather than having the optional worker?

see response on a later comment, it's possible to attach the packet trailer transformer directly to the transceiver in the main thread to avoid the issue of worker not being initialized. However, we should try to use the worker whenever possible to not have to process frames in the main thread.

1egoman · 2026-04-16T15:17:51Z

+      } catch {
+        // best-effort: never break the media pipeline if trailer parsing fails
+      }


question: Is it worth at least logging something here to make it clear that a failure happened?

1egoman · 2026-04-16T15:33:52Z

+  } catch {
+    // getMetadata() might not be available
+  }


suggestion: Another place where it might be worth logging

1egoman · 2026-04-16T15:41:10Z

+ * When E2EE is active, the E2EE FrameCryptor worker handles trailer
+ * extraction directly (before decryption), so this manager only creates
+ * the extractor/metadata cache — no separate insertable-streams pipeline
+ * is needed.


question: I'm not sure I understand why this packet trailer logic exists both in the e2ee worker and in the new packet trailer worker. Can you shed some light on why you made this decision? Does it have to do with the goal of encrypting the packet payload and trailer separately so the SFU can strip out the trailer without that process requiring decrypting a single large block?

I don't have enough context to be incredibly confident in this suggestion, but if you can't attach multiple transforms which run in series there might be some value in adding some sort of "stream transform composer" layer above both this and the e2ee manager which would connect a series of transforms together and allows each to fully own their respective responsibilities.

The main reason is each track transceiver can only have a single transform attached, so if e2ee is enabled, we can't also attach the packetTrailer transform worker. Thus the need to have the same logic inside e2ee for parsing packet trailers if both features are enabled.

I think your suggestion makes sense but would mean a much bigger refactor where e2eeManager & e2ee worker needs to be replaced by a more generic frameTransformManager & a new worker that can handle both e2ee & packetTrailer.

1egoman · 2026-04-16T15:47:37Z

+  /**
+   * @experimental
+   * Options for enabling packet trailer extraction on received video tracks.
+   * Packet trailers carry frame-level metadata such as user timestamps and frame IDs.
+   */
+  packetTrailer?: PacketTrailerOptions;
+


thought: After this gets eventually is merged, you may want to add a todo for yourself to wire this through in components-js in a similar way to what we recently did for encryption, see here for more info: livekit/components-js#1317

1egoman · 2026-04-16T15:55:13Z


 export const version = v;
-export const protocolVersion = 16;
+export const protocolVersion = 17;


^ This is probably my biggest question as well.

It sort of sounds like you want a "feature" you can communicate support of unrelated to a protocol / client version. I recently wrote a note about this in the context of the in flight rpc protocol updates here: #1832 (comment)

Keeping in mind we haven't discussed mechanically how this pattern would work yet (ie, new fields in ParticipantInfo, something in the JoinResponse, etc?), I think if you did something like that then the SFU could know if packet trailer processing is enabled on the client independent of the protocol version and opt to strip or not strip conditionally.

Thoughts on that proposal?

…m js

chenosaurus added 10 commits April 14, 2026 11:39

add support for packet trailer features

c1e4763

update frame metadata rendering in demo

3d00ac0

when e2ee is enabled, handle the packet trailer parsing within FrameC…

0e61562

…ryptor for performance

remove orphaned file

f11025a

remove orphaned file

e00d1dd

cleanup for consistency

09e2b1f

dont display frame id if not set

c9250dd

clean up

5d9d1bc

improve parser

df3dc78

clean up

de281af

chenosaurus requested review from 1egoman and lukasIO April 14, 2026 23:43

github-advanced-security AI found potential problems Apr 14, 2026

View reviewed changes

Comment thread examples/demo/demo.ts Dismissed

chenosaurus mentioned this pull request Apr 14, 2026

User Timestamp Parsing #1812

Closed

chenosaurus added 8 commits April 14, 2026 16:47

ensure user_timestamp is bigint

a47dee7

fix bigint literals

2028be1

lint

cbda15e

lint

c9721ff

bump @livekit/protocol to 1.45.3

96b1cec

update pnpm lock file

02ec087

formatting

ed291a7

fix access to packetTrailerExtractor

b3d39e3

lukasIO reviewed Apr 16, 2026

View reviewed changes

1egoman reviewed Apr 16, 2026

View reviewed changes

chenosaurus added 4 commits April 16, 2026 10:11

fix reconnect logic & dont attach transformer to published tracks fro…

4bd5ada

…m js

backup transformer can be added in main thread if worker is not created

9f91c94

move packetTrailer files

f120282

refactor parsing into helper, clean up framecryptor

b730f81

chenosaurus added 3 commits April 16, 2026 16:24

add the moved files

e0d173c

revert unintended change

6db94bd

lint

49942ef

		@@ -0,0 +1,198 @@
		export const PACKET_TRAILER_MAGIC = Uint8Array.from([

Conversation

chenosaurus commented Apr 14, 2026

Uh oh!

changeset-bot bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

Uh oh!

github-actions bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

size-limit report 📦

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1egoman Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1egoman Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot bot commented Apr 14, 2026 •

edited

Loading

github-actions bot commented Apr 15, 2026 •

edited

Loading

1egoman Apr 16, 2026 •

edited

Loading

1egoman Apr 16, 2026 •

edited

Loading