Repository navigation
feat: add trajectory template variables to LLM classifiers - #271
Draft
kev (kevchoi) wants to merge 1 commit into
Draft
kev (kevchoi) wants to merge 1 commit into
kev (kevchoi) wants to merge 1 commit into
Conversation
Braintrust eval report
|
kev (kevchoi)
force-pushed
the
kevchoi/20261113-trajectory-template-vars
branch
2 times, most recently
from
November 14, 2026 04:23
e6619ae to
0821a6a
Compare
{{trajectory}} and {{trajectory_with_system}} render a trace's trajectory
as plain text: user turns, tool calls with results or errors, and agent
responses marked when interrupted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
force-pushed
the
kevchoi/20261113-trajectory-template-vars
branch
from
November 14, 2026 04:54
0821a6a to
6c05c3a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
LLM classifiers can read a trace's thread through
{{thread}}, which comes from the LLM calls in the trace. A voice call is better described by its trajectory: what the caller said, which tools ran, and what the agent actually said, cut short when the caller interrupted. This adds trajectory variables so classifiers can score voice calls, and any other trace that has a trajectory, without custom code.Description
{{trajectory}}and{{trajectory_with_system}}are new template variables. When a template uses one and a trace is passed, the classifier callstrace.getTrajectory()(JS) orawait trace.get_trajectory()(Python). It throws if the trace has no such method.trajectory-utils.tsandtrajectory_utils.pyrender the trajectories as plain text, one line per item:System:lines appear only in{{trajectory_with_system}}, and only when the instructions change from the previous turn. Voice turns carry them in the LLM call, not the request.Tool:lines covertoolsteps, with the result or the error. Voice speech-to-text and text-to-speech steps aren't tool steps, so they're skipped.[interrupted]marks a response the caller cut off.The text contains no timestamps and isn't truncated. Only text content renders, so audio parts are skipped. Sub-agents aren't rendered.
{{thread}}is unchanged. In Python, explicit render arguments now override template variables computed from the trace, as in JS, instead of raising a duplicate-keywordTypeError.Testing
No automated tests yet, to keep the change small. An earlier revision of this branch had tests with a hand-written trajectory, and they passed for both languages on the same rendering code. This revision has not been run, and nothing has been run on a real trace from a Braintrust SDK.
🤖 Generated with Claude Code