Skip to content

feat: add trajectory template variables to LLM classifiers - #271

Draft
kev (kevchoi) wants to merge 1 commit into
mainfrom
kevchoi/20261113-trajectory-template-vars
Draft

kev (kevchoi) wants to merge 1 commit into
mainfrom
kevchoi/20261113-trajectory-template-vars

Conversation

@kevchoi

@kevchoi kev (kevchoi) commented Nov 13, 2026 •

Copy link
Copy Markdown

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-13.

Context

LLM classifiers can read a trace's thread through {{thread}}, which comes from the LLM calls in the trace. A voice call is better described by its trajectory: what the caller said, which tools ran, and what the agent actually said, cut short when the caller interrupted. This adds trajectory variables so classifiers can score voice calls, and any other trace that has a trajectory, without custom code.

Description

  • {{trajectory}} and {{trajectory_with_system}} are new template variables. When a template uses one and a trace is passed, the classifier calls trace.getTrajectory() (JS) or await trace.get_trajectory() (Python). It throws if the trace has no such method.

  • trajectory-utils.ts and trajectory_utils.py render the trajectories as plain text, one line per item:

    System: You are Pip.
    User: Where is order A1?
    Tool: lookup_order({"order_id":"A1"}) -> {"status":"shipped"}
    Assistant: Order A1 shipped on [interrupted]
    
    • System: lines appear only in {{trajectory_with_system}}, and only when the instructions change from the previous turn. Voice turns carry them in the LLM call, not the request.
    • Tool: lines cover tool steps, with the result or the error. Voice speech-to-text and text-to-speech steps aren't tool steps, so they're skipped.
    • [interrupted] marks a response the caller cut off.
    • Separate trajectories are separated by a blank line.
  • The text contains no timestamps and isn't truncated. Only text content renders, so audio parts are skipped. Sub-agents aren't rendered.

  • {{thread}} is unchanged. In Python, explicit render arguments now override template variables computed from the trace, as in JS, instead of raising a duplicate-keyword TypeError.

Testing

No automated tests yet, to keep the change small. An earlier revision of this branch had tests with a hand-written trajectory, and they passed for both languages on the same rendering code. This revision has not been run, and nothing has been run on a real trace from a Braintrust SDK.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (HEAD-1794632124)

Score Average Improvements Regressions
NumericDiff 77.9% (+0pp) 9 🟢 11 🔴
Time_to_first_token 10.29tok (+0.23tok) 93 🟢 126 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 483.7tok (+0tok) - -
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 456.69tok (-9.44tok) 107 🟢 109 🔴
Completion_reasoning_tokens 329.6tok (-7.56tok) 78 🟢 86 🔴
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 940.38tok (-9.44tok) 107 🟢 109 🔴
Estimated_cost 0$ (0$) 45 🟢 41 🔴
Duration 10.29s (+0.23s) 93 🟢 126 🔴
Llm_duration 11.15s (+0.38s) 88 🟢 131 🔴

@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261113-trajectory-template-vars branch 2 times, most recently from e6619ae to 0821a6a Compare November 14, 2026 04:23
{{trajectory}} and {{trajectory_with_system}} render a trace's trajectory
as plain text: user turns, tool calls with results or errors, and agent
responses marked when interrupted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261113-trajectory-template-vars branch from 0821a6a to 6c05c3a Compare November 14, 2026 04:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant