AI Agent Evals for PromptOps

Adds an AI Agent Evals entry to the menu of PromptOps. It measures how fast your AI models really answer, through the same CLI you use every day, and tells you whether today is normal.

Plugin id	promptops.agent-evals
Section	pages
Works with	Claude Code and Codex
Status	Version 0.1.1. Speed measurements. Accuracy on your codebase is not here yet: see below

Where the code is. All of it is in src/plugin.js: one file of plain JavaScript, with no build step. What you read there is exactly what PromptOps runs, and what was reviewed.

What it does

• Lists the models PromptOps can measure, grouped by provider. A provider that is not installed is explained, not hidden.
• Runs the same short writing task three times per model and keeps the middle run.
• Ranks the models and reads each result twice: against the expected range of that model, and against your own history on this computer.
• Shows the progress while it measures, and the reason when a provider refuses a model.
• Remembers your last selection.

The speed is the output tokens the provider reports, divided by the whole time, CLI start included. It is what you actually wait for, and it is comparable between a CLI that streams and one that answers in one block.

What it can do

People approve these permissions before installing.

Permission	Meaning
ai:benchmark	Measure how fast your AI models answer. PromptOps writes the prompt and the plugin gets numbers only. You confirm every run
storage	Keep its own small local data: your last selection

Why this is safe with every provider

The plugin never talks to a model. PromptOps owns the prompt, which is short and fixed, and runs the CLIs itself in an empty folder. The plugin chooses models from the list PromptOps offers and receives numbers: tokens, seconds, tokens per second. It never sees what a model wrote, and it has no network permission. Before a run you see which models are called and how many calls that makes on your accounts.

Not here yet

The built-in AI Agent Evals of the Electron app also graded the accuracy of each model on questions about your codebase. That part runs the CLIs with their read tools inside your repository, so it needs its own design before a plugin can ask for it. This plugin measures speed only.

Try it without PromptOps

git clone https://github.com/shellonback/promptops-plugin-templates
git clone https://github.com/shellonback/promptops-plugin-agent-evals
cd promptops-plugin-templates
node playground/server.mjs --plugin ../promptops-plugin-agent-evals

Open http://127.0.0.1:4173 and choose local:promptops-plugin-agent-evals. The playground calls no model for measurements: the models and the numbers come from benchmark in fixtures.json, and the progress reports are replayed with a short pause.

Tests

Zero dependencies, Node 18 or later:

node --test

Files

File	What it is
promptops-plugin.json	The manifest: id, the menu entry, permissions, the providers it works with
src/plugin.js	The source of the plugin. One file of plain JavaScript, no dependencies and no build step: what you read is what runs
fixtures.json	Sample models and sample numbers for the playground
test/plugin.test.mjs	The tests

License

MIT, Shellonback Team.