Evals

A voice regression is invisible to every tool a coding agent has. Change one line of a system prompt and the agent quietly stops confirming before it books — no diff, type check or unit test registers it. The user finds out from a real customer.

speko-cli eval generate --agent <id>            # propose a suite; prints it, saves nothing
speko-cli eval generate --agent <id> --persist  # keep it
speko-cli eval list --agent <id>                # what is stored
speko-cli eval run --agent <id>                 # run it, report what broke
speko-cli eval trends --agent <id>              # pass rate over time

Generation writes the suite for you

eval generate builds cases from the agent's own system prompt, tools and knowledge-base titles, so nobody hand-authors test cases.

Preview is the default. Generation calls a model, so the same agent yields a different suite each run; writing on the first run would leave someone who only wanted to look with rows to clean up. Add --persist to keep them.

Exit 6 means the agent is wrong

speko-cli eval run --agent <id>
echo $?   # 6 on a failing case

Not 1. A CI step has to tell "the agent's behaviour is wrong" from "the network was down" — collapse them and the first failure that was really a flake teaches everyone to ignore the gate.

speko-cli eval run --agent "$AGENT" || {
  status=$?
  [ "$status" -eq 6 ] && { echo "behaviour regressed"; exit 1; }
  echo "eval could not run (exit $status)"; exit 0
}

A queued run is not a failed run

Runs are queued for a worker that places the simulated call and scores it. A run that never leaves queued means nothing is consuming the queue — reported as exactly that, separately from a slow run that did reach running.

The two need opposite responses: one is missing infrastructure, the other is a prompt to fix. Calling an unclaimed run a test failure would send someone to edit a prompt that was never tested.

Reading a failure

eval run prints the broken case, what the worker said about it, and the command to re-run that one case alone — with the real ids, so it can be copied.