Evaluations
POST /v1/evaluations — ask typed questions about an audio clip or text and get calibrated probabilities back.
POST /v1/evaluations answers typed questions about material you send. The material is either one short audio clip or text state. Each question is one of three types:
- a yes/no (
noul); - a pick from options you define (
choice); - a rating on a scale you define (
score).
Every answer comes back as calibrated probabilities. The Router only reports what the model decided. What happens next stays in your code: routing the call, flagging a turn, or changing how your agent responds.
The question and answer shapes are the same on every evaluation model. Between models that take the same input (audio or text), switching is a routing change; the rest of your code stays the same.
Models
| Model | Decides over | Routing |
|---|---|---|
| Gemini Decision 1 | One audio clip (WAV) | {"mode": "explicit", "provider": "gemini", "model": "gemini-decision-1"} |
| TypeSafe Jev | Text state (JSON) | {"mode": "explicit", "provider": "typesafe", "model": "jev-1.13.0"} |
Evaluation routing is always explicit, and there is no automatic failover between models. GET /v1/models?kind=evaluation lists the evaluation models available to your organization. On each entry:
capabilities.evaluation_inputslistsaudioand/ortext. When it is absent, the model takes text only; Jev's entry omits it.capabilities.evaluation_limitsholds the model's published request bounds. See Limits.audio_formats, at the top level of the entry, lists the clip formats a model that takes audio accepts.
Ask about an audio clip
The body is multipart/form-data with exactly two parts, in this order:
request: a JSON document (at most 1 MiB).audio: one WAV file containing PCM s16le.
Part order is load-bearing. The idempotency content hash covers the parts in this order, and extra or reordered parts are rejected.
curl -s https://router.speko.dev/v1/evaluations \
-H "Authorization: Bearer $SPEKO_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-F 'request=<request.json;type=application/json' \
-F 'audio=@caller-turn.wav;type=audio/wav'With request.json:
{
"routing": { "mode": "explicit", "provider": "gemini", "model": "gemini-decision-1" },
"state": "One caller turn from a support call. The agent had just asked for the order number.",
"questions": {
"sentiment": {
"type": "choice",
"instructions": "How does the caller sound in this turn?",
"criteria": {
"positive": "pleased, warm, relieved",
"neutral": null,
"confused": "unsure, asking for clarification",
"frustrated": "impatient, annoyed, repeating themselves",
"angry": "hostile, raised voice",
"other": "none of these fit"
}
},
"wants_human": {
"type": "noul",
"instructions": "Is the caller asking to speak to a person?"
},
"agitation": {
"type": "score",
"instructions": "How calm or agitated does the caller sound?",
"criteria": ["calm", "tense", "agitated", "shouting"]
}
}
}The same request in Python:
import json, os, uuid
import requests
request = {
"routing": {"mode": "explicit", "provider": "gemini", "model": "gemini-decision-1"},
"questions": {
"wants_human": {"type": "noul", "instructions": "Is the caller asking to speak to a person?"},
},
}
with open("caller-turn.wav", "rb") as clip:
response = requests.post(
"https://router.speko.dev/v1/evaluations",
headers={
"Authorization": f"Bearer {os.environ['SPEKO_API_KEY']}",
"Idempotency-Key": str(uuid.uuid4()),
},
# The list keeps the parts in order: request first, then audio.
files=[
("request", (None, json.dumps(request), "application/json")),
("audio", ("caller-turn.wav", clip, "audio/wav")),
],
timeout=30,
)
response.raise_for_status()
print(response.json()["answers"]["wants_human"]["noul"])The request part
routingobjectrequiredThe routing object, with mode set to explicit and the
evaluation model's provider and model.
statestringOptional plain-text context about the clip, such as who is speaking or what
was said just before. The questions are answered about the clip; state
only frames it. Omit it, or send null, when there is none.
questionsobjectrequiredA map from question id to question. Ids are 1 to 64 characters
from A–Z, a–z, 0–9, _ and -. Each answer comes back under its
question's id. Ask everything you want to know about a clip in one request:
the clip is sent and billed once, however many questions ride with it.
Questions
Every question has a type and instructions. Instructions are the question itself, as a plain string. choice and score questions also need criteria; on a noul question it is optional.
noulquestionA yes/no question. criteria is optional:
{"true": "...", "false": "..."} describes what each answer means, and
either value may be null.
choicequestionPick one of your options. criteria maps each option key to a description,
or to null when the key explains itself.
- The answer's
choiceis one of your keys. - Keys may contain inner spaces. They may not start or end with whitespace, and may not contain control characters or any whitespace other than an ASCII space.
scorequestionRate on your scale. criteria is an array of 2 to 10 level descriptions,
ordered from lowest to highest. Levels must be unique.
In an audio request, every text value is a plain string: instructions, option descriptions, noul criteria and score levels. Structured JSON content is a text evaluation feature.
Response
{
"model": "gemini-decision-1",
"answers": {
"sentiment": {
"type": "choice",
"choice": "frustrated",
"probabilities": {
"angry": 0.04,
"confused": 0.06,
"frustrated": 0.71,
"neutral": 0.12,
"other": 0.03,
"positive": 0.04
},
"confidence": 0.71
},
"wants_human": { "type": "noul", "noul": 0.82 },
"agitation": {
"type": "score",
"score": 1.31,
"probabilities": { "0": 0.12, "1": 0.49, "2": 0.35, "3": 0.04 },
"confidence": 0.49,
"legend": { "0": "calm", "1": "tense", "2": "agitated", "3": "shouting" }
}
},
"usage": { "input_tokens": 742 }
}modelstringThe model that answered.
answersobjectExactly one answer per question, under the question's id, with the
question's type:
- noul:
noulis the probability of yes, from 0 to 1. There is noconfidencefield; it would always bemax(noul, 1 - noul). - choice:
choiceis the most likely option.probabilitiescovers every option and sums to 1.confidenceis a 0-to-1 certainty score (see below). - score:
scoreis the probability-weighted average level index (0 is your first level), so it can fall between levels.probabilitiesis keyed by level index and sums to 1.confidenceis a 0-to-1 certainty score (see below).legendechoes your levels by index.
What confidence measures depends on the model. On Gemini Decision 1 it is
the probability of the top option or level. On Jev it is TypeSafe's own
statistic over the whole distribution, passed through unchanged
(how Jev computes it); it is not
the probability of any one answer. To act on how likely a particular answer
is, threshold its entry in probabilities.
usageobjectinput_tokens: the input tokens billed for the request (audio, context and
question text). Some models also report output_tokens (Jev does); the
count is passed through but never billed. Don't rely on it being absent or
zero.
Each vendor's raw answers are converted to this shape before they reach you, so the same code reads every model. The route is echoed in the Speko-Request-ID, Speko-Provider, Speko-Model and Speko-Region response headers.
Using the answers
- Set thresholds on your own data. The probabilities are calibrated. Treat them as probabilities, and pick thresholds from labeled examples of your own traffic rather than a fixed 0.5.
- Stay inside 0.01 to 0.99. Gemini Decision 1 keeps yes/no probabilities in that range, so a threshold above 0.99 never matches.
- Give choices a way out.
choiceprobabilities always sum to 1, even when none of your options fits. Add a catch-all such asother, or act only when the chosen option's entry inprobabilitiesclears your threshold. - Don't encode meaning in option order. Gemini Decision 1 reads option keys in alphabetical order and breaks exact ties toward the alphabetically first key. A key like
none_of_the_abovedoes not mean "last".
Live conversations
The endpoint answers one complete clip per request; it does not stream.
For a live call:
- Cut each caller turn, from the start of speech to the end of the turn, into a WAV clip. 16 kHz mono is plenty.
- Send it when the turn ends.
- Run the request alongside your agent's reply rather than in front of it, unless the decision must shape that same reply.
Ask about text
TypeSafe Jev decides over text state. Send a JSON body (Content-Type: application/json) with the same routing and questions, plus a required state. For text evaluations, state, instructions and criteria may be strings or structured JSON.
curl -s https://router.speko.dev/v1/evaluations \
-H "Authorization: Bearer $SPEKO_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{
"routing": { "mode": "explicit", "provider": "typesafe", "model": "jev-1.13.0" },
"state": { "transcript": ["Agent: Can I have your order number?", "Caller: I already gave it to you twice."] },
"questions": {
"frustrated": { "type": "noul", "instructions": "Is the caller frustrated?" }
}
}'The response has the same shape as for audio.
Limits
Each model publishes its bounds as capabilities.evaluation_limits on GET /v1/models?kind=evaluation. A field that is absent means the model publishes no such bound. A request outside the routed model's limits is refused before admission, so it is never charged. If you switch between models, validate against the intersection of their limits.
Gemini Decision 1 today:
| Bound | gemini-decision-1 |
|---|---|
| Audio | WAV, PCM s16le, mono, 8 to 48 kHz |
| Clip length | 60 seconds (max_audio_ms 60000) |
| Clip size | 2 MiB of PCM (max_audio_bytes) |
| Questions per request | 1,000 |
Options per choice question | 255 |
| Option key length | 128 characters |
Levels per score question | 10 |
Refusals:
| You sent | Response |
|---|---|
| A JSON body to an audio model, or a clip to a text model | 400 capability_unsupported; the hint says which body shape the model takes |
More questions, options, levels or characters than the model's evaluation_limits allow | 400 capability_unsupported naming the limit |
A clip longer than max_audio_ms or larger than max_audio_bytes | 413 payload_too_large naming the limit |
Audio that is not WAV with PCM s16le, or a sample rate or channel count outside the model's audio_formats | 415 unsupported_media |
An invalid request part: a bad question id, structured text in an audio request, blank instructions, a malformed option key or a repeated score level | 400 invalid_request naming the field |
A request part and clip that together exceed about 4 MiB | 413 payload_too_large |
Convert compressed audio (MP3, M4A, OGG and so on) to WAV first, for example with:
ffmpeg -i call.mp3 -ac 1 -ar 16000 -sample_fmt s16 caller-turn.wavBilling
Evaluations are billed on input tokens only: the audio, the state and the question text. A model may also report output_tokens; those are not billed.
Gemini Decision 1 is billed at Google's list price of $0.10 per million input tokens, plus 5%: $0.105 per million.
- Audio comes to roughly 30 tokens per second.
- So a 13-second caller turn with four questions is about 740 tokens, or $0.00008.
As on every Router route, admission holds a conservative ceiling against your credit, and you are billed for the usage the provider reports. See Billing and usage.