Speko Docs

Evaluations

POST /v1/evaluations — ask typed questions about an audio clip or text and get calibrated probabilities back.

POST /v1/evaluations answers typed questions about material you send. The material is either one short audio clip or text state. Each question is one of three types:

  • a yes/no (noul);
  • a pick from options you define (choice);
  • a rating on a scale you define (score).

Every answer comes back as calibrated probabilities. The Router only reports what the model decided. What happens next stays in your code: routing the call, flagging a turn, or changing how your agent responds.

The question and answer shapes are the same on every evaluation model. Between models that take the same input (audio or text), switching is a routing change; the rest of your code stays the same.

Models

ModelDecides overRouting
Gemini Decision 1One audio clip (WAV){"mode": "explicit", "provider": "gemini", "model": "gemini-decision-1"}
TypeSafe JevText state (JSON){"mode": "explicit", "provider": "typesafe", "model": "jev-1.13.0"}

Evaluation routing is always explicit, and there is no automatic failover between models. GET /v1/models?kind=evaluation lists the evaluation models available to your organization. On each entry:

  • capabilities.evaluation_inputs lists audio and/or text. When it is absent, the model takes text only; Jev's entry omits it.
  • capabilities.evaluation_limits holds the model's published request bounds. See Limits.
  • audio_formats, at the top level of the entry, lists the clip formats a model that takes audio accepts.

Ask about an audio clip

The body is multipart/form-data with exactly two parts, in this order:

  1. request: a JSON document (at most 1 MiB).
  2. audio: one WAV file containing PCM s16le.

Part order is load-bearing. The idempotency content hash covers the parts in this order, and extra or reordered parts are rejected.

curl -s https://router.speko.dev/v1/evaluations \
  -H "Authorization: Bearer $SPEKO_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -F 'request=<request.json;type=application/json' \
  -F 'audio=@caller-turn.wav;type=audio/wav'

With request.json:

{
  "routing": { "mode": "explicit", "provider": "gemini", "model": "gemini-decision-1" },
  "state": "One caller turn from a support call. The agent had just asked for the order number.",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "How does the caller sound in this turn?",
      "criteria": {
        "positive": "pleased, warm, relieved",
        "neutral": null,
        "confused": "unsure, asking for clarification",
        "frustrated": "impatient, annoyed, repeating themselves",
        "angry": "hostile, raised voice",
        "other": "none of these fit"
      }
    },
    "wants_human": {
      "type": "noul",
      "instructions": "Is the caller asking to speak to a person?"
    },
    "agitation": {
      "type": "score",
      "instructions": "How calm or agitated does the caller sound?",
      "criteria": ["calm", "tense", "agitated", "shouting"]
    }
  }
}

The same request in Python:

import json, os, uuid
import requests

request = {
    "routing": {"mode": "explicit", "provider": "gemini", "model": "gemini-decision-1"},
    "questions": {
        "wants_human": {"type": "noul", "instructions": "Is the caller asking to speak to a person?"},
    },
}

with open("caller-turn.wav", "rb") as clip:
    response = requests.post(
        "https://router.speko.dev/v1/evaluations",
        headers={
            "Authorization": f"Bearer {os.environ['SPEKO_API_KEY']}",
            "Idempotency-Key": str(uuid.uuid4()),
        },
        # The list keeps the parts in order: request first, then audio.
        files=[
            ("request", (None, json.dumps(request), "application/json")),
            ("audio", ("caller-turn.wav", clip, "audio/wav")),
        ],
        timeout=30,
    )
response.raise_for_status()
print(response.json()["answers"]["wants_human"]["noul"])

The request part

routingobjectrequired

The routing object, with mode set to explicit and the evaluation model's provider and model.

statestring

Optional plain-text context about the clip, such as who is speaking or what was said just before. The questions are answered about the clip; state only frames it. Omit it, or send null, when there is none.

questionsobjectrequired

A map from question id to question. Ids are 1 to 64 characters from A–Z, a–z, 0–9, _ and -. Each answer comes back under its question's id. Ask everything you want to know about a clip in one request: the clip is sent and billed once, however many questions ride with it.

Questions

Every question has a type and instructions. Instructions are the question itself, as a plain string. choice and score questions also need criteria; on a noul question it is optional.

noulquestion

A yes/no question. criteria is optional: {"true": "...", "false": "..."} describes what each answer means, and either value may be null.

choicequestion

Pick one of your options. criteria maps each option key to a description, or to null when the key explains itself.

  • The answer's choice is one of your keys.
  • Keys may contain inner spaces. They may not start or end with whitespace, and may not contain control characters or any whitespace other than an ASCII space.
scorequestion

Rate on your scale. criteria is an array of 2 to 10 level descriptions, ordered from lowest to highest. Levels must be unique.

In an audio request, every text value is a plain string: instructions, option descriptions, noul criteria and score levels. Structured JSON content is a text evaluation feature.

Response

{
  "model": "gemini-decision-1",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "frustrated",
      "probabilities": {
        "angry": 0.04,
        "confused": 0.06,
        "frustrated": 0.71,
        "neutral": 0.12,
        "other": 0.03,
        "positive": 0.04
      },
      "confidence": 0.71
    },
    "wants_human": { "type": "noul", "noul": 0.82 },
    "agitation": {
      "type": "score",
      "score": 1.31,
      "probabilities": { "0": 0.12, "1": 0.49, "2": 0.35, "3": 0.04 },
      "confidence": 0.49,
      "legend": { "0": "calm", "1": "tense", "2": "agitated", "3": "shouting" }
    }
  },
  "usage": { "input_tokens": 742 }
}
modelstring

The model that answered.

answersobject

Exactly one answer per question, under the question's id, with the question's type:

  • noul: noul is the probability of yes, from 0 to 1. There is no confidence field; it would always be max(noul, 1 - noul).
  • choice: choice is the most likely option. probabilities covers every option and sums to 1. confidence is a 0-to-1 certainty score (see below).
  • score: score is the probability-weighted average level index (0 is your first level), so it can fall between levels. probabilities is keyed by level index and sums to 1. confidence is a 0-to-1 certainty score (see below). legend echoes your levels by index.

What confidence measures depends on the model. On Gemini Decision 1 it is the probability of the top option or level. On Jev it is TypeSafe's own statistic over the whole distribution, passed through unchanged (how Jev computes it); it is not the probability of any one answer. To act on how likely a particular answer is, threshold its entry in probabilities.

usageobject

input_tokens: the input tokens billed for the request (audio, context and question text). Some models also report output_tokens (Jev does); the count is passed through but never billed. Don't rely on it being absent or zero.

Each vendor's raw answers are converted to this shape before they reach you, so the same code reads every model. The route is echoed in the Speko-Request-ID, Speko-Provider, Speko-Model and Speko-Region response headers.

Using the answers

  • Set thresholds on your own data. The probabilities are calibrated. Treat them as probabilities, and pick thresholds from labeled examples of your own traffic rather than a fixed 0.5.
  • Stay inside 0.01 to 0.99. Gemini Decision 1 keeps yes/no probabilities in that range, so a threshold above 0.99 never matches.
  • Give choices a way out. choice probabilities always sum to 1, even when none of your options fits. Add a catch-all such as other, or act only when the chosen option's entry in probabilities clears your threshold.
  • Don't encode meaning in option order. Gemini Decision 1 reads option keys in alphabetical order and breaks exact ties toward the alphabetically first key. A key like none_of_the_above does not mean "last".

Live conversations

The endpoint answers one complete clip per request; it does not stream.

For a live call:

  1. Cut each caller turn, from the start of speech to the end of the turn, into a WAV clip. 16 kHz mono is plenty.
  2. Send it when the turn ends.
  3. Run the request alongside your agent's reply rather than in front of it, unless the decision must shape that same reply.

Ask about text

TypeSafe Jev decides over text state. Send a JSON body (Content-Type: application/json) with the same routing and questions, plus a required state. For text evaluations, state, instructions and criteria may be strings or structured JSON.

curl -s https://router.speko.dev/v1/evaluations \
  -H "Authorization: Bearer $SPEKO_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{
    "routing": { "mode": "explicit", "provider": "typesafe", "model": "jev-1.13.0" },
    "state": { "transcript": ["Agent: Can I have your order number?", "Caller: I already gave it to you twice."] },
    "questions": {
      "frustrated": { "type": "noul", "instructions": "Is the caller frustrated?" }
    }
  }'

The response has the same shape as for audio.

Limits

Each model publishes its bounds as capabilities.evaluation_limits on GET /v1/models?kind=evaluation. A field that is absent means the model publishes no such bound. A request outside the routed model's limits is refused before admission, so it is never charged. If you switch between models, validate against the intersection of their limits.

Gemini Decision 1 today:

Boundgemini-decision-1
AudioWAV, PCM s16le, mono, 8 to 48 kHz
Clip length60 seconds (max_audio_ms 60000)
Clip size2 MiB of PCM (max_audio_bytes)
Questions per request1,000
Options per choice question255
Option key length128 characters
Levels per score question10

Refusals:

You sentResponse
A JSON body to an audio model, or a clip to a text model400 capability_unsupported; the hint says which body shape the model takes
More questions, options, levels or characters than the model's evaluation_limits allow400 capability_unsupported naming the limit
A clip longer than max_audio_ms or larger than max_audio_bytes413 payload_too_large naming the limit
Audio that is not WAV with PCM s16le, or a sample rate or channel count outside the model's audio_formats415 unsupported_media
An invalid request part: a bad question id, structured text in an audio request, blank instructions, a malformed option key or a repeated score level400 invalid_request naming the field
A request part and clip that together exceed about 4 MiB413 payload_too_large

Convert compressed audio (MP3, M4A, OGG and so on) to WAV first, for example with:

ffmpeg -i call.mp3 -ac 1 -ar 16000 -sample_fmt s16 caller-turn.wav

Billing

Evaluations are billed on input tokens only: the audio, the state and the question text. A model may also report output_tokens; those are not billed.

Gemini Decision 1 is billed at Google's list price of $0.10 per million input tokens, plus 5%: $0.105 per million.

  • Audio comes to roughly 30 tokens per second.
  • So a 13-second caller turn with four questions is about 740 tokens, or $0.00008.

As on every Router route, admission holds a conservative ceiling against your credit, and you are billed for the usage the provider reports. See Billing and usage.

On this page