Speko Docs
Build

Provider-direct speech-to-speech

Mint a short-lived OpenAI, Gemini Live, or xAI session and connect the browser directly to the provider.

Provider-direct speech-to-speech (S2S) keeps Speko in the control and billing path without putting Speko in the media path:

  1. Your backend calls POST /v1/sessions with mode: 's2s' and its Speko API key.
  2. Speko reserves a bounded entitlement and returns a scoped, short-lived provider credential.
  3. The browser connects directly to OpenAI Realtime, Gemini Live, or xAI Grok Voice.
  4. Speko stores content-free lifecycle and billing evidence. Audio never traverses a Speko media proxy or LiveKit room.

Only xAI Grok Voice (grok-voice-latest, grok-voice-3.0) runs provider-direct today, and only on sessions without an agentId. Every other session runs on Speko's LiveKit worker: Grok Voice when the session names an agent, and openai:gpt-live-1 in place of any OpenAI or Google model. The response then has transport: "livekit" with a transportToken and transportUrl, which neither RealtimeVoiceConversation nor realtime.connect() can use. Join those sessions with VoiceConversation.

Use this path for the lowest possible browser-to-model latency. If you need Speko's routed STT → LLM → TTS pipeline, recordings, or server-persisted transcripts, use a VoiceConversation cascade session instead.

Install

npm install @spekoai/client

1. Mint an S2S session on your backend

Keep SPEKO_API_KEY on your server. Return the S2S response to the browser unchanged; it contains a scoped provider credential and Speko control URLs, never a provider root key.

import crypto from 'node:crypto';

app.post('/api/realtime-session', async (req, res) => {
  const idempotencyKey = req.get('Idempotency-Key') || crypto.randomUUID();
  const response = await fetch('https://api.speko.dev/v1/sessions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.SPEKO_API_KEY}`,
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey,
    },
    body: JSON.stringify({
      mode: 's2s',
      s2s: {
        provider: 'xai',
        model: 'grok-voice-latest',
        voice: 'eve',
        systemPrompt: 'You are a concise voice assistant.',
      },
      ttlSeconds: 900,
    }),
  });

  const body = await response.json();
  res.status(response.status).json(body);
});

Idempotency-Key is required for S2S. Reuse the same value only when retrying an ambiguous bootstrap timeout; Speko will return the same reservation instead of minting a second one.

2. Connect directly from the browser

import { useEffect, useRef, useState } from 'react';
import { RealtimeVoiceConversation } from '@spekoai/client';

export function RealtimePanel() {
  const conversationRef = useRef<RealtimeVoiceConversation | null>(null);
  const [status, setStatus] = useState('idle');
  const [transcript, setTranscript] = useState<string[]>([]);

  async function start() {
    setStatus('connecting');
    const response = await fetch('/api/realtime-session', { method: 'POST' });
    if (!response.ok) throw new Error('Could not create an S2S session');
    const session = await response.json();

    const conversation = await RealtimeVoiceConversation.create({
      ...session,
      onStatusChange: setStatus,
      onMessage: ({ source, text, isFinal }) => {
        if (isFinal) setTranscript((items) => [...items, `${source}: ${text}`]);
      },
      onError: (error) => console.error(error),
      onDisconnect: () => setStatus('idle'),
    });

    conversationRef.current = conversation;
  }

  async function stop() {
    await conversationRef.current?.endSession();
    conversationRef.current = null;
  }

  useEffect(() => () => { void conversationRef.current?.endSession(); }, []);

  return (
    <div>
      <button onClick={start} disabled={status !== 'idle'}>Start</button>
      <button onClick={stop} disabled={status === 'idle'}>Stop</button>
      <p>Status: {status}</p>
      <ul>{transcript.map((item, index) => <li key={index}>{item}</li>)}</ul>
    </div>
  );
}

RealtimeVoiceConversation acquires the microphone, performs the provider handshake, plays response audio, normalizes transcript events, reports content-free telemetry, and tears down every media resource in endSession().

Providers and defaults

ProviderDefault modelDefault voiceDirect transportPCM input / output
OpenAIgpt-realtime-2.1marinWebRTC (not served today; see the warning above)24 kHz / 24 kHz
Googlegemini-3.1-flash-live-previewPuckGemini Live WebSocket (not served today; see the warning above)16 kHz / 24 kHz
xAIgrok-voice-latesteveGrok Voice WebSocket24 kHz / 24 kHz

Pin s2s.model to select another model from the same provider. xAI's grok-voice-latest currently resolves to grok-voice-3.0; pin grok-voice-3.0 to stay on it when the alias moves. Grok Voice 3.0 reasons before it answers by default (reasoning.effort: "high"); set s2s.reasoningEffort: "none" to turn that off (see below). Google also serves gemini-3.8-live (its current Live generation) and gemini-3.8-live-extended-thinking (the same model with background reasoning, which answers later) on the same socket, sample rates and voices as the default above.

Thinking and reasoning

Three models take a thinking dial, and each one is strict about it. s2s.reasoningEffort drives all three with one value:

FieldApplies toValuesDefault
s2s.thinkingLevelgoogle:gemini-3.8-live-extended-thinkinglow, medium, highlow
s2s.backendModelopenai:gpt-live-1an OpenAI LLM catalog idgpt-5.6-luna
s2s.backendReasoningEffortopenai:gpt-live-1low, medium, high (minimal is accepted and runs as low)the backend model's own default
s2s.reasoningEffortall three abovenone, minimal, low, medium, high, xhigheach model's own default

thinkingLevel is a property of the model, not a preference. The extended-thinking model requires one — Speko defaults it to low so a session that names none still connects — and every other Gemini Live model refuses one, so sending it there is a 400. Google rejects minimal on the extended-thinking model, which is why it is not offered.

GPT-Live's voice model does no reasoning itself: it delegates to a backend Responses model, so backendModel and backendReasoningEffort are the only knobs that change how hard it thinks. Higher effort buys better answers and costs the caller silence while it works.

Grok Voice 3.0 reasons before every reply unless the session says otherwise. xAI accepts only high and none for voice, so reasoningEffort maps to the nearer of the two (low becomes none, medium becomes high). On GPT-Live it sets backendReasoningEffort, and on the extended-thinking Gemini model it sets thinkingLevel; the model-specific field wins when you send both. A model with no dial returns 400 for the field. To set it once for every call an agent makes, web and phone alike, save it on the agent as llmOptions.reasoningEffort; a value on the session wins over the agent's.

Both fields ride the s2s block of POST /v1/sessions, the same request the server example above makes:

{
  "mode": "s2s",
  "s2s": {
    "provider": "google",
    "model": "gemini-3.8-live-extended-thinking",
    "thinkingLevel": "high"
  }
}
{
  "mode": "s2s",
  "s2s": {
    "provider": "openai",
    "model": "gpt-live-1",
    "backendModel": "gpt-5.6-terra",
    "backendReasoningEffort": "medium"
  }
}

The TypeScript SDK takes the Gemini option as a realtime.connect parameter:

const session = await speko.realtime.connect({
  provider: 'google',
  model: 'gemini-3.8-live-extended-thinking',
  thinkingLevel: 'high',
});

GPT-Live's two options have no realtime.connect equivalent: that model is hosted on Speko's LiveKit worker, and connect returns a provider-direct handle. Send them on POST /v1/sessions as above and join the returned room with VoiceConversation.

You may omit s2s.provider and s2s.model to let Speko choose an eligible S2S route. If you set one, set both. Use constraints.allowedProviders.s2s to limit routing to provider names or exact provider:model pairs.

Duration and billing

For S2S, ttlSeconds is the requested maximum session horizon, not a LiveKit join-token TTL. The API defaults to 1,800 seconds and accepts up to 3,600 seconds. Provider credential limits can make the first authorized slice shorter:

  • OpenAI uses a provider-authenticated sideband to bind the WebRTC call, collect provider events, and enforce the authorized deadline without receiving media.
  • Gemini Live uses renewable entitlement slices. RealtimeVoiceConversation reserves the next slice and resumes the Gemini session automatically while it remains inside the requested horizon.
  • Browser telemetry is operational evidence and may update the dashboard estimate, but it is not trusted to reduce the final charge. When exact provider-side evidence is unavailable, Speko settles the bounded entitlement that was authorized before the credential was minted.

The create response reports reservation.authorizedDurationSeconds, reservation.billing.maximumAmountMicros, and any reservation.billing.renewalUrl. Treat those returned values—not only the requested ttlSeconds—as the active authorization.

Current artifact behavior

  • S2S transcript callbacks are available in the browser, but Speko does not yet persist them to GET /v1/sessions/{id}/transcript.
  • Speko does not record provider-direct S2S media. recordingStatus is null, and the recording endpoint has no artifact for these sessions.
  • endSession() closes the direct provider connection locally. The provider credential and server-side entitlement remain bounded even if a modified client omits the final telemetry event.

Next

On this page