realtime

Provider-direct speech-to-speech with OpenAI Realtime, Gemini Live, and xAI Grok Voice.

Open a speech-to-speech (S2S) session. speko.realtime.connect() reserves a Speko entitlement and mints a short-lived provider credential via POST /v1/sessions, then connects directly to OpenAI Realtime, Gemini Live, or xAI Grok Voice. Audio never traverses a Speko media proxy.

Use this low-level resource only in a trusted runtime because it creates the session with your Speko API key. Browser apps should mint on their backend and pass the response to @spekoai/client. OpenAI uses WebRTC; Gemini Live and xAI use WebSockets.

import { Speko } from '@spekoai/sdk';

const speko = new Speko({ apiKey: process.env.SPEKO_API_KEY! });

const session = await speko.realtime.connect({
  provider: 'openai',
  model: 'gpt-realtime-2.1',
  agentId: 'agent_123',
  webhookTags: { environment: 'production' },
});

session.on((frame) => {
  if (frame.type === 'audio') play(frame.pcm);
  else if (frame.type === 'transcript') console.log(frame.role, frame.text);
});

session.sendAudio(pcm16Chunk);
// ... end of user turn
session.commit();

`speko.realtime.connect(params)`

Signature

speko.realtime.connect(
  params: RealtimeConnectParams,
): Promise<RealtimeSessionHandle>

`RealtimeConnectParams`

Field	Type	Description
`provider`	`'openai' \| 'google' \| 'xai'`	S2S provider.
`model`	`string`	Provider-specific model id, such as `gpt-realtime-2.1`, `gemini-3.1-flash-live-preview`, or `grok-voice-latest`.
`voice`	`string?`	Voice id override — interpreted per provider.
`systemPrompt`	`string?`	Initial system instruction.
`temperature`	`number?`
`inputSampleRate`	`16000 \| 24000?`	PCM rate you'll be sending.
`outputSampleRate`	`16000 \| 24000?`	PCM rate you want back.
`tools`	`RealtimeToolSpec[]?`	Tool definitions the assistant may call.
`agentId`	`string?`	Agent used for workspace webhook routing.
`webhookTags`	`Record<string, string>?`	Case-sensitive workspace webhook routing tags; requires `agentId`.
`metadata`	`Record<string, unknown>?`	Free-form metadata attached to the session record.
`ttlSeconds`	`number?`	Requested maximum session duration; the returned entitlement may impose a lower cap.
`idempotencyKey`	`string?`	Stable bootstrap key to reuse after an ambiguous timeout. Generated when omitted.

`RealtimeSessionHandle`

Property	Type	Description
`sessionId`	`string`	Server-assigned session id.
`expiresAt`	`string`	ISO-8601 expiry of the delegated credential.
`inputSampleRate`	`16000 \| 24000`	PCM rate the session accepts.
`outputSampleRate`	`16000 \| 24000`	PCM rate the session returns.

Methods

Method	Description
`sendAudio(pcm: Uint8Array): void`	Send a PCM16 audio chunk directly to the selected provider.
`commit(): void`	Signal an end-of-user-turn to the provider.
`interrupt(): void`	Cancel the assistant's in-flight response.
`sendToolResult(callId, output): void`	Return the result of a previously-issued `tool_call`.
`on(handler): () => void`	Subscribe to frames. Returns an unsubscribe callback.
`close(code?, reason?): void`	Close the socket. Idempotent.

`RealtimeFrame` variants

type RealtimeFrame =
  | { type: 'ready'; inputSampleRate: 16000 | 24000; outputSampleRate: 16000 | 24000 }
  | { type: 'audio'; pcm: Uint8Array; sampleRate: number }
  | { type: 'transcript'; role: 'user' | 'assistant'; text: string; final: boolean }
  | { type: 'tool_call'; callId: string; name: string; arguments: string }
  | { type: 'usage'; inputAudioTokens: number; outputAudioTokens: number }
  | { type: 'interruption'; at: 'user' | 'assistant' }
  | { type: 'server_tool_call'; id: string; name: string; status: 'started' | 'completed' | 'failed' }
  | { type: 'error'; code: string; message: string }
  | { type: 'close'; code: number; reason: string };

Example — tool calls

const session = await speko.realtime.connect({
  provider: 'openai',
  model: 'gpt-realtime-2.1',
  tools: [
    {
      name: 'get_weather',
      description: 'Current weather for a city.',
      parameters: {
        type: 'object',
        properties: { city: { type: 'string' } },
        required: ['city'],
      },
    },
  ],
});

session.on(async (frame) => {
  if (frame.type === 'tool_call' && frame.name === 'get_weather') {
    const { city } = JSON.parse(frame.arguments);
    const result = await fetchWeather(city);
    session.sendToolResult(frame.callId, result);
  }
});

Transport notes

Provider authentication. OpenAI uses its ephemeral bearer credential during WebRTC setup; xAI uses its delegated credential as a WebSocket subprotocol; Gemini Live uses its scoped token on the provider URL.
Runtime support. OpenAI requires WebRTC and Web Audio globals. Gemini Live and xAI require a WebSocket global; Node 22 can provide one with globalThis.WebSocket = (await import('ws')).WebSocket;.
WebSocket audio. For Gemini Live and xAI, the SDK forces binaryType = 'arraybuffer'; normalized inbound audio is emitted as Uint8Array.
Missing PCM. Until you call sendAudio, the provider sees no user input. Browser applications should normally use @spekoai/client, which owns microphone capture and playback.

On this page