Realtime voice

A voice call is a session like any other — it has an id, a transcript in its message list, and a brief when it ends — but its conversation runs over a WebSocket: your client streams the user's microphone audio up and plays the assistant's voice as it comes down.

1. Start the session

curl -s "$NEWEYES_API_BASE/v1/realtime/sessions" -X POST \
  -H "Authorization: Bearer $NEWEYES_USER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "object": "realtime_session",
  "id": "5e7a…",
  "client_secret": { "value": "…", "expires_at": 1760861190 },
  "websocket_url": "wss://…",
  "model": "neweyes-voice-1"
}

client_secret.value opens one connection, and only before expires_at (Unix seconds, 90 seconds after it was made). Connect right away.
Starting a voice session needs a spendable balance on the project's billing account.
The body may choose the session's capabilities with capabilities or exclude_capabilities, as for a text session (Choosing a session's capabilities). The voice assistant is told what the session left out, so it does not offer or delegate it.

2. Connect

Open a WebSocket to websocket_url with the secret in the Authorization header:

Authorization: Bearer <client_secret.value>

A browser cannot set that header on a WebSocket, so it offers the secret as a subprotocol instead, next to neweyes-realtime:

const socket = new WebSocket(session.websocket_url, [
  'neweyes-realtime',
  `neweyes-secret.${session.client_secret.value}`,
]);

The server sends {"type": "session.ready"} when the call is ready.

3. Frames

Every frame is a JSON text message with a type.

Client → server

`type`	Fields	What it carries
`input_audio`	`audio`: `data` (base64), `format` (`pcm16`), `sample_rate`	The user's microphone audio: raw 16-bit little-endian PCM, mono. `sample_rate` is 8000, 16000 (the default), 24000, 44100 or 48000. Send it in short chunks, continuously.
`input_image`	`image`: `data` (base64), `format`	An image the user shows, such as a camera frame: `jpeg` (the default) or `jpg`, `png`, `webp`, `gif`, `heic` or `heif`.
`input_text`	`text`	Something the user typed during the call: up to 32,000 characters.
`tool_result`	`tool_call_id`, `name`, `output`	Your answer to a `tool_call` (below).

{ "type": "input_audio", "audio": { "data": "AAABAAIA…", "format": "pcm16", "sample_rate": 16000 } }

Server → client

`type`	Fields	What it carries
`session.ready`	—	The call is ready.
`output_audio`	`audio`: `data` (base64), `format` (`pcm16`), `sample_rate` (24000)	The assistant's voice: raw 16-bit little-endian PCM, mono, at 24 kHz. Play it as it arrives.
`input_transcript`	`text`	What the user said, as it is recognized.
`output_transcript`	`text`	What the assistant says.
`turn_complete`	—	The assistant finished speaking its turn.
`interrupted`	—	The user spoke over the assistant: stop playing the audio you have queued.
`tool_call`	`tool_call_id`, `name`, `arguments`	The assistant asks your client to run a tool (below).

New frame types may be added: ignore a type you do not recognize.

A frame your client sends is dropped, with no reply, when it is over 4 MiB; when its audio or image data is not base64; when its sample_rate or image format is not one listed above; when an input_text is over 32,000 characters or holds a NUL character; or when a tool_result's output is over 32,000 characters as JSON.

4. Tools

The voice assistant hands longer work to the full assistant through two tools, which your client runs and answers with a tool_result carrying the same tool_call_id:

create_task — arguments.task describes the work. Send that text as a message to a text session — POST /v1/sessions, or POST /v1/sessions/{session_id}/messages on one you keep for the call — and answer with the returned task_id. Its progress and reply stream on that session's event stream, as usual. Create that text session with the same capabilities or exclude_capabilities you sent for the call: the call's choice does not carry over to it, and a text session created without one opens every capability.
get_task_status — arguments.task_id names a task (or none, for the latest). Answer with what the event stream has shown for it: still running, or its end — completed with its reply, stopped, or failed.

{ "type": "tool_result", "tool_call_id": "call_1", "name": "create_task", "output": { "task_id": "exec-5a2c…" } }

5. Close codes

The socket has no response body, so how a connection ended is told by its close code:

Code	Meaning	What to do
`1000`	Normal closure, or the session was ended.	Nothing.
`1001`	Going away: one side is ending the session.	Nothing.
`1011`	A server-side error, or the voice service is temporarily unavailable.	Retry later.
`1012`	The server is restarting for a planned update.	Reconnect shortly (step 6).
`4001`	The secret is missing, invalid, expired or already used; the session was deleted or has ended; or the caller's access was revoked.	Get a fresh secret — with the same `session_id`, while the session has not ended.
`4009`	A newer connection took the session over.	Do not reconnect from this one, or the two take it from each other in turn.
`4401`	Authentication failed, or the account is not active.	Check the user and their token.
`4429`	The billing account is out of credits, or too many voice sessions are active.	Once credits are added, or another session ends, retry.

6. Rejoin after a dropped connection

A closed connection does not end the session. To go on with the call, create a realtime session again with its session_id for a fresh secret, and connect again: the conversation picks up where it left off.

curl -s "$NEWEYES_API_BASE/v1/realtime/sessions" -X POST \
  -H "Authorization: Bearer $NEWEYES_USER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"session_id": "5e7a…"}'

The rejoined session keeps the capabilities it was created with: send neither capabilities nor exclude_capabilities, or the request is refused with 422 validation_failed.

One connection holds a session at a time: connecting again takes the session over, and the older connection closes with 4009.

7. End the call

curl -s "$NEWEYES_API_BASE/v1/realtime/sessions/$SESSION_ID/end" -X POST \
  -H "Authorization: Bearer $NEWEYES_USER_TOKEN"

A connection holding the session closes with 1000, and the session's brief is built. Ending a session that has already ended returns it unchanged.

A session left without a connection ends by itself, but never sooner than 10 minutes after its last connection closed — a time that may grow, but will not shrink. A rejoin after that is refused with 409 session_ended: start a new session.

After the call

GET /v1/sessions/{session_id} reads the voice session (type is realtime).
GET /v1/sessions/{session_id}/messages lists its transcript.
GET /v1/sessions/{session_id}/brief returns its brief.

1. Start the session​

2. Connect​

3. Frames​

4. Tools​

5. Close codes​

6. Rejoin after a dropped connection​

7. End the call​

After the call​