Events you send

Your page sends these as JSON on the nuance_live topic, one reliable data packet per event (the nuance_live topic). Send nothing before session.started: an event sent earlier can be lost before the AI persona has joined, and no error comes back.

Every event may carry an event_id of your own (up to 512 characters): the events that answer it, an error included, name it as client_event_id.

This page's examples are in one file:

// In the browser: steer a Nuance Live conversation from your page, as things
// happen in it. `send` is the session's, from nuance-live-room.ts.
import type { LiveSession } from "./nuance-live-room";

type Send = LiveSession["send"];

/** News that changes what the persona should be saying: it stops and carries on with the news. */
export function tellThePersonaNow(send: Send, news: string) {
  return send({
    type: "session.thinking.append",
    delegation_id: null,
    content: news,
    while_speaking: "revise",
  });
}

/** Context the persona's next reply reads; the reply it is giving plays on as drafted. */
export function tellThePersonaLater(send: Send, context: string) {
  return send({
    type: "session.thinking.append",
    delegation_id: null,
    content: context,
    while_speaking: "finish",
  });
}

/** Asks the persona to speak now, with a note for that reply. */
export function askThePersonaToSpeak(send: Send, note: string) {
  return send({
    type: "response.create",
    response: { instructions: note },
  });
}

/** Stops the persona's reply where it stands, and keeps the persona quiet until the user speaks. */
export async function stopThePersona(send: Send): Promise<void> {
  await send({
    type: "session.output_audio.interrupt",
  });
  await send({
    type: "session.floor.hold",
    ends_on_speech: true,
  });
}

/** Keeps the persona from taking a turn of its own for `seconds`, and tells it why. */
export function holdTheFloor(send: Send, seconds: number, reason: string) {
  return send({
    type: "session.floor.hold",
    seconds,
    reason,
  });
}

export function releaseTheFloor(send: Send) {
  return send({ type: "session.floor.release" });
}

/** While muted, the persona hears the user's microphone as silence. */
export function muteTheUser(send: Send, muted: boolean) {
  return send({
    type: muted ? "session.input_audio.mute" : "session.input_audio.unmute",
  });
}

/** The user's typed words, then a reply to them (needs a responses delegation). */
export async function sendTyped(send: Send, text: string): Promise<void> {
  await send({
    type: "response.item.create",
    item: { type: "message", role: "user", content: text },
  });
  await send({ type: "response.create" });
}

AppendsCopy Anchor Link

The three appends add a line to the conversation, each with:

• content: up to 2,000 characters.
• delegation_id: null, unless the line reports on a task the persona handed to your page (client delegation). The key is required.
• while_speaking (optional): what the line does to a reply the persona has already started. "finish", the default, lets the persona finish it as drafted. "revise" cuts it and has the persona carry on with the line (below).

{ "type": "session.thinking.append", "event_id": "goal_1", "delegation_id": null, "content": "The user just scored a goal: it's 2 to 1.", "while_speaking": "revise" }

What each one does:

• session.thinking.append is quiet context. While the persona and the user are both quiet, it may speak on the line about half a second later, or let it pass. That needs stay_quiet on (the default), delegation_id: null, and a conversation that has already opened; and the persona leaves the floor to the user while its question waits on their answer. While the user speaks, the line goes in and the persona's next reply reads it. While the persona speaks, "finish" has it read the line after the reply it's giving. When your app needs the persona to answer it, send response.create after the append. An empty content turns off the wait line of the tasks under way.
• session.instructions.append is an instruction, and the persona is owed a turn to act on it. With "finish", it completes the reply it's giving first.
• session.commentary.append is something to tell the user in the persona's own words, at the next opening.

Send app state a reply should read (a tap, what the screen shows) as soon as it changes, before any response.create.

Revise the persona's replyCopy Anchor Link

Something happens in your app while the persona is talking, and the persona shouldn't go on as if nothing had. Send the append with while_speaking: "revise":

• While the persona speaks, it stops at once: its audio fades out, as when the user cuts in, and the conversation keeps only the words that played. The persona's next reply starts straight away with the new line, carrying on from where it stopped without repeating what was heard. The cut reply's session.output_audio.done says revised.
• While the persona's last words are still playing (the runtime has sent the whole reply, and about the last fifth of a second is still due to play), it counts as while the persona speaks: its next reply starts at once, and the reply that played ends completed.
• While the persona is quiet, while the user talks, or while you hold the floor, it is a plain append.

Unlike the user cutting in, there is no wait for anyone's turn to end, and the persona continues rather than starts over. A backend task under way is untouched: only the persona's speech is cut. A response.create sent with a revise asks nothing more, since the revised reply is the answer.

The *.appended event that answers an append says where it landed (start_ms, end_ms). For a revise, it also says revised: true with cut_ms when it cut the persona's reply, false when the persona wasn't speaking.

A line in another script, one that asks the persona to speak another language, one that asks it to name the AI model or company it runs on, or one that asks it to say it is human or to deny being an AI is refused with instructions_refused, and nothing of it reaches the persona. The same goes for response.create's instructions and a hold's reason.

Ask the persona to speakCopy Anchor Link

response.create with nothing pending asks the persona to speak now. The persona never opens a conversation on its own, so this is how it starts:

{ "type": "response.create", "response": { "instructions": "Welcome them back and ask how the trip went." } }

• response.instructions (optional, up to 2,000 characters) is a note the persona reads for that reply only. response.tool_choice: "none" asks the persona to answer itself rather than hand the request off; it's a request, not a guarantee.
• It never cuts a reply. Sent while the persona is talking, it waits for the reply to end, then asks once, however many you send. If the reply the persona is giving already reads what you sent, that reply is the answer.
• With stay_quiet on, the persona is asked once more before it lets the reply pass, but it can still end the reply saying nothing.
• When a backend task waits on your tool results, response.create hands them back instead: see the tool round trip.
• response.conversation: "none" is refused with unsupported_feature.

Typed wordsCopy Anchor Link

The user's typed words go in as a user message, then response.create asks for the persona's answer:

{ "type": "response.item.create", "item": { "type": "message", "role": "user", "content": "Can you make it Friday instead?" } }
{ "type": "response.create" }

content is a string, or input_text parts, up to 24,000 characters. Typed words never cut a reply, and the persona doesn't answer them until your response.create. They aren't echoed in the transcript events, so show them yourself. A message of another role, or an image beside its text, is refused. Typed words need a responses delegation: under client delegation, response.item.create is refused with delegation_type_mismatch.

Stop the personaCopy Anchor Link

session.output_audio.interrupt stops the persona's reply where it stands, and the rest of it is never said. It takes only type and event_id.

{ "type": "session.output_audio.interrupt", "event_id": "stop_1" }

• The persona's transcript stops, and session.output_audio.done follows with reason: "interrupted" once its audio has ended.
• With nothing to stop, it's refused with command_refused. A stop sent just after the persona's reply ended gets that too.
• A response.create waiting for that reply is dropped with it; its instructions still reach the persona's next reply. One you send after the stop asks for a new reply, so a stop then a response.create has the persona say something else.
• It doesn't keep the persona quiet afterwards: the persona answers the user's next words as usual. Send session.floor.hold after it to keep the persona from speaking.

Use the stop when the user pressed stop, or the screen changed and there is nothing to add. When new information changes what the persona should be saying, send it as an append with while_speaking: "revise" instead: the line and the cut arrive together, and the persona carries on from what it actually said.

Hold the floorCopy Anchor Link

session.floor.hold keeps the persona from taking a turn of its own until session.floor.release:

{ "type": "session.floor.hold", "seconds": 30, "reason": "The user is reading the results." }
{ "type": "session.floor.release" }

• Bare, it's a pause the persona isn't told of, and the user's words don't end it.
• With seconds (1 to 3,600), ends_on_speech or reason (up to 200 characters), it's a hold the persona is told of. It runs out after seconds, and the user's words end it unless ends_on_speech is false. When a timed hold runs out the persona may check in once.
• A newer hold replaces the one under way. A hold doesn't stop a reply already playing: send session.output_audio.interrupt for that.
• session.quiet.started and session.quiet.ended report it, with by: "client".

Mute the userCopy Anchor Link

session.input_audio.mute has the persona hear the user's microphone as silence of the same length, so the user can't cut the persona off or end a turn; the track stays published. session.input_audio.unmute turns it back on. session.input_audio.muted and .unmuted answer them.

Change the delegationCopy Anchor Link

session.update changes the delegation's settings during the session, and nothing else:

{ "type": "session.update", "session": { "delegation": { "type": "responses", "responses": { "tool_choice": "none" } } } }

Send the same type. The settings you name replace the live ones, and a setting you leave out, or send as null, keeps its value:

• tool_choice, parallel_tool_calls, max_output_tokens and reasoning change how the backend works from its next step.
• The backend's instructions and tools were reviewed at mint, so an update may send them only as minted: instructions exactly as at mint, and tools as a list of tools the session started with, each exactly as you listed it, to take some out and put them back (tools that aren't available right now).

A changed type, changed instructions, a tool the session didn't start with, or a setting fixed at startup is refused with immutable_field_update; a key the session doesn't know, with unknown_parameter. Either way, none of the update applies. An update must fit in one packet (64,000 bytes), never as a text stream, and the session it leaves must still be echoed in one 15 KiB packet; otherwise it is refused with event_too_large, and the session runs on as before. session.updated answers with the session as it now stands.

Hang upCopy Anchor Link

session.close ends the session. session.closed follows with reason: "close_requested", then the events route's ended.