Set up a session

Mint a session tokenCopy Anchor Link

Your server mints the token with POST /session-tokens and your API key, as for any conversation. A Nuance Live token also names the interface, and settles the session in interface_setup:

// On your server: a session token for a Nuance Live conversation whose
// backend can run one of your tools.
const NUANCE_API = process.env.NUANCE_API_URL ?? "https://api.nuancelabs.ai";

const GET_WEATHER = {
  type: "function",
  name: "get_weather",
  description: "Looks up the current weather in a city.",
  parameters: {
    type: "object",
    properties: { city: { type: "string" } },
    required: ["city"],
  },
};

/** Mints a token for `endUserId`, your id for the signed-in person. */
export async function mintNuanceLiveToken(endUserId: string): Promise<string> {
  const response = await fetch(`${NUANCE_API}/session-tokens`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.NUANCE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      interface: "nuance_live",
      // The persona's instructions: say when it hands work to its backend.
      personality: {
        system_prompt:
          "You are Sam, a friendly weather assistant. When the user asks about " +
          "the weather, hand the question to your backend, then tell them what it found.",
      },
      interface_setup: {
        delegation: {
          type: "responses",
          responses: {
            // Required: any non-empty name.
            model: "default",
            instructions:
              "Use get_weather for any weather question, then answer in one short sentence.",
            tools: [GET_WEATHER],
          },
        },
        // The persona may hang up after its goodbye.
        runtime_tools: { end_call: true },
      },
      end_user_id: endUserId,
    }),
  });
  if (!response.ok) {
    throw new Error(
      `Couldn't mint a token: ${response.status} ${await response.text()}`,
    );
  }
  // `warnings` names each tool the session will run without, and why.
  const { token } = (await response.json()) as { token: string };
  return token;
}

• The token fixes the session. The page can't change any of this later, except the delegation's settings that session.update allows.
• Tools go in the delegation. A Nuance Live session never runs the token's top-level tools; the token's warnings say so if you send them.
• expressiveness is a number from 0 to 2. Anything else, such as 2.5 or "1", is refused with 422, and detail names expressiveness.
• interface_setup takes only these three keys. Any other key is refused with 422, code interface_setup, and a detail that names it. So is a delegation the session couldn't run, such as one missing its model.
• The backend's instructions and tools are reviewed against the Acceptable Use Policy at mint, as your prompt is. One the review refuses answers 400, and the detail names interface_setup.delegation.responses.

The token starts one conversation, within 60 seconds.

Write the promptsCopy Anchor Link

• The persona's prompt names the requests to hand off. The persona decides on its own, and hands off when its prompt tells it to. Without that, it may talk about doing something and never hand it off.
• Have the persona say what it's doing, not that it's done: "Booking it now…", not "It's booked!". The persona's words play before your tool runs, and the result comes after.
• The delegation's instructions tell the backend how to use your tools and how to answer.

The persona's own toolsCopy Anchor Link

interface_setup.runtime_tools turns the persona's own tools on or off, each with true or false. The persona runs them on the server: no event asks your page to run them.

• A tool runs only where the service offers it.
• null is the same as leaving the tool out. Any other key, or a value that isn't a boolean, is refused with 422 interface_setup.
• session.started echoes runtime_tools as the session runs them: always stay_quiet, and end_call and call_time when you named them. A tool you named that the service doesn't offer reads false.

Start the conversationCopy Anchor Link

The rest is the same as for any conversation, with the session token as the bearer:

1. POST /conversation with the person's timezone creates it: the persona reads the date and time in that zone, else in UTC.
2. Follow GET /conversation/{id}/events until admitted, and keep one request open until the conversation ends.
3. Join the admitted room and publish the microphone right away.

Connect has each step.

session.started comes once the persona can hear the user's microphone, so the user can speak the moment it arrives. Publish the microphone when you join: without one, session.started may not come at all. Send audio for the whole session, as a microphone does: when the user says nothing, that is silence.

The nuance_live topicCopy Anchor Link

Nuance Live events travel on the room's data channel, on the topic nuance_live, as JSON: one reliable data packet per event, both ways, in the order sent.

• What you send may be up to 64,000 bytes a packet. Send a larger event, such as a tool result with images, as a LiveKit text stream on the same topic. A response.create you send after a stream waits until the runtime has read it.

• What you get comes one event a packet, up to 15 KiB. A larger event arrives as consecutive message.part packets, before anything else:

  Copy Text

  { "type": "message.part", "id": "evt_41", "index": 0, "count": 2, "text": "{\"type\":\"response.event\",…" }

  Join the texts in index order and parse the result at the part whose index is count - 1. If any other packet arrives between parts, or a part is out of order, drop the event you were joining.

• Clock sync. The room's clock-sync topic carries {"type": "clock_sync_ping", "seq": …, "t0": …}. Answer each at once, reliably, with {"type": "clock_sync_pong", "seq": …, "t0": …, "t1": …, "t2": …}: seq and t0 copied, t1 when the ping arrived and t2 as you reply, in Unix seconds. The session waits up to half a second for the replies before it starts.

• Ignore other topics, and event types and fields you don't know: new ones are added.

End itCopy Anchor Link

Send {"type": "session.close"}, then wait a few seconds for session.closed (or the events route's ended, if it comes first), and leave the room. When the session ends by itself, session.closed says why: see Events you get.