# MeshLite by Toda Labs

> MeshLite by Toda Labs adds a photoreal, real-time conversational AI avatar to a website or app. Live lip-sync runs in the visitor’s browser, with no MeshLite server inference per speaking stream, so avatar minutes are unlimited at a flat monthly or yearly price, sized by simultaneous sessions or monthly users. Voice and the LLM run on providers you choose.

MeshLite adds a live, speaking AI avatar to a website or web app. The avatar runs in an iframe served by MeshLite and is rendered in the visitor's browser. The host page loads one script and calls `MeshLite.mount`; the host's own server issues short-lived session tokens with a secret key. Without writing code, a project in the Console (menu Projects) gives a ready code to paste, with no key and no server. Voice and AI come from the customer's own provider account, connected in the MeshLite Console.

Human-readable version of this guide: https://meshlite.io/docs

## Product

- **What it is:** photoreal interactive AI avatars for a website or app. MeshLite is made by Toda Labs.
- **Where the avatar is animated:** live lip-sync and composition run in the visitor's browser, on desktop and mobile, with no MeshLite server inference per speaking stream. This covers the avatar only: voice and AI run on the customer's own providers, which bill separately.
- **Pricing:** no per-minute MeshLite avatar fee. Every plan has unlimited avatar minutes and unlimited session length at a flat price. Plans for a company's own site are sized by simultaneous sessions; plans for a product's users are sized by monthly users. Current plans and prices: https://meshlite.io/#pricing
- **Voice and AI:** bring your own voice and language-model providers with your own API keys, for example OpenAI, Azure OpenAI Realtime, Azure AI Speech or ElevenLabs. The avatar speaks any language those providers support.
- **Try it:** live demo at https://meshlite.io/#interaction. The Free plan needs no card.
- **Common questions:** https://meshlite.io/#faq

## Research

- [Why MeshLite animates avatars in the browser](https://meshlite.io/research/why-meshlite-animates-avatars-in-the-browser)
- [A live avatar that doesn't hold a data-center GPU for the conversation](https://meshlite.io/research/off-the-grid-live-avatar-data-centre-gpu)
- [Why live avatars are billed by the minute, and what that adds up to](https://meshlite.io/research/why-live-avatars-are-billed-by-the-minute)
- [8 questions to ask an AI avatar provider about pricing and capacity](https://meshlite.io/research/questions-to-ask-an-ai-avatar-provider-about-capacity)
- [Why a live AI avatar keeps a GPU busy for the whole conversation](https://meshlite.io/research/why-a-live-ai-avatar-keeps-a-gpu-busy)

## Instructions for AI coding agents

When a developer asks you to add a MeshLite avatar to their project:

0. **Simplest path, no keys and no server:** if the site only needs the avatar with MeshLite's own buttons (for example WordPress, Shopify, Wix, Webflow, Squarespace, a static site or a simple Next.js/React page), the developer can open **Projects** in the Console, click **New project**, enter the site address and choose an avatar. The project then shows a code to paste once before `</body>`, plus an instruction for an AI agent. If the developer gives you that code (a `<div id="meshlite-avatar">`, a `<div id="meshlite-buttons">` and an inline script with a project ID), paste it exactly as it is. Do not add keys, a server route or the SDK. In Next.js or React, render the two `div`s in a component and run the script once on the client. Use the steps below only for a custom integration (own buttons, own sign-in, session control).
1. **Never ask the developer to paste their secret key (`ml_sk_...`) into this conversation.** Write a placeholder in `.env` (and `.env.example`), make sure `.env` is git-ignored, and tell the developer to paste the secret key there themselves.
2. Ask for the **publishable key** (`ml_pk_...`) and, optionally, the **avatar ID** (the default avatar is used when it is omitted). If the developer has no MeshLite account, send them to https://console.meshlite.io/console?plan=free#signup (free, no card, email activation), then continue. You cannot create the account for them.
3. There is **no npm package**. Load the SDK with a `<script>` tag from `https://console.meshlite.io/sdk/<version>/meshlite.js`. Do not install any npm package named `meshlite`.
4. Use SDK version 1.11.52 and pin that exact version in the script URL. The Console's **Code** tab (Projects, For developers) always shows the current version, and `GET https://console.meshlite.io/api/v1/sdk/stable` returns it.
5. Add a server endpoint that creates session tokens (see "Server"), then mount the avatar in the page (see "Browser"). Start the conversation from a click: browsers only open the microphone after a visitor gesture.
6. Tell the developer which Console settings they still need (see "Console setup"), in particular the exact origin they develop on, such as `http://localhost:3000`.

## Console setup

In https://console.meshlite.io/console. The developer settings are in **Projects**, under **For developers** (tabs **Keys**, **Allowed sites** and **Code**). Voice services and personalities have their own menus:

- **Allowed sites:** add every origin that will embed the avatar, exactly (scheme, host and port, no path). Test origins: `http://localhost:<port>`, `http://127.0.0.1:<port>` or any `https://` staging host. Live origins must use HTTPS.
- **Keys:** create a publishable key and a secret key in the **same environment**. Test keys (`ml_pk_test_`, `ml_sk_test_`) work only with test origins; live keys (`ml_pk_live_`, `ml_sk_live_`) only with live origins. On a public page (not localhost or a private address), a test key shows a MeshLite label on the avatar.
- **Services** (in the menu): connect a voice provider with the developer's own key: OpenAI, Azure OpenAI Realtime, Azure AI Speech or ElevenLabs. Every account also gets 5 minutes of MeshLite voice a day (reset at midnight UTC), one allowance per account shared by all its site visitors and the Console, to help during setup; after that, the avatar speaks through the developer's provider, on every plan.
- **Library**, tab **Personalities:** write what the avatar knows and how it speaks, publish it, then pick it on the avatar in the **Avatars** menu.
- **Code:** shows the avatar ID and this code with the workspace's own values.

**Free plan:** can go live on one site (a domain with its www address and all its subdomains), with default avatars, the MeshLite watermark on the avatar and up to 3 simultaneous conversations. Solo adds one custom avatar and keeps the watermark; Creator and higher plans remove it (Lite keeps a MeshLite attribution): https://meshlite.io/#pricing

## Environment variables

```
MESHLITE_SECRET_KEY=ml_sk_test_...      # server only, never in browser code or Git
MESHLITE_PUBLISHABLE_KEY=ml_pk_test_...
MESHLITE_ORIGIN=http://localhost:3000   # the exact origin registered under Allowed sites
MESHLITE_AVATAR=                        # optional; default avatar when empty
```

## Server: create a session token

Node.js 18+, no dependency. Call it from a `POST /api/meshlite-session` route and return `{ sessionToken }` as JSON. The origin comes from server configuration, never from the browser request. Tokens expire after 2 minutes; the SDK asks for a new one when it needs it.

```js
export async function createMeshLiteSession() {
  const response = await fetch("https://console.meshlite.io/api/v1/client-sessions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.MESHLITE_SECRET_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      publishableKey: process.env.MESHLITE_PUBLISHABLE_KEY,
      origin: process.env.MESHLITE_ORIGIN,
      avatar: process.env.MESHLITE_AVATAR || undefined,
      conversation: { language: "en", responseStyle: "concise", idleTimeout: 30 },
    }),
  });
  const body = await response.json();
  // Errors look like { error: { code: "ORIGIN_NOT_ALLOWED", message, requestId } }
  if (!response.ok) throw new Error(`MeshLite ${body.error?.code}: ${body.error?.message}`);
  return body.sessionToken;
}
```

This route is public: rate-limit it, or require your own sign-in if only signed-in users should talk to the avatar. Conversations count toward the plan's limit of simultaneous conversations.

## Browser: mount the avatar

```html
<div id="avatar" style="width:340px;height:510px"></div>
<button id="talk">Talk to the avatar</button>
<script src="https://console.meshlite.io/sdk/1.11.52/meshlite.js" crossorigin="anonymous"></script>
<script>
  const avatar = MeshLite.mount({
    container: "#avatar",
    avatar: "YOUR_AVATAR_ID",          // same value as MESHLITE_AVATAR; remove for the default avatar
    locale: "en",
    publishableKey: "ml_pk_test_...",  // safe to expose
    sessionToken: async () => {
      const response = await fetch("/api/meshlite-session", { method: "POST" });
      if (!response.ok) throw new Error("Could not start the avatar session");
      return (await response.json()).sessionToken;
    },
    conversation: { language: "en", responseStyle: "concise", idleTimeout: 30 },
    captions: "overlay",
  });
  avatar.on("error", (message, detail) => console.error("MeshLite", detail.code, message));
  document.getElementById("talk").addEventListener("click", () => avatar.startConversation());
</script>
```

The iframe takes the size of its container and the avatar reframes itself to fit (portrait, square or landscape).

**React, Next.js, Vue:** load the script once, mount in a client-side effect (`useEffect`, `onMounted`) and call `avatar.unmount()` in the cleanup. Do not mount during server-side rendering.

## Host page requirements

- HTTPS in production.
- Content Security Policy, if the site has one: `script-src https://console.meshlite.io` and `frame-src https://console.meshlite.io`. The code to paste from a project also needs `connect-src https://console.meshlite.io`.
- Permissions-Policy, if the site sends one: allow the microphone for the avatar frame, for example `microphone=(self "https://console.meshlite.io")`.
- Older browsers that can't run the avatar show an idle preview only, with no microphone. Check `MeshLite.supportsConversation()`.

## Going live

In **Projects**, under **For developers**, open **Allowed sites** and add the production domain as a live origin (HTTPS), create live keys and replace the test values in the environment variables. No code change is needed. The Free plan goes live on one site with the MeshLite watermark; paid plans add custom avatars, and from Creator up they also remove the watermark and allow more simultaneous conversations (Lite keeps a MeshLite attribution): https://meshlite.io/#pricing

## Common errors

- `KEY_INVALID`: the key is wrong or revoked.
- `ENVIRONMENT_MISMATCH`: the publishable and secret keys come from different environments (test and live).
- `ORIGIN_NOT_ALLOWED`: `MESHLITE_ORIGIN` is not registered under Allowed sites (Projects, For developers) in the keys' environment, or the page runs on a different origin (check the port).
- `ORIGIN_HTTPS_REQUIRED`: HTTP is only accepted for localhost test origins.
- `AVATAR_NOT_ALLOWED`: the avatar ID is wrong or not available on the plan.
- `VOICE_PROVIDER_REQUIRED` / `PROVIDER_NOT_CONNECTED`: connect a voice provider under Services.
- `CONCURRENCY_LIMIT`: the plan's simultaneous conversations are all in use.
- `DOMAIN_LIMIT`: the plan's number of sites is reached; another site needs a higher plan.

## `MeshLite.mount(options)`

- `container` (required): CSS selector or element.
- `avatar`: avatar ID from the Console.
- `locale`: interface language, `en`, `fr` or `zh-Hans`.
- `conversation.language`: `en`, `fr` or `zh-CN`. Default `fr`, so set it explicitly.
- `conversation.responseStyle`: `brief`, `concise` or `standard`.
- `conversation.idleTimeout`: `10`, `30`, `60` or `120` seconds.
- `captions`: `off`, `events` or `overlay`.
- `publishableKey` + `sessionToken`: provided together; `sessionToken` is a string or an async function called whenever a token is needed.
- `appearance`: `seamless` for flat controls without a frame.
- `background`: `opaque` for the full source background (default `alpha`).
- `controls`: `external` when the host page draws its own buttons.
- `autoStart`: `true` to start when mounted inside a visitor click.
- `preview`: `idle` for a muted 480p loop before speech loads.
- `languageControl`: `external` when the host page provides its own language selector (with `setLanguage`).
- `newSession`: `true` to start a new session instead of resuming the tab's session.
- `labels`: partial override of the interface labels. `title`: accessible title of the iframe.

## Controller

- `on(event, callback)` / `off(event, callback)`: `ready`, `visual-ready`, `preview-ready`, `speech-started`, `speech-ended`, `state`, `transcript`, `status`, `error`, `language-changed`, `framing-changed`, `session-start`, `user-turn`, `response-start`, `response-complete`, `idle-pause`, `session-resume`, `session-end`, `loading-progress`.
- `startConversation()`: opens the microphone and starts talking (call from a visitor gesture).
- `prepareConversation()`: preloads speech without opening the microphone.
- `say(text)`: sends a text turn; the avatar answers it (it does not read the text aloud verbatim).
- `micOff()`, `stopSpeaking()`, `setLanguage(lang)`, `endDemo(reason?)`, `getState()`, `getFraming()`, `refreshFraming()`, `unmount()`.
- `MeshLite.version`, `MeshLite.languages` and `MeshLite.supportsConversation()` help the host page adapt.

## LiveKit agents

The avatar can lip-sync the voice of the developer's own LiveKit agent, which keeps its own brain and voice; no MeshLite voice minutes are used and a session lasts at most 2 hours. Full guide: https://github.com/meshlite-avatar/livekit-plugins-meshlite

- Agent side (Python, `livekit-agents` 1.x): `pip install livekit-plugins-meshlite`, then `avatar = meshlite.AvatarSession()` and `await avatar.start(session, room=ctx.room)` after `ctx.connect()` and before `session.start()`. The agent then publishes no audio track: its voice reaches the page as LiveKit data. Disable false-interruption resume if the agent version offers it: `AgentSession(..., resume_false_interruption=False)`.
- Server: create the MeshLite session as usual (`POST /api/v1/client-sessions`) with `audio: "livekit"` added to the body.
- Page (`livekit-client` 2.9 or later, SDK 1.11.45 or later): `MeshLite.mount({ ..., audio: "livekit" })`, then `const link = avatar.attachLiveKit(room)` before `room.connect(...)`. Do not subscribe the agent audio yourself: the SDK plays the voice. `link` emits `ready`, `fallback` (the face is unavailable and the page keeps playing the voice) and `audio-blocked` (call `link.resumeAudio()` from a click).
- Limits: LiveKit recordings (egress) do not contain the agent voice.

### LiveKit quick start (local demo, from zero)

Human guide with the same steps: https://meshlite.io/docs#livekit

1. Console (Projects, For developers): in Allowed sites add `http://localhost:8080` as a test site; in Keys create a test publishable key (`ml_pk_test_...`) and a test secret key (`ml_sk_test_...`); in Code copy the avatar ID. Never ask the user to paste the secret key in the chat: the user writes it in `.env`.
2. Get the example: `git clone https://github.com/meshlite-avatar/livekit-plugins-meshlite`, then `cd livekit-plugins-meshlite/examples/browser-avatar` and `cp .env.example .env` (Windows: `copy .env.example .env`).
3. Fill `.env`: `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` (a LiveKit Cloud project, or `ws://localhost:7880` with `devkey` and `secret` for a local `livekit-server --dev` that the launcher starts), `OPENAI_API_KEY` (the example agent; for Azure OpenAI set `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_DEPLOYMENT` instead), `MESHLITE_PUBLISHABLE_KEY`, `MESHLITE_SECRET_KEY`, `MESHLITE_AVATAR`, `PUBLIC_ORIGIN=http://localhost:8080` (and `PORT` if 8080 is taken).
4. Run `python run.py` (Python 3.9 or later; Windows, macOS, Linux). It creates `.venv`, installs the dependencies, checks `.env` (it names missing or wrong settings and never prints a value; `python run.py --check` only checks), starts the agent and the web server with `[agent]` and `[server]` log prefixes, waits for the agent to connect to LiveKit, opens http://localhost:8080 and stops everything on Ctrl+C. Expected log: `[agent] registered worker`, then `[run] Ready: http://localhost:8080`.
5. In the page press Start and allow the microphone: the status reads "Avatar ready", then the face greets the visitor. Chrome, Edge and Safari play the voice in the avatar frame; Firefox plays it in the page and the avatar animates the lips silently (by design).
6. Production: live keys, a live HTTPS site in Allowed sites, `PUBLIC_ORIGIN` set to the HTTPS address, `/api/session` behind sign-in or a rate limit (it creates a conversation), and the agent run as a LiveKit worker (`python agent.py start`).
7. Troubleshooting: `ORIGIN_NOT_ALLOWED` (page origin, port included, missing in Allowed sites), `ENVIRONMENT_MISMATCH` (test and live values mixed), no sound (browsers need a click: show an "Enable sound" button on `audio-blocked` and call `link.resumeAudio()`), microphone (HTTPS or `http://localhost`, after a click), silent agent (the log must say `registered worker`; check `OPENAI_API_KEY` or the `AZURE_OPENAI_*` settings; an Azure `Virtual Network/Firewall rules` error means the resource does not allow this machine).

## Pipecat bots and other WebRTC voice agents

The avatar can lip-sync the voice of the developer's own Pipecat bot (or any WebRTC voice agent), which keeps its own brain, voice and transport. Nothing changes in the bot: no plugin, no extra participant. The page hands the bot's audio track to the SDK, which animates the face in the browser. No MeshLite voice minutes are used and a session lasts at most 2 hours.

- Server: create the MeshLite session as usual (`POST /api/v1/client-sessions`) with `audio: "track"` added to the body (not combinable with `profile`; a `track` session is not a `livekit` session).
- Page with Pipecat (`@pipecat-ai/client-js` 1.x, any transport: SmallWebRTC, Daily, ...; SDK 1.11.48 or later, older SDKs get `409 SDK_UPGRADE_REQUIRED`; RTVI on, the default): `MeshLite.mount({ ..., audio: "track" })`, then `const link = avatar.attachPipecat(client)` before `client.connect(...)`. Keep the page's usual bot audio element: the SDK mutes it while the face plays the voice and unmutes it when the face cannot, so the visitor hears the bot once. `link` emits `ready` (with `output`: `"frame"` or `"page"`), `fallback` (the face is unavailable and the voice continues), `audio-blocked` (call `link.resumeAudio()` from a click), `started`, `finished`, `stopped` and `failed`.
- Other WebRTC agents: `const link = avatar.attachAudioTrack(track)` with the agent's remote audio track; optionally forward the agent's events with `link.hint("bot-started" | "bot-stopped" | "bot-interrupted" | "user-started" | "user-stopped")` and `link.interrupt()`; `link.setTrack(newTrack)` follows a renegotiated track.
- Options of both methods: `output` (`"auto"` default: `"page"` on Firefox, `"frame"` elsewhere), `fallback` (default `true`), `threshold` (dBFS, default -45), `endSilenceMs` (default 600), `muteOthers` (default `true`), `interruptTail` (default `true`).
- Latency: the face starts about 0.6 s after the voice reaches the page (a live track has no "segment ready" signal); until then the page plays the voice itself. Interruptions stop the face at once and the rest of the cut answer is not replayed.

### Pipecat quick start (local demo, from zero)

Human guide with the same steps: https://meshlite.io/docs#pipecat

1. Console (Projects, For developers): in Allowed sites add `http://localhost:8080` as a test site; in Keys create a test publishable key (`ml_pk_test_...`) and a test secret key (`ml_sk_test_...`); in Code copy the avatar ID. Never ask the user to paste the secret key in the chat: the user writes it in `.env`.
2. Get the example: `git clone https://github.com/meshlite-avatar/livekit-plugins-meshlite`, then `cd livekit-plugins-meshlite/examples/pipecat-browser-avatar` and `cp .env.example .env` (Windows: `copy .env.example .env`). It uses no LiveKit.
3. Fill `.env`: `OPENAI_API_KEY` (the example bot; for Azure OpenAI set `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_DEPLOYMENT` instead), `MESHLITE_PUBLISHABLE_KEY`, `MESHLITE_SECRET_KEY`, `MESHLITE_AVATAR`, `PUBLIC_ORIGIN=http://localhost:8080` (and `PORT` if 8080 is taken).
4. Run `python run.py` (Python 3.11 or later, as Pipecat requires; Windows, macOS, Linux). It creates `.venv`, installs the dependencies, checks `.env` (it names missing or wrong settings and never prints a value; `python run.py --check` only checks), starts the bot with `[bot]` log prefixes, opens http://localhost:8080 and stops everything on Ctrl+C. Expected log: `[run] Ready: http://localhost:8080`, then `[bot] INFO    Visitor connected: the bot greets them.` once the visitor presses Start.
5. In the page press Start and allow the microphone: the status reads "Avatar ready", then the face greets the visitor. Chrome, Edge and Safari play the voice in the avatar frame; Firefox plays it in the page and the avatar animates the lips silently (by design).
6. Production: live keys, a live HTTPS site in Allowed sites, `PUBLIC_ORIGIN` set to the HTTPS address, `/api/session` and `/api/offer` behind sign-in or a rate limit (they create a conversation and start a bot), and a STUN/TURN server or another Pipecat transport (such as Daily) when the bot runs behind a firewall or NAT.
7. Troubleshooting: `meshlite_session_refused` with `ORIGIN_NOT_ALLOWED` (page origin, port included, missing in Allowed sites) or `ENVIRONMENT_MISMATCH` (test and live values mixed), no sound (show an "Enable sound" button on `audio-blocked` and call `link.resumeAudio()`), silent bot (check `OPENAI_API_KEY` or the `AZURE_OPENAI_*` settings and the `[bot]` lines; an Azure `Virtual Network/Firewall rules` error means the resource does not allow this machine), "Avatar unavailable" (the voice continues; the code says why, for example `SDK_UPGRADE_REQUIRED`).

## Versions

- The current SDK is 1.11.52. Only recent SDK versions are served. Older version URLs redirect (301) to the current one, so update the pinned version when a new one is released.
- TypeScript types: `https://console.meshlite.io/sdk/<version>/meshlite.d.ts`.

## Links

- Quickstart: https://meshlite.io/docs
- Sign up (Free): https://console.meshlite.io/console?plan=free#signup
- Console sign-in: https://console.meshlite.io/console
- Pricing: https://meshlite.io/#pricing
- Live demo: https://meshlite.io/#interaction
- Contact: hello@meshlite.io