ElevenLabs is the voice many AI characters use: natural, expressive and available in many languages. What it doesn't give you is mouth animation. If your character is a 3D model, you need a second step that turns the speech into blend shape weights. This guide shows what ElevenLabs gives you to work with, and the options from quick to natural.
What ElevenLabs returns
The regular text-to-speech endpoint returns only audio. The with-timestamps variant returns the audio and the timing of every character of the text (ElevenLabs docs):
const res = await fetch(`https://api.elevenlabs.io/v1/text-to-speech/${voiceId}/with-timestamps`, {
method: 'POST',
headers: { 'xi-api-key': process.env.ELEVENLABS_API_KEY, 'Content-Type': 'application/json' },
body: JSON.stringify({ text, model_id: 'eleven_multilingual_v2' }),
});
const { audio_base64, alignment } = await res.json();
// alignment.characters ["H", "i", ",", " ", "I", "'", "m", ...]
// alignment.character_start_times_seconds [0.0, 0.07, 0.12, ...]
// alignment.character_end_times_seconds [0.07, 0.12, 0.17, ...]
There's also a streaming version (/stream/with-timestamps) that sends audio and alignment in chunks, which matters for live conversation.
Keep the server call on your back end, so your API key never reaches the browser.
Option 1: word timings for captions
Group the characters between spaces into words, each with the start of its first letter and the end of its last:
function words({ characters, character_start_times_seconds: s, character_end_times_seconds: e }) {
const out = [];
let cur = null;
characters.forEach((ch, i) => {
if (/\s/.test(ch)) { cur = null; return; }
if (!cur) out.push(cur = { word: '', start: s[i], end: e[i] });
cur.word += ch;
cur.end = e[i];
});
return out;
}
That's enough for karaoke-style captions, which visitors with the sound off will thank you for. It isn't lip sync yet.
Option 2: a rough viseme approximation
Letters aren't sounds: "ough" is said four different ways in English. But a rough mapping from letters to mouth shapes, timed by the alignment, gives better than nothing:
m,b,p→ lips closed;f,v→ lower lip under the teeth;o,u,w→ rounded lips;a,e,i→ open or spread mouth.
Expect it to look robotic. Letters miss silent letters, diphthongs and co-articulation, where one sound shapes the next. It's a fallback, not a finish.
Option 3: ARKit curves from the audio and text
For natural speech, use a model that works from sounds (phonemes), not letters. You send the ElevenLabs audio and the text that was spoken. The service aligns them to get when each phoneme starts and ends, then outputs ARKit blend shape weights for every frame:
{
"fps": 30,
"curves": ["jawOpen", "mouthClose", "mouthFunnel", "..."],
"frames": [[0.47, 0.008, 0.002], [0.562, 0.007, 0.002]]
}
This handles what letter rules can't: lips that close on "m" even in fast speech, rounding that starts before "oo", and smooth movement between sounds. Our Lip sync API works this way, and calibrates the curves to the blend shapes your model actually has.
Playing it back on the character
Whatever the source, play the weights against the audio's own clock, not a separate timer. In three.js, read audio.currentTime each frame, interpolate between the two nearest frames of curves, and write the weights into morphTargetInfluences on the head, teeth and tongue meshes. Ease into the first shape before the audio starts and back to rest after it ends. The full code is in our three.js lip sync guide. In Unity, the same weights go into SkinnedMeshRenderer.SetBlendShapeWeight, on a 0–100 scale; see lip sync in Unity.
What about other voices?
- OpenAI text-to-speech returns audio only, with no timestamps. Use option 3: send the audio and text for alignment.
- Azure Speech returns viseme events and blend shapes, but only for its own voices.
- Recorded voice actors: the same as OpenAI, audio plus script.
The rule is the same for all of them. If the voice service doesn't give you phoneme timing, get it from the audio and the text, then turn it into curves.