Lip sync in three.js: driving ARKit blend shapes from speech

A talking character in three.js needs three things: a model with mouth blend shapes, a stream of weights for those shapes over time, and a clock that keeps the weights in step with the audio. This guide covers all three, with code you can drop into a project.

1. Find the mouth shapes

Load the model and collect every mesh that has morph targets. Head, teeth and tongue are often separate meshes, and all of them need the same weights, or the teeth stay still while the jaw opens.

import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';

const gltf = await new GLTFLoader().loadAsync('/character.glb');
const faceMeshes = [];
gltf.scene.traverse((o) => {
  if (o.isMesh && o.morphTargetDictionary) faceMeshes.push(o);
});
console.log(faceMeshes.map((m) => [m.name, Object.keys(m.morphTargetDictionary)]));

If you see names like jawOpen, mouthFunnel and mouthPressLeft, the model uses ARKit blend shapes. If you see viseme_aa, viseme_PP and so on, it uses the Oculus viseme set.

2. Choose where the mouth motion comes from

There are four common sources, from simplest to best:

The rest of this guide uses the last one, because the playback code is the same however the curves are made. A clip of curves looks like this:

{
  "fps": 30,
  "names": ["jawOpen", "mouthClose", "mouthFunnel", "..."],
  "frames": [[0.47, 0.008, 0.002], [0.562, 0.007, 0.002]]
}

One row per frame, one column per shape. The numbers above are real output from our custom lip sync model for "Hi, I'm Riya".

3. Play the curves against the audio clock

The most common mistake is timing the animation with requestAnimationFrame time and starting it at the same moment as audio.play(). Audio starts with a delay you don't control, so the mouth runs ahead. Use the audio's own time:

const index = faceMeshes.map((m) => curves.names.map((n) => m.morphTargetDictionary[n]));

function applyAt(t) {
  const f = t * curves.fps;
  const last = curves.frames.length - 1;
  const i0 = Math.min(last, Math.floor(f));
  const i1 = Math.min(last, i0 + 1);
  const k = f - Math.floor(f);
  faceMeshes.forEach((mesh, m) => {
    curves.names.forEach((_, c) => {
      const slot = index[m][c];
      if (slot === undefined) return;
      const w = curves.frames[i0][c] * (1 - k) + curves.frames[i1][c] * k;
      mesh.morphTargetInfluences[slot] = w;
    });
  });
}

renderer.setAnimationLoop(() => {
  if (!audio.paused) applyAt(audio.currentTime);
  renderer.render(scene, camera);
});

Interpolating between frames matters: curves at 30 fps rendered at 60 or 120 fps would otherwise step visibly.

4. Remove the jumps

Even good curves look cheap if the mouth snaps. Three places cause snaps:

• The first frame. Speech often starts with the mouth already moving: the first frame might have the jaw half open. Ease from the neutral face into the first frame over about 0.3 s before starting the audio.
• The last frame. Ease back to neutral over 0.2–0.3 s after the audio ends, instead of setting every weight to 0.
• Clock hand-offs. When playback moves from a timer to audio.currentTime, time can jump by tens of milliseconds. A light follow filter hides it:

// dt: seconds since the last frame; target: the weight the curves ask for now
const follow = 1 - Math.exp(-dt / 0.035);
weight += (target - weight) * follow;

We use exactly this on the talking head on our home page: a 0.35 s lead-in, a 0.3 s tail and a 35 ms follow filter.

5. Don't forget the rest of the face

Lips alone look like a puppet. Add, on their own timing:

• Blinks every 2.5–5.5 s, faster while talking.
• Small head motion: a slow sway, a nod on stressed words.
• Eye contact with small saccades rather than a fixed stare.
• Brows that follow the emotion of the sentence, not the phonemes.

Getting curves for your own character

Making good ARKit curves is the hard part. You can capture them with an iPhone (Live Link Face) or request them from an API. Our Lip sync API takes the text with its audio or phoneme timings and returns ARKit curves fitted to your model's own blend shapes, ready for the playback code above.