Search for "AI lip sync" and you'll find two kinds of tool that barely overlap. One edits video: it takes a clip of a real person and redraws the mouth so it matches new audio. The other animates 3D characters: it produces numbers that move a model's face. Picking the wrong kind wastes days, so it's worth knowing the difference.
Video lip sync: editing the pixels
What it does: takes a video of a face and an audio track, and generates new frames where the mouth area matches the audio. The rest of the face is kept from the original footage.
Used for: dubbing videos into other languages, fixing a line in post-production, talking-head clips from a photo, and social video.
Main tools:
- Wav2Lip (2020): the research model that started the field. Fast and light, still a common accuracy baseline, but low resolution around the mouth.
- LatentSync (ByteDance, December 2024): a diffusion-based model with sharper results, at the cost of more GPU memory.
- MuseTalk: open source, built for real-time use.
- sync.so: a hosted API from the Wav2Lip team, aimed at products and production work.
- All-in-one platforms such as HeyGen add lip sync to translation and avatar video tools.
Limits: it needs a GPU and real footage, and it renders frames, so the output is video. You can't rotate the face, relight it or put it in a game engine.
3D lip sync: driving a character's face
What it does: takes speech (audio, text, or both) and outputs animation curves. For each frame, these are weights for the face's blend shapes, such as jawOpen 0.42 and mouthFunnel 0.18. The engine applies them to the model, which renders the face itself.
Used for: game characters, VTubers, AI assistants and companions, virtual receptionists, e-learning presenters, and any 3D avatar on a website.
Output formats:
- ARKit blend shapes: the 52 named shapes from Apple's face tracking, now the common standard. See ARKit blend shapes explained.
- Visemes: one mouth pose per group of sounds, such as the 15 Oculus visemes.
- Engine-specific rigs: MetaHuman controls, VRM vowels, and others.
Main approaches: audio analysis plugins (such as uLipSync and SALSA in Unity), viseme events from a TTS service (such as Azure Speech), capturing a performer with an iPhone, engine features such as MetaHuman's audio-driven animation, and lip sync APIs that return curves.
Strengths: the face stays fully 3D. It can turn, be lit, show expressions and keep eye contact, and it runs in real time on ordinary devices, because only a few numbers per frame are sent.
Which one do you need?
| You have... | You need | Output |
|---|---|---|
| Video of a real person, new audio | Video lip sync | A new video |
| A photo, and want a talking clip | Video lip sync / avatar video tool | A video |
| A 3D character in a game or engine | 3D lip sync | Blend shape curves or visemes |
| A 3D avatar on a website, talking live | 3D lip sync, fast enough for real time | Curves streamed with the voice |
| A VTuber model (VRM) | 3D lip sync, VRM output | Vowel weights |
A quick test: if the result is a video file, it's video lip sync. If the result drives a model, it's 3D lip sync.
What makes 3D lip sync look right
Whatever the tool, viewers judge the same few things:
- Closures: on "p", "b" and "m" the lips must actually meet, even in fast speech.
- Rounding: "oo" and "w" need the lips pushed forward, starting slightly before the sound.
- Timing: the mouth should be in step with the audio clock, not a separate timer. Drift of a tenth of a second is visible.
- Smooth starts and ends: easing into the first shape and back to rest, instead of snapping. Our three.js lip sync guide shows how.
- The rest of the face: blinks, small head movements and expressions, so the mouth isn't the only thing moving.
Where Lobbyface fits
We work on the 3D side. Our hosts' lip sync comes from a custom AI model that turns speech into 29 ARKit mouth curves at 30 fps. A 3-second sentence takes about 17 ms. The same model is available as the Lobbyface Lip sync API for teams with their own characters: send the text with its audio or phoneme timings, and get curves fitted to your model's blend shapes.