Getting Started with LipDub AI
Choose the right project type
The most important first step is selecting the correct project type.
From Home, you can quickly start a Translation or Personalization project. To access all four project types, open Projects and select Create Project.
Project type | Choose this project when | Main inputs |
|---|---|---|
Translation project | You want to translate a video featuring one visible face into one or more languages. LipDub generates translated speech and synchronizes it to the speaker’s lips. | A source video and your source and target languages |
Personalization project | You want to create many customized versions of a video by changing details such as a name, company, or other variable. | A source video and a CSV containing recipient data |
Dialogue Replacement project | You want to replace the dialogue in a video featuring one visible face, using uploaded audio or speech generated from written text. | A source video and either replacement audio or a script |
Multi-speaker project | Multiple faces appear anywhere in the footage, or you need to control speaker training and assign separate audio to specific people. It can also share subject training data across multiple videos in the same project. | One or more source videos and separate audio for each speaker |
Dialogue Replacement projects were previously called Single-speaker projects.
Important: Use a Multi-speaker project whenever more than one face appears at any point in the source footage— even when only one person speaks. It is the only project type that lets you choose which face detections are included in each person’s training data and assign audio to specific speakers.
In a Multi-speaker project, you only need to label and train the faces you intend to lip-sync.
Translate without lip-sync
When creating a Translation project, you can select Pure Dubbing to generate translated audio without changing the subject’s mouth movements. Pure Dubbing is a Translation project setting, not a separate project type.
Navigate LipDub AI
The main navigation appears on the left side of the platform.
Area | What you can do |
|---|---|
Home | Quickly start a Translation or Personalization project, or open your existing projects. |
Create Translation | Go directly to the Translation project setup. |
Create Personalization | Go directly to the Personalization project setup. |
Projects | Search, open, and manage existing projects, or select Create Project to access all four project types. |
Subscription | View and manage your subscription and available credits. |
Translation Memory | Create, edit, and delete preferred translation entries for use when reviewing translated segments. |
Your current credit balance appears near the top-right corner of the platform.
Open the account menu in the top-right corner to access:
- Account Settings
- Manage Subscription & Billing
- Credit Usage History
- Terms of use
- Log out
Detailed account, billing, and credit information is covered in Manage Your Account, Plan, Credits, and Integrations.
Understand the basic workflow
The exact steps differ by project type, but most LipDub projects follow the same general process:
- Create a project
Choose the project type that matches your footage and intended result. - Upload your source content
Depending on the workflow, this may include a video, audio files, a written script, or a CSV. - Process the footage
LipDub analyzes the video and prepares it for the selected workflow. When training is required, LipDub uses visible footage of the subject speaking to learn their individual lip movement. - Configure the new dialogue
Translate the original speech, add personalization variables, upload replacement audio, or assign audio to individual speakers. - Review your project
Check the transcript, audio, speaker assignments, timing, and project-specific generation settings. - Generate the video
LipDub displays the project status while it is processing, training, or generating. - Review and use the result
Preview the completed video, download it, or use any sharing and export options available for that project type.
You can often return to the same project and generate additional languages, audio versions, personalized videos, or quality variations without starting over.
Before creating your first project
Keep these three rules in mind:
- Choose the project type based on every face that appears in the footage—not only the person who speaks.
- Use the cleanest and most consistent video and audio available. The inputs you provide directly affect the generated result.
- Only upload voices and likenesses you have permission to use and clone.
For supported formats, recording recommendations, training-footage guidance, and upload requirements, see Prepare Videos, Audio, Transcripts, and CSV Files.
Continue with the appropriate guide
Once you have selected a project type, follow its complete workflow guide:
- Translate and Lip-Sync a Video
- Personalize Videos at Scale with a CSV
- Replace Dialogue in a Single-Speaker Video
- Create a Multi-Speaker LipDub