How TTS Models Are Trained: Data, Cleaning, Cloning (2026)

Data Sources and Requirements

Text-to-speech (TTS) models learn from recordings of real people paired with the exact words spoken. There are four main sources for this training data, each with distinct trade-offs:

• Studio recordings: Clean and high-quality, but costly to produce.
• Audiobooks: Public domain options exist (such as LibriVox), but the speech often sounds like reading.
• Internet speech: Natural but messy, and subject to licensing restrictions.
• Synthetic speech: Generated by other TTS models; cheap but offers less variety.

The volume of training data required varies by the model's goals. A classic model can be trained on about a day of audio (around 24 hours) for a single voice. Multi-voice models like Kokoro use a few hundred hours, while larger, more flexible models require significantly more data, ranging from 95,000 hours (F5-TTS) to a million hours (CosyVoice 3).

Data Cleaning Pipeline

Raw internet audio must undergo a rigorous cleaning process before it can be used for training. Using the Emilia pipeline as an example, the steps include:

1. Standardize: Converting every file into a single format.
2. Separate: Removing background noise and music.
3. Diarize: Splitting the audio based on who is speaking.
4. Segment: Cutting the audio into short clips of 3 to 30 seconds.
5. Transcribe: Generating text from the audio using speech-to-text tools.
6. Filter: Removing clips with incorrect languages, poor sound quality, or unusual speaking speeds.

This cleaning process is highly selective; in tests, less than a third of the raw audio survived the pipeline.

The Training Loop

Training is an iterative loop performed on GPUs:

• The model reads the text of a clip and makes a guess at the sound.
• The guess is compared to the real recording, and the difference is calculated as the loss.
• The model's internal weights are slightly adjusted to minimize this loss.
• This process is repeated millions of times. A portion of the data is held out and never trained on to test the model's performance on unheard audio.

Model Recipes and Voice Cloning

Models predict sound using different approaches. The classic recipe predicts a spectrogram (a picture of sound) and may train a vocoder against a critic network to detect generated audio. The language model recipe uses a speech tokenizer to convert audio into tokens, predicting the next token. Many language-based models build upon existing text models that already understand language.

To handle multiple voices, models use a speaker embedding, which acts as a numerical fingerprint for a voice. Models can have a fixed set of voices or support cloning. Zero-shot cloning copies a voice from a few seconds of audio without extra training, while professional cloning fine-tunes the model on longer samples (ideally two or more hours) and typically requires a consent check.

Evaluation and Ethical Considerations

After training, models are evaluated using human listeners, speech-to-text transcription to calculate the Word Error Rate, and speaker recognition models to assess voice similarity. Because data sourcing raises legal and ethical questions, some models incorporate inaudible watermarks to identify generated speech. Before selecting a TTS model, users should evaluate where the data came from, whether the speakers consented, if commercial use is permitted, and what safeguards exist against unauthorized voice cloning.