open-tts-eval

Inworld AI has open-sourced open-tts-eval, an open, transparent evaluation toolkit designed to solve one of the biggest challenges in speech AI: fair, reproducible performance comparisons across Text-to-Speech (TTS) and Speech-to-Text (STT) models.

For the Non-Technical Reader

Imagine buying a car based on fuel efficiency, only to realize the manufacturer tested it downhill with the wind at its back. In Voice AI, metric benchmarks like Word Error Rate (WER) often suffer from the same issue—different settings, text normalizers, and voice samples produce wildly inconsistent scores. This toolkit acts as a standardized, independent test track. It runs every speech model through identical conditions, evaluating not just word accuracy, but naturalness, audio glitches, awkward pauses, and tone variations.

For the Technical Reader

The toolkit provides a self-contained, offline-first evaluation pipeline that eliminates infrastructure dependencies during test execution:

• ASR & Transcript Normalization: Utilizes local ASR via faster-whisper to compute WER, CER, insertion/deletion/substitution rates, and hallucination heuristics.
• Audio Health & Quality Metrics: Features integrated checks for audio artifacts (silence, clipping, tail-clicks, speech-rate proxy, loudness), coupled with heavier metrics like NISQAv2 naturalness, ECAPA speaker similarity, expressiveness proxies, and vowel prolongation heuristics.
• Strict Provenance & Reproducibility: Records evaluator code versions, normalizers, ASR configurations, and provider parameters. The comparison engine explicitly rejects merging runs evaluated under non-identical configurations.
• Statistical Rigor & Offline Reporting: Includes paired per-text bootstrap statistical significance testing and outputs self-contained, zero-CDN black-and-white HTML reports (Metric Comparison tables, Model Health grids, and Threshold Violations). Includes a 100-utterance dialogue stress set.
• License: MIT License.

Why It Matters

As speech synthesis becomes hyper-realistic, objective evaluation is critical. Proprietary benchmarks often hide evaluator settings, making vendor comparisons misleading. By open-sourcing a standardized evaluation methodology, Inworld AI enables developers and enterprises to audit both open-source models and proprietary provider APIs (such as ElevenLabs, Hume, and Inworld) with complete transparency and zero vendor bias.

The "Voice AI Space Lab" Idea

The Automated Voice Actor Audition Board: Set up a continuous integration (CI/CD) pipeline for custom TTS models. Every time a new voice model or fine-tune is trained, trigger an evaluation against the 100-utterance dialogue stress set. Automatically reject models that violate audio health thresholds (e.g., tail-clicks or unnatural pauses) and generate offline HTML scorecards to systematically pick the best voice candidate for production deployment.

Explore the repository on GitHub: inworld-ai/open-tts-eval