audio-dfd-benchmark

Evaluating audio deepfake detectors has long suffered from self-reporting bias, overfitted test sets, and inconsistent evaluation criteria. The audio-dfd-benchmark project by Podonos introduces a neutral, standardized public evaluation framework designed to score deepfake voice detection systems against a private, format-rich gold-standard dataset.

1. For the Non-Technical Reader

Think of this benchmark as an independent crash-test facility for audio security. Instead of companies claiming their deepfake detector is 99% effective using their own internal tests, this benchmark subjects every tool to the exact same hidden exam. It gives business leaders and fraud prevention teams an unbiased, apples-to-apples leaderboard showing which solutions reliably catch cloned voices in real-world situations.

2. For the Technical Reader

The benchmark evaluates 23 systems across 12 vendors—comprising 13 commercial APIs and 11 downloadable/open-weights models—on a fixed dataset of 4,524 audio samples. Evaluation criteria focus on Accuracy, F1-score, False Positive Rate (FPR), False Negative Rate (FNR), Latency (ms), and Real-Time Factor (RTF).

• Private Ground Truth: Gold-standard labels are held privately by Podonos to prevent data leakage and benchmark gaming.
• Commercial Dominance: High-performing commercial systems like deetech.ai (99.56% Accuracy, 0.019 RTF) and Resemble DETECT-World (99.47% Accuracy) currently lead the board.
• Research Baselines Lagging: Traditional open research baselines like RawNet2 (50.66%) and AASIST (48.17%) struggle on modern dataset variations.
• Open-Source Champion: Pella Research provides a strong open-weights model (MIT license) achieving 95.82% accuracy with a 57ms inference latency (0.021 RTF).

3. Why It Matters

With generative voice models evolving rapidly, enterprise authentication and call centers face unprecedented security risks. This public leaderboard creates market transparency, demonstrating that legacy academic models no longer suffice while highlighting performant open-weights alternatives like Pella for teams looking to avoid proprietary vendor lock-in.

4. The Voice AI Space Lab Idea

The Dynamic VoIP Firewall Router: Build a multi-tier call-screening gateway for enterprise PBX systems. Incoming streams are routed through a high-speed local open-weights model (such as Pella Research) as a zero-cost, low-latency first line of defense. If synthetic audio confidence crosses a threshold, the stream seamlessly escalates to high-accuracy commercial APIs like deetech.ai for secondary verification before alerting the human operator.

Check out the benchmark and leaderboard on GitHub.