MuDuSi

MuDuSi is a visual, multi-agent acoustic simulator built specifically for full-duplex speech models, evaluation, and reinforcement learning research. It allows researchers and developers to simulate realistic, multi-speaker conversational environments in software before deploying models to physical hardware setups.

For the Non-Technical Reader

Imagine trying to train multi-person conversational AI systems for a noisy restaurant or busy front desk, but having to physically wire up microphones, speakers, and room acoustic panels every time you change a line of code. MuDuSi acts as a virtual sandbox for voice AI. It places independent AI conversationalists into a simulated 3D room where they can speak, listen, interrupt, and hear each other dynamically, eliminating the need for expensive physical acoustic testbeds during early development.

For the Technical Reader

MuDuSi provides a controlled software environment for end-to-end full-duplex multi-agent audio simulation. Key technical features include:

• Audio & Streaming: Operates on 24 kHz audio with independent input and output streams per agent, enabling true full-duplex simultaneous listening and speaking.
• Spatial Acoustic Engine: Models sample-level propagation delay, distance attenuation, divider occlusion, one-wall reflections, and per-microphone Gaussian noise.
• Model Compatibility: Out-of-the-box integration with Moshi and PersonaPlex native bridges, a Raon adapter, and a generic streaming PCM protocol for custom models.
• Observability & Metrics: Features a real-time web UI showing 3D spatial activity, buffer counters, overlap, TX/RX levels, and session replay while preserving clean source channels alongside microphone mixtures.

Why It Matters

Physical audio validation is slow, non-deterministic, and hard to reproduce across research teams. By providing a software-defined acoustic simulator, MuDuSi dramatically reduces iteration time for full-duplex voice research and multi-agent RL experiments, establishing standard benchmark scenes (such as roundtable or cocktail scenarios) without physical hardware overhead.

The "Voice AI Space Lab" Idea

The Simulated Town Hall Stress Test: Create a 3-agent acoustic simulation featuring a virtual mayor and two competing citizens who talk over each other in a reflective room with background chatter. Use this setup to train full-duplex models on real-time interruption handling and focus control before deploying them into real-world customer service or collaborative meeting bots.

Explore the source code and documentation at the MuDuSi GitHub Repository.