esp_ai

Ivy AI (esp_ai) is a fully offline, self-contained voice assistant running entirely on a $20 ESP32-S3 microcontroller. It executes an end-to-end voice processing pipeline—speech recognition, local LLM inference, and text-to-speech synthesis—completely on-device without relying on external servers, cloud connectivity, or smartphone pairings.

For the Non-Technical Reader

Imagine having a smart assistant like Amazon Alexa or Apple Siri, but completely stripped of internet reliance, monthly fees, and privacy concerns. Ivy AI acts like a self-contained brain on a tiny chip. You say "Hey Ivy", ask a question, and the device transcribes your voice, thinks through an answer using a built-in tiny AI language model, and speaks back out loud. Because everything runs locally on a $20 board, your voice recordings never leave the device, offering total data privacy and working flawlessly off-grid or during internet outages.

For the Technical Reader

Built in native ESP-IDF v6.1 (C) for the ESP32-S3R8 (dual Xtensa @ 240 MHz, 8 MB octal PSRAM, 16 MB flash), the system achieves end-to-end local voice interaction via a highly constrained architecture:

• Wake Word & STT: Utilizes Espressif WakeNet9 for always-on wake word detection and a Conformer CTC open-vocabulary speech-to-text model streamed directly from SD storage.
• Local LLM Engine: Runs TinyTalk 2 8M v8 (19.7M parameters, Q4 quantization, ~6.5 MB) at 5.4–5.9 tokens/sec. It also supports arbitrary llama.cpp GGUF models stored on SD (e.g., delphi 6.4m at 7.3 tok/s).
• Hardware Acceleration: Achieves execution speed through 4-bit weights with int8 activations, a hand-written ESP32-S3 PIE SIMD assembly kernel, and matrix-vector distribution across the second CPU core.
• TTS & Memory Strategy: Synthesizes speech using SVOX Pico en-US. To fit within 8 MB PSRAM and ~300 KB SRAM, STT (5.4 MB) and LLM (6.5 MB) execute sequentially inside a 7 MB boot-reserved PSRAM arena rather than concurrently.

Why It Matters

Ivy AI highlights a critical milestone in edge computing and privacy-first Voice AI. Moving an entire voice-to-voice pipeline—including LLM generation—down to a single $20 microcontroller eliminates API costs, cloud latency dependencies, and privacy risks. It showcases that meaningful, interactive voice agents no longer require energy-intensive GPUs or continuous internet connectivity.

The "Voice AI Space Lab" Idea

The Off-Grid Emergency First-Aid Companion: Embed the board into an off-grid survival kit or disaster relief medic pack. First responders or outdoor hikers can speak hands-free to query triage steps, dosage calculations, or wilderness survival guides when cellular networks and power grids are entirely unavailable.

Check out the full source code on GitHub or read more on the official Ivy AI Documentation.