LlamaPIE

This repository introduces LlamaPIE, a proactive in-ear conversation assistant designed to enhance human-to-human communication by providing discreet, real-time guidance via hearable devices.

For the Non-Technical Reader

Imagine having a tiny, helpful whisper in your ear during a conversation. LlamaPIE is like that – it's designed to provide subtle cues and reminders, almost like a 'conversation coach' that anticipates what you might need to say or remember. Instead of awkwardly pausing or forgetting a name, LlamaPIE gives you a gentle nudge, ensuring smoother and more engaging interactions. Think of it as the ultimate wingman for your ears, offering support only when needed and staying silent the rest of the time. This changes the game for anyone who wants to improve their conversational skills or needs a little extra help in social situations.

For the Technical Reader

LlamaPIE is implemented using the MLX framework, optimized for Apple Silicon, enabling streaming and real-time operation. The repository provides inference code and checkpoints for both large and small models. The large model is available with INT8 quantization, while the small model uses FP16 quantization. The system proactively determines when to provide assistance, delivering concise messages. The repo includes training code and instructions for both small and large models. Evaluation datasets include synthetic conversations generated from Claude (Syncclaude), PerLTQA (Syncperl), SODA (Syncsoda), and real-recorded conversations from the MIT interview dataset. Rubric score evaluation uses ChatGPT-4o. GitHub Repository and Hugging Face links are available for further exploration.

Why It Matters

LlamaPIE represents a significant step towards proactive and unobtrusive AI assistance. Its open-source nature fosters community development and innovation, potentially leading to widespread adoption in various applications. By operating on-device, it addresses privacy concerns associated with cloud-based AI assistants and offers a cost-effective solution for real-time conversation support.

The "Voice AI Space Lab" Idea

Imagine building a "LlamaPIE-powered language learning assistant." Users could wear the in-ear device while practicing a new language, receiving real-time pronunciation corrections, vocabulary suggestions, and grammatical guidance, making language acquisition more immersive and effective.

The Collaborative CTA

How can we expand LlamaPIE's capabilities to support multi-lingual conversations or integrate with other wearable technologies for a more seamless user experience? Share your thoughts!

#VoiceAI #Hearables