100% Local AI Speech to Speech with RAG - Low Latency | Mistral 7B, Faster Whisper ++

System Overview

The system is a 100% local speech-to-speech system integrated with Retrieval-Augmented Generation (RAG). It allows users to converse with an AI assistant using voice commands while retrieving information from a locally stored database of embeddings.

Key Components

• Local LLM: Supports various models such as Dolphin Mistral 7B, Mistral AI, or Qwen Chat 14B. The performance of the RAG system scales with the quality of the chosen model.
• Voice-to-Text Transcription: Uses Faster Whisper to transcribe microphone input directly into text.
• Local Text-to-Speech (TTS) Engine: Supports XTTS v2 for high-quality voice generation with adjustable parameters (such as temperature, speed, and emotion) or OpenVoice for low-latency conversations.
• Local RAG (Embeddings): Uses the all-MiniLM-L6-v2 model to generate embeddings from a local text file (vault.txt), which serves as the system's knowledge base.

Voice Commands and Functionality

The system supports specific voice commands to manage the local knowledge base:

• "Insert info": Records voice input, transcribes it using Faster Whisper, and appends the text to the local vault file, which is then converted into embeddings.
• "Print info": Displays the current contents of the local vault file.
• "Delete info": Deletes the local vault file after requiring a verbal "yes" confirmation from the user.

PDF Integration

A custom Python script allows users to upload a PDF file, convert its contents into text, and append it directly to the local RAG database. Once uploaded, the local LLM can access this information to answer specific questions about the document's content.