peekaboo

Peekaboo is an experimental, context-aware macOS voice assistant built on the Pipecat framework. It sits in your menu bar, continuously monitors open windows, and maintains an indexed memory of your screen activity—allowing you to query past tasks, read active windows, or trigger alerts based on screen events using natural voice commands.

1. The Non-Technical Lens: A Photographic Memory for Your Screen

Imagine having an executive assistant sitting next to you who silently takes notes on your workspace without getting in your way. Instead of digging through endless browser tabs, terminal logs, or chat histories, you can simply ask out loud, "What PR was I reviewing yesterday?" or "Tell me when the build finishes." Peekaboo connects voice AI directly to your desktop workflow, transforming visual history into a conversational interface.

2. The Technical Lens: Architecture & Multi-Agent Pipelines

Peekaboo showcases multi-agent orchestration using the Pipecat framework, running six worker agents on a single runner over a shared event bus:

• Local Speech Processing: Runs on-device STT via Moonshine and TTS via Kokoro. Voice processing remains entirely local until the wake word is triggered.
• Visual Memory Pipeline: Window stills are captured every 2 seconds. A local change gate filters duplicate frames; modified frames are sent to a cloud Vision LLM (Anthropic or OpenAI) at most once every 15 seconds per window.
• Store & Retrieval: Text extractions, visual descriptions, and frames are saved to a local memory store, which the Cloud Voice LLM queries via tool calling to answer historical questions.
• Proactive Action: Monitored windows can trigger conditional voice alerts, while notification banner parsing allows single-word meeting joins and UI window controls.

3. Why It Matters

Peekaboo highlights an effective hybrid privacy design for desktop voice AI: keeping continuous audio stream processing local while offloading scene description and complex reasoning to cloud LLMs. By combining real-time vision processing with a multi-agent pipeline, it offers a modular blueprint for localized, context-aware desktop copilots using open-source tools.

4. The "Voice AI Space Lab" Idea

The Hands-Free QA & Stream Assistant: Build an automated debugging partner. While running complex UI tests or live coding streams, ask Peekaboo to continuously monitor background terminal logs or error windows in real time, alerting you verbally the exact second an exception occurs and summarizing the call stack without breaking your context.

Explore the project repository: GitHub - pipecat-ai/peekaboo