Voice AI Space Paris - ​Tuning Voice AI agent for Long Conversations - Swati Singh PM @ Criteo

Designing Voice AI for Long Conversations

Most voice AI systems are optimised for short, transactional interactions lasting three to ten minutes, such as customer support or booking appointments. However, meaningful conversations, such as extracting a user's professional experience to build a CV, require sustained interactions for up to an hour. Building such a system reveals several challenges in conversation design and requires specific tuning strategies.

Key Tuning Strategies

• Responding Before Pivoting: Standard bot patterns follow a rigid question-and-answer structure that can feel like an interrogation. To create a natural flow, the AI must be programmed to react to the user's input, either by reinforcing their statement or asking a direct follow-up before transitioning to a new topic.

• Flexible Topic Hopping: Human conversations are non-linear and frequently jump between subjects. Instead of using a rigid, step-by-step script, conversations should be organised into independent "buckets" (such as career history, technical skills, and project impact). This allows the AI to dynamically transition between topics based on the user's flow while still ensuring all necessary areas are eventually covered.

• Ensuring Depth: While flexibility is important, rapid topic switching can result in shallow information. To prevent this, the AI should use completion thresholds for each bucket, requiring specific details (such as metrics or project outcomes) to be established before officially closing a topic.

• Cognitive Breathers: Spending an hour in a high-intensity interview format is mentally exhausting. Introducing brief "breathers" every 20 minutes, by lowering the tone, using light humour, or making casual remarks, helps reset the user's cognitive load.

• Accent Localisation: Users communicate more comfortably and for longer durations when the AI speaks with a familiar accent. Unfamiliar accents increase the user's cognitive effort, distracting them from the actual conversation.

Measuring Success

Evaluating the performance of a long-form voice AI agent relies on three primary metrics:

• Engagement Ratio: Monitoring whether users are providing brief, one-word answers or sharing detailed stories.

• Turn-Taking Metrics: Tracking "mis-hits", such as the AI interrupting the user or misinterpreting pauses.

• Outcome Quality: Ensuring the final output (the generated CV) is detailed and meaningful rather than a shallow list of bullet points.