Demo - Fini, a multi-modal AI agent for enterprise fintech support - Voice AI Space Amsterdam

Overview of Finny

Finny is a voice AI company that provides autonomous voice agents for inbound customer support and outbound outreach. Unlike simple answering bots, Finny’s agents do not only respond to customer questions. They can also take direct actions through APIs, such as rescheduling appointments, unlocking scooters, handling payment-related issues, or confirming customer requests.

The platform is built for real-world customer support scenarios where users expect fast, natural, and helpful responses. Its voice agents can manage complex conversations, adapt to customer emotions, and complete tasks without always needing a human support representative.

Main Functions

Inbound Customer Support:
Finny’s inbound agents handle customer calls and resolve issues by taking action directly. For example, an agent can verify a user, check account details, unlock a scooter remotely, or help with a payment-related problem.

Outbound Outreach:
The platform also supports outbound calling workflows, such as appointment confirmations, reminders, and rescheduling. These agents can call customers, confirm details, and update systems automatically.

System Architecture

Finny uses modular voice AI pipelines that combine speech-to-text, LLM processing, and text-to-speech. This setup allows the system to listen to a customer, understand the request, generate a response, and speak back naturally.

The platform is also evolving toward low-latency speech-to-speech models. These models reduce delays between turns and make conversations feel faster and more natural. The system can adjust speaking speed, respond to user frustration, and even switch accents during the call if requested.

For complex actions, Finny uses graph workflows. These workflows define the agent’s decision-making path for multi-step tasks, including what should happen if something goes wrong. For example, if a scooter cannot be unlocked because of unpaid fines, the agent must follow a safe and structured path instead of making an uncontrolled decision.

Key Learnings

Voice Agents Must Take Action, Not Just Talk:
The strongest use case is not answering basic questions. It is letting the agent complete real tasks through APIs, such as unlocking a scooter, rescheduling an appointment, or managing a customer request.

Low Latency Is Critical for Natural Calls:
Customer support calls need smooth turn-taking and fast responses. Speech-to-speech models help reduce delays and make the conversation feel less robotic.

Personalization Improves User Experience:
Finny’s agents can adapt to the user’s speaking speed, frustration level, and accent preferences. A customer can even ask the agent to switch to a different accent, such as British English, during the call.

Autonomous Actions Need Guardrails:
Legal and compliance concerns are reduced by limiting autonomous decisions to safer, lower-risk tasks. The system is designed to handle actions like cancellations, appointment changes, or remote unlocks within controlled workflows.

Technical Details and Q&A

Technology Stack:
Finny uses modular pipelines that include speech-to-text, LLMs, text-to-speech, API integrations, and newer speech-to-speech models for low-latency voice interaction.

Modular Pipelines:
The system architecture is flexible, allowing different components to be customized. Clients can adjust speech recognition, language model behavior, voice output, and workflow logic depending on their use case.

Speech-to-Speech:
Finny is moving toward speech-to-speech interaction, which allows faster and more natural voice conversations. These models can reduce latency and support more adaptive speaking behavior.

Tool and API Calling:
The agents connect with external systems through APIs. This allows them to perform real actions, such as checking scooter status, unlocking vehicles, updating appointments, or managing customer records.

Graph Workflows:
Complex tasks are handled through graph workflows. These workflows map out possible user paths, required checks, failure scenarios, and allowed actions. This helps the agent stay reliable during multi-step support cases.

User Authentication:
The system can authenticate users through their phone number before performing sensitive actions like remote unlocking. This ensures the agent only executes commands for verified users.

User Experience Features:
Finny’s agents can adjust their speech speed, respond to signs of frustration, and switch accents when requested. This makes the call experience feel more personalized and natural.

Legal and Compliance Approach:
Finny reduces risk by limiting autonomous actions to low-risk or clearly defined tasks. The agent can perform actions like cancellations, remote unlocks, or appointment updates, but within strict workflow boundaries.

Quality Assurance:
Quality is measured by the agent’s ability to complete tasks correctly, maintain low latency, handle realistic customer interactions, and follow safe workflows during edge cases.

Important Keywords and Definitions

Finny:
The company providing autonomous voice agents for customer support and outreach.

LLM:
A Large Language Model that processes the user’s request and generates the agent’s response.

API:
A connection method that allows the agent to interact with external systems, such as scooter databases, appointment tools, or payment systems.

Modular Pipelines:
Flexible voice AI architecture where speech-to-text, LLM, and text-to-speech components can be customized.

Graph Workflows:
Structured decision maps that guide the agent through complex tasks and failure scenarios.

Speech-to-Speech:
A voice AI approach that enables direct voice-to-voice interaction with lower latency and more natural flow.

Accent Switching:
The ability of the agent to change its speaking accent during a call based on customer preference.

Analogy:
Finny’s AI agent is like a skilled bilingual concierge at a luxury hotel. It does not just answer the phone; it solves problems. It can unlock a guest’s room remotely, switch language or accent when needed, slow down if the guest is confused, and still follow strict security rules about who it is allowed to help.