How Voice Agents Work: STT, LLM, TTS, WebRTC

Creating first voice agent gives this dopamine hit. You just plug in your keys and everything starts working. This is a trap, however, because without really understanding how each of the pieces work, and understanding differences on the market with different ways of creating voice agents, it's difficult to create a solution that really works best for you, and it's also very difficult to debug if something is going wrong, especially that LLMs aren't really that great at fixing voice agents because they are so new.

In today's video, we'll cover typical voice AI agent pipeline and also cover different possibilities to create voice agents, and how everything works blended together so that you, after watching this video, will be able to understand what might be going wrong, what might need fixing, how everything connects. So, stay tuned.

The easiest way to describe how everything works is to start with a human conversation example. Let's say you're having a conversation with someone. This other person says something to you. You try to understand the words that are coming out of this person's mouth. You try to understand when you should start talking. Then you think about what you want to say, and you speak it out loud. Voice agent pipelines are very similar in that sense. You have speech detection that tries to understand what's been said, and when it's time to speak—this part is actually sometimes a little bit more complicated than that, but let's assume it's that easy for now—then you have an LLM, the brain that does the thinking what should be said, and then you have text-to-speech model that speaks words that's been produced by an LLM. We have most of the pieces there, but that's not all of them.

We have understanding speech, we have thinking, and we have speaking out loud, but we still have to connect to the conversation somehow. You're able to see this video because you clicked on it, and it's being delivered to you via network, kind of obvious, right? But the details matter here. The reason why you're able to see this video is because you clicked on it. Under the hood, it's being delivered to you via WebRTC, and there's also a different way of transporting the media, which is called WebSockets. And what you really need to remember is that WebRTC is simply better for voice AI, but it's not always possible. So what's the big difference here?

Well, the difference is that with WebRTC, the quality of the conversation is much, much higher. Everything passes on 48 kHz bandwidth, which means that the quality is really, really high and all the models work better because you get much better transcription, you get better sending off the output from text-to-speech, so the spoken words are cleaner and nicer for the end user, and overall the quality is just much, much better. With WebSockets, it really, really depends on the country. In US, you have much better telephony than in EU, for instance, but also if you're calling from EU to US, you still get the lowest of all the qualities, essentially. And obviously, the example here is accidental. There can be many different variations from where you're calling to, to whom, etc. But what you need to remember is that the quality really, really drops a lot with WebSockets, and that means that understanding of what was said would be poorer, the audio sent would be of lower quality, and essentially, the call is more difficult to handle that way.

So to recap, we connected to the video via WebRTC or WebSockets. We are able to understand the speech, we are able to think, and we are able to speak things out loud. However, we still miss quite important mechanic that really makes a huge difference. The way I presented it looks almost like step A happens, then B, then C, then D. So you understand what's been said, then you think, then you speak, then you listen, and so on and so forth.

But this is not how typical conversation looks like, does it? There are many things that can overlap, and it's similar with voice agents. Voice agents pretty much always hear what's being said. It's an ongoing loop of trying to understand what the other person is saying, and they also pass the data in chunks. So that means that as soon as LLM thinks something, it tells the text-to-speech model to start outputting the response. So even if it hasn't finished thinking fully just yet, it already spits out the response to the text-to-speech model. It matters because thanks to that, everything is much faster. It's a much smoother experience, because you don't have to wait until one of the steps fully finishes. And had it not been the case, the conversation quality with voice agents would be just really, really terrible. So we're pretty much finished with traditional pipeline.

You also have a different paradigm, which is called speech-to-speech. So, you know, some smart people have said, why would I have all those moving parts and have to tweak them individually? It feels very annoying, very complex, it takes a lot of time, so why not create one model to rule them all, to do everything for you? Welcome, speech-to-speech. It's essentially the same as the previous pipeline, however, it's all baked into one model. So what does that really mean for you? That means that it's much easier to get started with it. It means that sometimes the latency can be better because everything is performed on one hardware, that you don't have to make multiple network calls to perform the operations, and overall you get much simpler pipeline. Just one model, dump it in, tune it a little bit, and that's it.

However, this comes at a cost, because the models for speech-to-speech are typically more expensive. It also means that they are not as tweakable and not as flexible as the traditional pipeline. This is especially difficult in situations where you serve multiple customers from different countries, and you perhaps need to change the models based on the region that you're calling, or you need a very custom solutions for specific part of the pipeline. In such cases, speech-to-speech models are simply not good enough. However, they do have their use cases, and you should not forget about them.

Let's make a very short recap at the end of this video. You connect to the conversation somehow via WebRTC or WebSockets. This part is actually similar for both of the models. Then you have speech-to-text that understand what's been said and tells you when somebody finished speaking. Then you have an LLM that thinks, then you have the text-to-speech, and the loop continues with the conversation. A lot of steps. You connect to the conversation with WebSockets or WebRTC. This step is the same for both models. Then for the cascaded pipeline, you have speech-to-text that understands what's been said and tells you when somebody finished speaking, you have LLM that thinks, and you have text-to-speech that speaks things out loud, and the recognizing of the speech happens at all times, and the information passes as fast as possible from an LLM to the TTS, text-to-speech. You also have the other world, where you have speech-to-speech model that does all of it by default. You have the complexity hidden away from you, you get much bigger simplicity at the cost of the price and being stuck with one vendor, so having, well, less flexibility.

That's it. These are the absolute fundamentals of how voice AI agents work. But I have one interesting thing based on the description that I just gave you. I want you to think what happens if somebody tries to manipulate you in a normal conversation? How do you react? And how would your pipeline react with the speech-to-speech model or the traditional pipeline? What happens if you have a sore throat, so your text-to-speech doesn't work? How would similar situation look like in a traditional cascaded pipeline, or with the speech-to-speech that the provider is down? Will the solution work? What will happen? Do you cover for that case? What if you have a day on which you, which it takes you a little bit longer to think, because you're sick maybe? Can the pipelines have similar situation? This is a fun exercise. Try to ask yourself these type of questions, like how you feel in certain situations in natural conversations that you have with other people, and try to transfer these into situations with your voice AI pipeline. It's really interesting to see how many of the issues actually are replicating themselves into our artificial world. And I do encourage you to try to experiment with what happens if type of thinking, because this is how you create much better experiences for your users.