One of the most common use cases we see in the industry today is building AI agents around voice.

These can be:

AI agents answering phone calls
Real-time voice assistants
Conversational support bots
Multi-modal agents that can hear, speak, and reason

The “easy” way to build them?

Use platforms like:

Vapi
Kore.ai
Bland.ai

And to be clear - these are great products. They work well.

But when you start thinking about production scale, flexibility, and control, challenges quickly appear:

The Common Pain Points

Vendor lock-in & limited portability
You don’t run your own infrastructure
Hard to deeply customize your agents
Slow tool-calling workflows
A big black box with limited visibility

If you're building something strategic - something core to your product - these limitations matter.

Our Vision for Voice Agents

When we build voice agents, we aim for:

Natural, human-like real-time conversations
Ultra-low latency
Interruptions (“barge-in”) like real human dialogue
Web, mobile, and telephony support
Production-grade scalability

So the real question becomes:

How do you build this without massive complexity - while keeping full control?

Meet LiveKit - https://livekit.io

Why LiveKit?

LiveKit is an open-source agent framework designed for building real-time, programmable participants that run on servers.

It enables conversational, multi-modal voice agents that can:

See
Hear
Understand
Respond in real time

And if you're wondering whether it’s production-ready:

OpenAI uses LiveKit to deliver voice to millions of ChatGPT users.

More production references: https://livekit.io/customers

What Makes It Different?

LiveKit Cloud goes beyond managed infrastructure. It combines:

Real-time audio, video, and data streaming
Agent development tools
Managed agent hosting
Built-in inference
Native telephony
Production-grade observability

All in one cohesive platform.

The Voice Pipeline Architecture

With LiveKit, you can build agentic voice conversations using modular pipelines:

Typical Pipeline

STT - Speech-to-Text

LLM - Reasoning engine

TTS - Text-to-Speech

You can use reference models such as:

OpenAI
Deepgram
ElevenLabs

Or plug in your own:

Azure OpenAI

This flexibility makes the framework extremely customizable.

Diagram showing voice pipeline agent processing path.

You can also use real-time AI models for even better conversational flow.

Diagram showing realtime agent processing path.

Voice Activity Detection (VAD)

A key component in real-time agents is VAD (Voice Activity Detection).

It allows the agent to:

Detect when the user starts speaking
Detect when the user stops speaking
Filter silence/noise (saving STT costs)

This enables true “barge-in” conversations - where users can interrupt naturally.

Architecture Design

Language Support

LiveKit supports:

Python
Node.js

Which makes it easy to integrate into modern backend architectures.

Getting Started with LiveKit (Python Example)

Create an account at https://livekit.io
Create a new project
Install LiveKit CLI
Initialize a project:

uv init livekit-voice-agent --bare

Install dependencies (Python 3.11 required):

uv add "livekit-agents[silero,turn-detector]~=1.2"
uv add "livekit-plugins-noise-cancellation~=0.2"
uv add "livekit-agents[openai]~=1.3"
uv add "python-dotenv"

6. basic agent configuration:

session = AgentSession(        
          stt="deepgram/nova-3:en",        
          llm=openai.LLM.with_azure(            
              azure_deployment=AZURE_OPENAI_DEPLOYMENT_NAME,            
              azure_endpoint=AZURE_OPENAI_ENDPOINT,            
              api_key=AZURE_OPENAI_API_KEY,            
              api_version="2025-04-01-preview",        ),        
          tts="elevenlabs/eleven_flash_v2:iP95p4xoKVk53GoZ742B",        
          vad=silero.VAD.load(),        
          turn_detection=MultilingualModel(),    
        )

Using Azure OpenAI Models

You can find a complete code example in the Git repository: https://github.com/m8tly/livekit-agents-demo

My Demo: Multi-Model Voice Agent

In my GitHub demo, I built a voice agent using:

STT: deepgram/nova-3:en
LLM: Azure OpenAI gpt-5-mini
TTS: elevenlabs/eleven_flash_v2

It also includes two tools:

Answering questions about the m8tly.ai community
Listing available blog posts

You can clone the repo, switch models, and experiment freely.