From Voice Agents to AI Avatars

Voice AI has gotten remarkably good, but natural conversation remains a high bar. Small delays, awkward interruptions, or the wrong tone can quickly break the illusion, and adding vision and visual presence only raises the stakes.

In this episode, Alex Smola, co-founder and CEO of Boson AI, explores the path from today’s voice agents to audiovisual agents and AI avatars. We discuss the technical tradeoffs behind real-time voice, including audio tokenization, latency, model size, and inference cost, as well as what changes when these systems can both see and be seen.

We also explore the role of emotional intelligence in AI, how agents can learn from human interactions, and what it will take to move beyond impressive demos toward interactions that actually feel natural.

Full show notes including references: https://twimlai.com/go/777.

🔔 Subscribe to our channel for more great content just like this:
https://youtube.com/twimlai?sub_confirmation=1

📖 CHAPTERS
===============================
00:00 – Introduction
01:36 – Challenges of Voice AI in Real-World Environments
06:12 – Why Voice Systems Are a Whole-System Challenge
12:50 – Trade-Off Between Size, Compute, and Affordability
13:53 – Boson AI’s Path to Audio
19:59 – Training Dataset
23:49 – Dealing with Noisy Data
28:41 – Model Training Approach
33:14 – Architectural Decisions: End-to-End vs. Multi-Threaded Systems
39:26 – MCP, Context Windows, and Inference Costs
41:04 – Agentic Systems for Voice
46:31 – Benchmarking
50:03 – Why EQ Matters for Voice and Video
53:43 – Recursive Self-Improvement and Personalization
01:00:03 – Brilliance, Interaction Improvement, and Wrap-Up

🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Follow us on Twitter: https://twitter.com/twimlai
Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/