The Race to Production-Grade Diffusion LLMs [Stefano Ermon] – 764

Today, we’re joined by Stefano Ermon, associate professor at Stanford University and CEO of Inception Labs to discuss diffusion language models. We dig into how diffusion approaches—traditionally used for images—are being adapted for text and code generation, the technical challenges of applying continuous methods to discrete token spaces, and how diffusion models compare to traditional autoregressive LLMs. Stefano introduces Mercury 2, a commercial-scale diffusion LLM that can generate multiple tokens simultaneously and achieve inference speeds 5-10x faster than small frontier models, paving the way for latency-sensitive applications like voice interactions and fast agentic loops. We also cover the open research challenges in diffusion LLM training, serving infrastructure requirements, and post-training for diffusion-based systems. Finally, Stefano shares his perspective on whether diffusion models can rival or surpass autoregressive LLMs at scale, the advantages for highly controllable generation, and what the future of multimodal diffusion models might look like.

🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/764.

🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1

🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Follow us on Twitter: https://twitter.com/twimlai
Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/

📖 CHAPTERS
===============================
00:00 – Introduction
04:11 – Origins of diffusion models
07:24 – From image diffusion to text diffusion
08:07 – Discrete data challenges
09:54 – Limitations of embeddings
11:10 – Diffusion versus autoregressive models
14:31 – Masking
19:27 – Reasoning in diffusion models
20:58 – Context windows and variable-length outputs
25:36 – RL pre-training and post-training
28:44 – Can autoregressive models be converted to diffusion?
31:11 – Serving challenges for diffusion models
34:50 – Speed, cost, quality tradeoffs
40:07 – Current limitations of Mercury 2 model
43:06 – Open science questions
45:22 – Hallucinations and generalization challenges
47:46 – Controllable generation
51:41 – Will diffusion overtake autoregressive models?
53:57 – Diffusion in agentic applications
56:00 – Gemini diffusion
57:51 – Key research groups working on diffusion models
01:00:09 – Cross-pollination between image and text diffusion
01:01:18 – Future of multimodal diffusion models

🔗 LINKS & RESOURCES
===============================
Inception – https://www.inceptionlabs.ai/
Ermon Group – https://cs.stanford.edu/~ermon/website/index.html
Introducing Mercury 2 – https://www.inceptionlabs.ai/blog/introducing-mercury-2
Domain Knowledge in Machine Learning Models for Sustainability with Stefano Ermon – 15 – https://twimlai.com/podcast/twimlai/domain-knowledge-in-machine-learning-models-for-sustainability

📸 Camera: https://amzn.to/3TQ3zsg
🎙️Microphone: https://amzn.to/3t5zXeV
🚦Lights: https://amzn.to/3TQlX49
🎛️ Audio Interface: https://amzn.to/3TVFAIq
🎚️ Stream Deck: https://amzn.to/3zzm7F5