High-Efficiency Diffusion Models for On-Device Image Generation and Editing [Hung Bui] – 753

In this episode, Hung Bui, Technology Vice President at Qualcomm, joins us to explore the latest high-efficiency techniques for running generative AI, particularly diffusion models, on-device. We dive deep into the technical challenges of deploying these models, which are powerful but computationally expensive due to their iterative sampling process. Hung details his team’s work on SwiftBrush and SwiftEdit, which enable high-quality text-to-image generation and editing in a single inference step. He explains their novel distillation framework, where a multi-step teacher model guides the training of an efficient, single-step student model. We explore the architecture and training, including the use of a secondary ‘coach’ network that aligns the student’s denoising function with the teacher’s, allowing the model to bypass the iterative process entirely. Finally, we discuss how these efficiency breakthroughs pave the way for personalized on-device agents and the challenges of running reasoning models with techniques like inference-time scaling under a fixed compute budget.

🗒️ For the full list of resources for this episode, visit the show notes page: https://twimlai.com/go/753.

🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1

🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Follow us on Twitter: https://twitter.com/twimlai
Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/

📖 CHAPTERS
===============================
00:00 – Introduction
04:44 – VinAI Research
07:03 – Building an AI team
09:02 – First AI residency program
09:47 – Focus on model efficiency
11:49 – Training a Vietnamese LLM
16:29 – Model optimizations for Vietnamese LLM
17:54 – Benchmarks
19:33 – Image generation
22:35 – Distillation
24:30 – Secondary ‘coach’ network
28:03 – Results
29:55 – SwiftEdit paper
35:46 – On-device agents
37:26 – Research directions
40:44 – Inference time-scaling on mobile devices
46:35 – Lessons learned on acquisition
48:16 – Qualcomm AI Residency Program

🔗 LINKS & RESOURCES
===============================
SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation – https://arxiv.org/abs/2312.05239
SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion – https://arxiv.org/abs/2412.04301
PhoGPT: Generative Pre-training for Vietnamese – https://arxiv.org/abs/2311.02945
Qualcomm AI Residency Program – https://www.qualcomm.com/research/artificial-intelligence/ai-residency-program
Distilling Transformers and Diffusion Models for Robust Edge Use Cases with Fatih Porikli – 738 – https://twimlai.com/podcast/twimlai/distilling-transformers-and-diffusion-models-for-robust-edge-use-cases/

📸 Camera: https://amzn.to/3TQ3zsg
🎙️Microphone: https://amzn.to/3t5zXeV
🚦Lights: https://amzn.to/3TQlX49
🎛️ Audio Interface: https://amzn.to/3TVFAIq
🎚️ Stream Deck: https://amzn.to/3zzm7F5