Why Bigger Context Windows Don’t Solve the RAG Problem

As context windows grow into the millions of tokens, many AI practitioners are questioning whether retrieval-augmented generation (RAG) is still necessary. If modern models can ingest entire libraries of documents, why bother with retrieval at all?

In this episode, Alex Bowcut, Head of Engineering at Sphere, explains why the answer depends on the application. Sphere uses AI to automate global tax compliance—an environment where getting the answer right isn’t enough. Every conclusion must be backed by the correct legal citation, and every decision must withstand expert review.

We explore how Sphere built TRAM (Tax Review and Assessment Model), a production AI system that combines retrieval, reasoning models, legal review workflows, reinforcement learning, and deterministic systems to help tax experts move nearly two orders of magnitude faster while maintaining accuracy.

Along the way, we discuss why RAG remains critical in high-stakes domains, how Sphere processes legal and regulatory documents from jurisdictions around the world, retrieval architectures, semantic chunking, dense versus sparse retrieval, expert feedback loops, and the challenges of building AI systems that people can actually trust.

🗒️ Full show notes: https://twimlai.com/go/769.

🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1

📖 CHAPTERS
===============================
00:00 Intro: Is RAG Obsolete?
01:24 Meet Sphere and TRAM
02:02 Why Tax Content Is Hard
03:54 How AI Supercharges Tax Experts
05:14 Alex Background Story
07:03 Messy Legal Data Ingestion
08:58 How TRAM Review Works
11:19 What Triggers Updates
15:13 Legal Review Not Labeling
16:08 Chunking and Indexing Law
21:21 Dense Versus Sparse Search
25:07 Taxonomy Driven Queries
27:55 RAG Is Dead Debate
29:55 Citations and Traceability
31:22 RFT for Accuracy Gains
34:50 Evals and Model Drift
37:47 LLM Reranking and Expansion
40:28 Chasing Nines Accuracy
42:39 Context Windows Impact
44:46 Costs and Latency Reality
45:40 Future Roadmap for TRAM
48:57 Personal AI Workflow Tools
50:30 Closing

🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Follow us on Twitter: https://twitter.com/twimlai
Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/