Post by Appen

Most AI agents aren't slow because they're "thinking." They're slow because they're managing context. On the latest episode of The Data Layer, Karla Heredia (Director - Gen AI Delivery, Appen) sat down with Alexander Whedon (Co-founder & CTO, Subquadratic) and Sergio Bruccoleri (VP of Delivery, Appen) to talk about where agentic AI actually breaks down - and how long context is starting to fix it. A few things: → Transformers scale quadratically. Double the input, quadruple the compute. Alex put it bluntly: it's the kind of complexity that would fail a technical interview - and the industry has built on it for nearly a decade. → In one benchmark, a frontier model took 66 steps to solve a task on TAU-Bench Pro. Roughly 90% of those steps were just re-reading context, not executing or reviewing. The bottleneck isn't intelligence - it's memory management. → Long-horizon memory is the next frontier. Not memory across a session, but across weeks - which matters as agents move from coding assistants to general workforce tools. → Benchmarking is still catching up. Appen independently validated Subquadratic's model at up to 12 million tokens - 98% retrieval accuracy - but as Alex pointed out, retrieval isn't reasoning. There's still no robust benchmark for reasoning at multi-million-token scale. That's genuinely greenfield work. → The "bitter lesson" is playing out again. Every layer of human curation - chunking, prompt engineering, conditional logic - narrows what a system can generalize to. Remove the constraints on context, and a lot of that curation becomes unnecessary. → Enterprises are sitting on tabular data, not text. Sergio's point that ~99% of enterprise data is tabular, not document-based, is a good reminder that RAG isn't always the right first move. Full conversation link in comments. Would love to hear how others are thinking about long context in production systems. #AI #ArtificialIntelligence #AIAgents #LLM #DataInfrastructure #MachineLearning