Post by DoorDash

In Part 3 of our Building Ask DoorDash engineering series, we share the evaluation harness we built to make agent quality observable at scale. Agent quality is not just about the final response. Ask DoorDash helps consumers discover restaurants, shop for groceries, and place orders through multi-turn conversations. Behind each conversation, the agent calls tools, retrieves context, reasons through user intent, and acts on the user’s behalf. Evaluating that experience means judging the full session: what the user saw, what the agent did, and whether the outcome matched the user’s goal. Our harness turns full agent traces into scalable quality signals. It reconstructs sessions, applies task-specific rubrics, runs repeatable offline simulations, and uses a calibrated LLM judge to evaluate agent behavior end to end. That changed how we built and shipped Ask DoorDash: • Quality signal expanded from an average of 1 employee-submitted feedback item to 2,000 auto-graded sessions per day • Agent quality scores improved by 8 points ahead of nationwide launch, cutting error rates nearly in half and helping us meet our production launch bar • A comprehensive pre-ship regression test that previously took more than 6 hours by hand now runs in about 20 minutes • The same harness helped validate a base-model migration that reduced latency by 35% while preserving quality The post covers how we built the rubrics, transcript builder, simulator, and calibrated LLM judge behind this evaluation system, and how those pieces helped us catch trust-breaking failures sooner, prioritize recurring failure modes, and move faster with more confidence. Read more here: https://lnkd.in/edzJ_bEf