Post by Karyna Naminas

Most autoraters still treat human judgment as a single label: A or B. That makes evaluation brittle because disagreement and uncertainty get erased. New research from Google DeepMind, Vanderbilt University, Cornell University, University of Alberta, Virginia Tech, and Scale AI reframes autorating as a probabilistic task. Instead of predicting one verdict, the model learns the distribution of human preferences – how often people choose A over B. They train it two ways: 1. Supervised fine-tuning when multiple annotators per item exist 2. Reinforcement learning with Brier rewards when only one label per item is available RL turns out more data-efficient. With 10x more prompts but single labels, it outperformed dense multi-annotator SFT while staying well-calibrated. 💡Human annotation plays a central role. The researchers show that how you collect and structure labels matters more than how many you have. Sparse single-label data, when paired with reinforcement learning and proper scoring rules, can outperform smaller dense datasets with multiple annotators. A practical takeaway for teams balancing annotation budgets and data quality. On 105K JudgeLM prompts, their calibrated autoraters reduced error by up to 45% and matched or beat GPT-4 on human alignment benchmarks – while nearly eliminating order bias. The key idea is simple but important: reliable autoraters estimate how much people would disagree. That depends on human data that preserves disagreement instead of averaging it out. Variation across annotators becomes part of the signal, not noise to remove. And that might be the most honest way to represent human judgment in machine evaluation. 🧠 Kudos to the team: Zhuohang Li, Xiaowei Li, Chengyu Huang, Guowang Li, Katayoon(Kati) Goshvadi, Bo Dai, Dale Schuurmans, Xin Zhou, Hamid Palangi, Yiwen Song, Palash Goyal, Murat Kantarcioglu, Brad Malin, and Yuan (Emily) Xue. All details and results in the paper – link in the comments. #LLM #AIAlignment #RLHF #Calibration #MachineLearning #DataAnnotation #AIResearch #LLMEvaluation #HumanFeedback #ModelTraining