Post by Maithra Raghu

Excited to be releasing FrontierFinance, the largest and most challenging open benchmark for evaluating AI Agents across the full investment workflow! 𝐅𝐫𝐨𝐧𝐭𝐢𝐞𝐫𝐅𝐢𝐧𝐚𝐧𝐜𝐞 𝐢𝐬 𝐬𝐮𝐛𝐬𝐭𝐚𝐧𝐭𝐢𝐚𝐥𝐥𝐲 𝐡𝐚𝐫𝐝𝐞𝐫 𝐭𝐡𝐚𝐧 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐟𝐢𝐧𝐚𝐧𝐜𝐞 𝐛𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐬: Existing benchmarks like Finance Bench and Finance Agent focus almost entirely on data extraction. FrontierFinance spans diverse use cases including: Screening & Discovery, Company Research, Sector/Industry/Macro, Earnings & Events, and Coverage & Catalyst Monitoring. 𝐂𝐫𝐞𝐚𝐭𝐞𝐝 𝐟𝐨𝐫 𝐚𝐦𝐛𝐢𝐠𝐮𝐨𝐮𝐬 & 𝐥𝐨𝐧𝐠-𝐡𝐨𝐫𝐢𝐳𝐨𝐧 𝐀𝐠𝐞𝐧𝐭𝐬: FrontierFinance has 220 examples, paired with 11,543 expert-crafted rubrics, following Samaya's Criteria Eval methodology. Rubrics are critical in evaluating the reasoning and steps behind a true expert-level output that covers the key details and factors. 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐧 𝐥𝐞𝐚𝐝𝐢𝐧𝐠 𝐟𝐫𝐨𝐧𝐭𝐢𝐞𝐫 & 𝐨𝐩𝐞𝐧-𝐬𝐨𝐮𝐫𝐜𝐞 𝐦𝐨𝐝𝐞𝐥𝐬: We evaluated Claude Fable 5, Claude Opus 4.8, GPT 5.5, Gemini, open-source models including GLM and DeepSeek, and others. Samaya's AI system achieved state-of-the-art accuracy at 50.8%, at 4x lower inference cost than Fable 5. Next best was Claude Fable 5 (49.2%), then Claude Opus 4.8 (45%) and GPT 5.5 (43.5%). 𝐃𝐚𝐭𝐚𝐬𝐞𝐭, 𝐦𝐞𝐭𝐡𝐨𝐝𝐨𝐥𝐨𝐠𝐲, 𝐜𝐨𝐝𝐞 𝐫𝐞𝐥𝐞𝐚𝐬𝐞: We're releasing the benchmark, methodology, our extensive evaluation results. Link is in the comments! 𝐅𝐮𝐭𝐮𝐫𝐞 𝐫𝐞𝐥𝐞𝐚𝐬𝐞𝐬: FrontierFinance was curated from Samaya's larger internal evaluation set of ~5000 examples. We plan to release subsequent, harder benchmarks and a detailed technical report.