Post by Sayna Ebrahimi

📢 So proud of the incredible work led by our exceptional intern, Shayne Longpre, on multilingual scaling laws: 𝗔𝗧𝗟𝗔𝗦 🗺️! Scaling laws tell us how to build efficient AI models. The problem: they're almost entirely based on English. That leaves practitioners building multilingual models—which serve billions of users—flying blind. I'm excited to share research that changes this. 𝗔𝗧𝗟𝗔𝗦 (Adaptive Transfer Scaling Laws) represents the largest public multilingual scaling study to date: 774 experiments spanning 10M to 8B parameters, trained on 400+ languages, and evaluated across 48 languages. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: Prior scaling laws focused overwhelmingly on English, leaving practitioners without guidance for efficient multilingual model development. Our work fills this gap by:  • Significantly outperforming prior scaling laws (often >0.3 R² improvement) through explicit modeling of cross-lingual transfer, data repetition, and multilingual capacity constraints.  • Providing empirically-measured transfer scores across 1,444 language pairs—the most comprehensive resource for understanding which languages benefit or interfere with each other during training.  • Deriving practical scaling equations that tell practitioners how to efficiently allocate compute budgets across model size, data, and language coverage. 𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁: Shared script matters more than shared language family for positive transfer. Languages using the same writing system show dramatically better transfer (mean: -0.23) versus different scripts (mean: -0.39). This work enables more equitable AI development by providing the tools to efficiently build models that serve billions of non-English speakers. This was a massive team effort from MIT, Stanford, UoW, Google Cloud AI, and Google DeepMind! A huge thank you to all the collaborators Sneha Kudugunta, Niklas Muennighoff, I-Hung Hsu, Isaac Caswell, Alex 'Sandy' Pentland, Sercan Arık, Chen-Yu Lee.