Post by Appen

New Research at Association for Computational Linguistics 2026 How well do today's leading multilingual LLMs capture culture, not just language? At ACL 2026, Appen is presenting "Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs, a new study evaluating whether AI translations actually resonate with native speakers, beyond grammatical correctness. The title comes from a real example. An LLM translated the Valentine's Day pun "Will you brie mine?" as "Be my cheese?" Grammatically correct. Culturally meaningless. Our study evaluated: • 7 leading multilingual LLMs • 15 target languages • 75 native-speaker evaluators Key findings: • The best-performing models averaged only 2.1/3 for cultural appropriateness. • Puns and idioms remain a consistent failure mode across every model. • Popular automatic metrics such as BLEU and COMET often miss these issues. • Top-performing models clustered closely together, suggesting larger models alone won't solve cultural-pragmatic understanding. • Some languages saw extremely low scores, including 0.15/3 for Urdu and 0.16/3 for Swahili. As AI becomes increasingly global, evaluating cultural nuance is just as important as evaluating translation accuracy. If you're attending #ACL 2026, join Dr Casey Ford from Appen at the GEM Workshop (Natural Language Generation, Evaluation, and Metrics) on Saturday, July 4, where she'll be presenting this work. Appen is to be a Platinum Sponsor of ACL 2026. Stop by Booth #2 to meet the team, learn more about this research, and discuss how high-quality multilingual human data is helping advance the next generation of AI. Link: https://lnkd.in/efcHE9fb #ACL2026 #MachineTranslation #MultilingualAI #LLMs #GenerativeAI #NLP #AIResearch #LanguageAI