Post by Sonny Panesar

Incredibly proud to have mentored the team with Sanju Menon PhD from Singapore Management University (SMU) on their Final Year Project. The team tackled a critical and timely challenge: detecting audio deepfakes and spoofs in a banking call context Here is a quick look at what they achieved, the hurdles they overcame, and their biggest takeaways: 𝗢𝘂𝘁𝗰𝗼𝗺𝗲 They successfully built a comprehensive real-time spoof detection pipeline and Proof of Concept application  • 𝗥𝗼𝗯𝘂𝘀𝘁 𝗘𝗻𝘀𝗲𝗺𝗯𝗹𝗲 𝗠𝗼𝗱𝗲𝗹𝗶𝗻𝗴: They combined WavLM and HuBERT models using Mean and Fallback ensemble methods, successfully beating publicly available baseline models.  • 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲-𝗚𝗿𝗮𝗱𝗲 𝗖𝗹𝗼𝘂𝗱 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: The team engineered a fully native CI/CD pipeline on the Google Cloud Platform (GCP).  • 𝗦𝗽𝗲𝗲𝗱 & 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆: They achieved an impressive model inference time of under 2 seconds. Integrated Gemini to provide human-readable explainability for the risk scores based on extracted acoustic features. 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 The journey wasn't without its technical roadblocks.  • 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗖𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁𝘀: Training large models initially hit severe GPU quota limits on AWS, which prompted a strategic and successful migration to GCP to leverage student-led project compute access.  • 𝗗𝗮𝘁𝗮 𝗦𝗰𝗮𝗿𝗰𝗶𝘁𝘆: The team found it incredibly difficult to source publicly available, high-quality spoof datasets (such as those from ElevenLabs) that convincingly mimic modern threats. 𝗟𝗲𝘀𝘀𝗼𝗻𝘀 𝗟𝗲𝗮𝗿𝗻𝘁  • 𝗚𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗧𝗼𝘂𝗴𝗵: The team discovered that models performing exceptionally well on one dataset (like MLAAD) often struggled on others (like ASVSpoof5) and models tend to forget earlier training when exposed to high-variance data.  • 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗶𝘀 𝗦𝗲𝗰𝗼𝗻𝗱𝗮𝗿𝘆: Through testing across English and German, they found that when a model is well-trained on acoustic features, language and accents are not major barriers to accurate spoof detection.  • 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗗𝗲𝗽𝗲𝗻𝗱 𝗼𝗻 𝗣𝗿𝗶𝗼𝗿𝗶𝘁𝘆: They learned to align their ensemble strategies with their goals, utilizing Fallback ensembling to minimize Equal Error Rate (EER) and Mean ensembling to maximize accuracy in controlled environments. Congratulations Yong Ray Teo, Darius Ng, John Ernest, Shaun Zhou, Pang Hyin Ki, and Lim Wei Lun on delivering a phenomenal project and diving deep into the complexities of AI security! Special thanks to the SMU Professors Paul Griffin and Karthikeyan Kannan