Noida, Uttar Pradesh, India
Senior Engineering Manager leading a 10+ member data engineering team at Paytm, building and scaling data platforms that process 400M+ events/day (~2TB/day) across real-time and batch pipelines. I architect central ingestion pipelines that power multiple business verticals, ingesting from diverse sources at massive scale. Over the past year, my work has delivered 12 CR in cost savings through performance marketing pipeline optimization and $50K+ in infrastructure cost reductions across projects. What sets me apart — I'm deeply hands-on with AI: -> Built a fully automated AI Agentic system that handles end-to-end data pipeline development — from JIRA ticket to production deployment — with zero manual intervention -> Designed RAG systems for similar feature detection and internal knowledge bases that power this automated development workflow -> Developed and productionized multiple MCP servers enabling AI-driven tooling across teams -> Migrated legacy architectures to open-format frameworks, reducing end-to-end latency from 70-80ms to 1-2ms using Go 11+ years across fintech (Paytm), banking (Deutsche Bank), consulting (IBM, TCS), and fraud detection — building data systems that directly impact business outcomes. Tech: Spark, Kafka, Go, Scala, Python, Cassandra, Redis, AWS (EMR, S3, Lambda, DynamoDB, RDS), GCP, Airflow, Azkaban, Akka, Next.js, Node.js, Claude Code, Cursor
Leading a 10+ member data engineering team building Paytm's central data ingestion platform processing 400M+ events/day (~2TB/day) across all business verticals. Data Platform: - Own central ingestion pipelines processing ~2TB/day from diverse sources, serving as the backbone of Paytm's data infrastructure. - Delivered 12 CR in annual cost savings by re-architecting performance marketing pipelines - Saved $50K+ in infra costs through architecture optimization and resource right-sizing - Migrated pipelines to open-format frameworks (Iceberg/Hudi), saving ~$30K annually - Built real-time system using Go — reduced latency from 70-80ms to 1-2ms AI & Automation: - Built autonomous 5-agent AI pipeline: JIRA ticket → code generation → testing → PR review → production deployment — zero manual intervention - Designed RAG system over 1700+ features using ChromaDB for semantic search, duplicate detection, and context-aware Spark code generation - Developed 5 production MCP servers (50+ tools) enabling AI assistants to interact with Spark, Kafka, Grafana, ArgoCD, Jenkins, Azkaban, and 11,000+ datasets via natural language - Built AI-driven cost reporting across 32 infrastructure components Tech: Spark, Kafka, Go, Scala, Python, FastAPI, ChromaDB, Celery, Redis, AWS, GCP, Airflow, Azkaban, Claude Code, Cursor
Led architecture and delivery of Paytm's core data platform systems, driving technical decisions for large-scale pipeline infrastructure serving multiple business verticals. - Architected the central data ingestion framework handling 400M+ events/day from diverse sources — the system that now powers all verticals - Designed and built config-driven data pipeline frameworks enabling teams to ship pipelines 3x faster with standardized patterns - Led migration from legacy batch architectures to real-time event-driven systems using Kafka and Spark Streaming - Built the foundational data quality and validation frameworks used across all data pipelines - Drove adoption of open-format table frameworks (Iceberg/Hudi) for cost and latency optimization - Mentored 8+ engineers on system design, Spark optimization, and distributed systems best practices - Owned technical roadmap, conducted design reviews, and drove engineering excellence across the team - Collaborated with product and analytics teams to design data models serving business-critical dashboards and reporting Tech: Spark, Scala, Kafka, Python, Cassandra, Redis, AWS (EMR, S3, DynamoDB, RDS), Airflow, Azkaban, Akka, Go
Data Engineering
1. Designed and built ETL pipelines using PySpark and Hive, processing 500GB/day of data on AWS 2. Developed data solutions for clients in [energy/utilities/telecom] domains 3. Managed CI/CD pipeline deployments via Jenkins for automated data pipeline releases 4. Worked with Palantir platform for data pipelines Tech: Spark,Hive ,scala,python,AWS cloud, linux, oozie, hbase, palantir , S3 storage ,jenkins, Git, Agile framework, EC2 instances
1. Built fraud detection systems using PySpark, analyzing banking transaction patterns across multiple countries. 2. Designed rule engines for identifying fraudulent transactions based on geographic, behavioral, and transactional parameters 3. Implemented CI/CD pipelines using DevOps practices for automated deployments 4. Developed data quality frameworks and data modeling solutions
1. Designed and implemented applications on Hadoop ecosystem (Spark, HDFS, Flume, Oozie) for enterprise clients 2. Migrated client's legacy application architecture to distributed big data platforms 3. Built real-time streaming applications using Spark Streaming 4. Developed end-to-end Scala applications from scratch for data processing workflows 5. Worked across Finance, Retail, Oil & Chemical, Energy & Utilities, and Telecom domains