Toronto, Ontario, Canada
Lead Data engineer close to 9+ years of experience in Information Technology .An engineer who is solving real business problems and enabling businesses to harness the power of big data analytics. Has worked in multiple advanced big data projects end to end, from conceptualizing, designing, and architecting to delivering the product. Have extensive experience in working as a data engineer in the financial , insurance and retail industry. Skilled in Python,Scala, AWS,Apache Spark, Hadoop eco-system(Hive,sqoop, etc.),
• Led enterprise AML data platforms processing TB-scale transaction and customer data for monitoring, investigations, regulatory reporting, and model consumption, with integration of KYC data (Fenergo) where required. • Designed and governed end-to-end AML pipelines with full source-to-target lineage, reconciliation, metadata, and data quality controls, delivering 99.9% accurate, audit-ready datasets. • Translated complex AML regulatory and compliance requirements into enforceable Spark-based data logic, partnering with AML, Risk, Compliance, and Audit teams. • Built Apache Spark pipelines in Python, leveraging DataFrames/Datasets, strict schema enforcement, partitioning strategies, and checkpointing for reliable and auditable processing. • Embedded governance by design through lineage frameworks, control totals, reconciliation logic, and metadata management, reducing audit findings by 40%+. • Expertise in Databricks, implementing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake tables, ensuring ACID compliance, schema evolution, and performance optimization. • Optimized high-volume AML Spark workloads (joins, broadcast thresholds, file sizing, partition pruning, AQE), achieving 100% batch SLA adherence and reducing processing windows by 30–50%. • Mentored and led 10+ senior engineers and tech leads, building deep AML domain expertise and a governance-first engineering culture. • Deep expertise in Apache Spark (Scala) and the Hadoop ecosystem (HDFS, Hive, HBase, Sqoop), processing terabyte-scale datasets using RDDs and Data Frames. • Proven ability to design and optimize high-performance data pipelines using Spark, Hive, HDFS, AWS Glue, and Athena. • Advanced knowledge of Hive and Spark optimization techniques includes partitioning, bucketing, window functions, UDFs, compression, and memory tuning.
* Led enterprise-scale AML reconciliation platforms for validating and reconciling multi-source banking data against downstream regulatory and compliance systems using Python, Scala, Bash, and SQL-based processing frameworks, processing 50M+ records daily. * Designed and developed scalable reconciliation microservices capable of processing high-volume transactional datasets, ensuring data accuracy, completeness, auditability, and regulatory compliance across critical AML workflows. * Built and optimized complex SQL-based reconciliation and validation frameworks integrated with distributed processing and mainframe batch environments (JCL), improving processing performance by 35%. * Engineered automated data quality, exception-handling, and reconciliation control frameworks to identify mismatches, duplicate records, and integrity breaks across heterogeneous banking systems, reducing reconciliation failures by 40%. * Supported regulatory reporting and AML compliance initiatives across enterprise banking platforms by implementing robust reconciliation controls, audit-ready pipelines, and end-to-end data traceability solutions. * Collaborated with AML, compliance, risk, and infrastructure teams to deliver scalable reconciliation architectures aligned with enterprise governance and regulatory reporting standards. * Led root cause analysis and resolution of complex upstream/downstream data inconsistencies, improving SLA adherence, operational stability, and production reliability across mission-critical AML systems.
• Designed end-to-end pipeline designs and data solutions on Hadoop to build robust and efficient data flow solutions • Proficient in creating data pipelines in spark/Scala to handle end-to-end data cleaning, data transformation, and estimation techniques • Perform engineering activities for performance tuning and deep monitoring of applications and services and In-depth understanding of Spark architecture including Spark Core, Spark SQL, Transformer Pipelines, and Spark Data Frames. • Optimized Spark pipelines in Scala by identifying and removing data skewness in complex logic, which improved the logic performance by more than 50% • Resolved critical production issues in different areas across the big data, Spark pipelines, Hadoop-ecosystem, and Oracle in a timely manner • Advised the team of 7+ people on data pipeline development and deployment continuously to unblock and deliver projects that the team owned. • Provides technical guidance to a team of engineers who are developing the data pipelines, and innovating in core areas of big data and large-scale data processing.
• Built end-to-end ETL pipelines using SQL and Snowflake, transforming raw operational data into analytics-ready datasets for fulfillment and business decision-making. • Optimized Trino queries over large-scale datasets, enabling fast ad-hoc analysis and improved performance for Product and Operations teams. • Created and maintained standardized (canonical) data models and metrics in Snowflake to ensure consistency across Analytics, Finance, Product, and Operations. • Developed interactive dashboards in Mode and Sigma to track KPIs, uncover trends, and support data-driven operational decisions. • Partnered with cross-functional stakeholders to translate ambiguous business requirements into scalable analytics solutions and actionable insights. • Used Docker-based local environments to test ETL pipelines and data quality checks, ensuring reliability, reproducibility, and smooth deployments.
4+ years of experience in the field of Banking & Financial Services (BFS) with strengths in Development, Data warehousing, ETL, and Quality Assurance. • Experienced in working with various tools in the Hadoop Ecosystem including Pig, Hive, HBase, HDFS, Sqoop, Spark, Kafka, Yarn, Oozie, and Zookeeper. • Excellent Understanding of Hadoop architecture and the daemons of Hadoop including Name-Node, Data Node, Job Tracker, Resource Manager. • Experienced in building Data-pipe lines using Big Data Technologies and ingesting data into Data Warehouse using various data loading techniques. • Vast experience in developing the code and validating source and target data as per transformation rule by creating and executing SQL queries and HIVE queries. • Experience working in processing Delimited, CSV and XML files, data analysis and data processing for projects in Hadoop file system and SQL Server • Extensive experience in working on Data Warehousing systems by Extract, Transform, and Load (ETL/ELT) processes using Pentaho, HIVE, and UNIX. • Created various shell scripts to assist with automated batch execution of the application and its different stages of testing (unit testing/QA/UAT). Developed scripts to automate the QA scripts generation (HQL). • Diversified Experience in ETL testing, Functional testing, Integration testing, End to end testing. • Experienced in handling multiple tasks and work independently as well as in a team and leading various team sizes and coordinating with both Off-shore and On-shore teams to deliver the projects within the SLA. • Exceptional interpersonal skills with the ability to communicate to all levels of management. • Career Interests: Project Management, Microsoft Azure, Agile, Big Data.