Burnaby, British Columbia, Canada
Data Engineer focused on cloud and streaming, building end-to-end pipelines across AWS and Azure and automating orchestration, ingestion, and ELT/ETL. I design Python DAGs in Airflow that run jobs on EMR/EC2 and coordinate cross-cloud tasks; ingest logs and application data with Azure Data Factory; and process daily and near-real-time workloads on Databricks (PySpark, Spark SQL, Structured Streaming), landing curated datasets in ADLS with partitioning and strong schema controls. I build Kafka consumers and Spark streaming jobs to process messages and upsert into operational stores; design warehouse objects and implement SCD Type 2 in T-SQL; and write optimized SQL/MySQL for transformations, validation, and reporting. I tune Snowflake for cost and performance (warehouse sizing, pruning, caching), migrate data from Oracle and Teradata into MongoDB and lake/warehouse zones using CDC and batch patterns with quality checks, and deliver big-data integrations on Hadoop/HDFS with Solr and Apache Storm, integrating enterprise flows via webMethods. I deploy Python models and data services as packages, REST APIs, and containerized microservices on Kubernetes with Docker; template and release ADF with JSON; script Unix utilities for parallel execution; right-size and monitor Databricks clusters; and run CI/CD for data apps and IaC with Terraform, including secrets, tests, reviews, and artifacted releases.
Built and modernized data platforms on Azure—delivering secure integrations, automated ELT, and analytics enablement. Migrated legacy Hadoop/Pig/Hive workloads to Spark/Synapse, improved pipeline performance, and enabled self-service reporting with Tableau/Power BI. Key contributions - Designed and implemented data integrations in Azure Data Factory (on-prem ↔ cloud), including hybrid connectivity and reusable pipeline patterns. Modernized legacy ETL into Azure Synapse–compatible architecture; converted batch jobs to Spark with Hive-compatible objects and Delta-style patterns where applicable. Developed a reusable migration framework to automate ELT from RDBMS sources to the Data Lake using Spark (Scala/Python) and Hive metastore objects. Built monitoring & alerting for pipelines using Python (health checks, SLAs, notifications) to improve reliability and incident response. Created analytics layers and dashboards in Tableau and Power BI; connected to diverse sources and modeled data for performant visualizations. Tuned Spark jobs (batch intervals, parallelism, memory) to reduce runtimes and stabilize streaming/batch workloads. Built/consumed SOAP (XML) and REST services to exchange data with external systems; hardened interfaces and error handling. Containerized data services with Docker and orchestrated runtimes on Kubernetes; integrated builds into CI/CD. Migrated and retired Hadoop/MapReduce/Pig/Hive jobs by re-platforming to Spark on Azure, simplifying ops and lowering maintenance overhead. Worked with common data formats: Avro, ORC, Parquet, CSV; enforced schema and partitioning best practices.
PepsiCo is a global food and beverage leader with iconic brands like Pepsi, Lay's, Gatorade, and Quaker, serving over 200 countries. I monitored data pipelines and infrastructure performance using CloudWatch and integrated auto-scaling mechanisms to optimize cost and performance Responsibilities: Leveraged AWS services like S3, Lambda, Glue, Redshift, and EMR to build data lakes and ensure seamless data ingestion, transformation, and storage Extracted and transformed the log data files from S3 by Scheduling AWS Glue jobs and loaded the transformed data into Amazon Elastic search. Developed Apache presto and Apache drill setups in AWS EMR cluster, to combine multiple databases like MySQL enables to compare results like joins and inserts on various data sources controlling through single platform, Build Jenkins jobs for CI/CD Infrastructure for GitHub repos. The AWS Lambda functions were written in Spark with cross - functional dependencies that generated custom libraries for delivering the Lambda function in the cloud. Performed raw data ingestion into, which triggered a lambda function and put refined data into ADLS, Looked into existing Java/Scala spark processing and maintained, enhanced the jobs. Integrated Kubernetes with cloud-native services, such as AWS EKS and GCP GKE, to leverage additional scalability and managed services. Zookeeper was utilized to manage synchronization, serialization, and coordination throughout the cluster after migrating from JMS Solace to Kinesis. Automated CI/CD pipelines using AWS Code Pipeline, Code Build, and Code Deploy, enabling faster and more reliable deployment of data engineering solutions, Used PowerBI as a front-end BI tool to design and develop dashboards, workbooks, and complex aggregate calculations. Developed Monitoring and notification tools using Python