Greater Kolkata Area
As a data engineer with almost 4 years of hands-on experience, I specialise in developing and managing robust data pipelines for retail analytics using Databricks and Azure. My work includes implementing Bronze-Silver-Gold medallion architecture tables, enhancing Kafka streaming pipelines to process high-volume online order data, and performing complex transformations with PySpark and Spark SQL. My expertise includes technologies such as PySpark, Spark SQL, Python, SQL, Databricks, Azure Data Factory, Azure Data Lake Storage and Airflow. I have a proven track record of successfully cleaning transforming and loading large datasets for Marks & Spencer’s BEAM data platform. I am currently exploring new opportunities that align with my data engineering expertise and career goals.
Client: Marks & Spencer Built and managed multiple batch and micro-batch based pipelines for retail analytics using Databricks and Azure. • Developed an configurable and reusable ingestion pipeline and implemented Bronze–Gold medallion architecture tables for ingesting and transforming IVR data from Twilio’s Azure PostgreSQL OLTP database. This enabled historical reporting and analytics. • Built and enhanced real-time Kafka streaming pipelines ingesting online order's data from IBM Sterling via Kafka, processing over 1.5 million records daily. • Developed and enhanced multiple Bronze–Silver-Gold medallion architecture tables, implementing complex transformations with PySpark and Spark SQL.