Mumbai, Maharashtra, India
Azure Data Engineer with 6 years of experience in designing and implementing scalable data solutions on Microsoft Azure. Expertise in Azure Data Factory and Microsoft Fabric for orchestrating complex ETL workflows and Azure Databricks for advanced data transformation using PySpark and Spark SQL. Skilled in building robust data pipelines, optimizing performance, and ensuring data quality across Data Lake (ADLS) and Synapse Analytics environments. Technical Expertise: Microsoft Fabric, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, PySpark, SQL Server.
Working as Big Data Developer for Insurance company client which Include Extracting data from multiple sources using Azure Synapse Pipelines and loading it in Raw Layer which further leads to transforming the data as per client requirement using Azure Databricks Py-spark coding and loading it to Gold Layer.
• The data from refineries were collected with the help of IoT sensers and captured by Azure Time Series Insights then it is loaded Azure Data Lake Storage with the help of Azure Synapse Analytics. • With the help of CI/CD pipeline and ansible scripts we are running pipeline in Azure DevOps to deploy various components of Azure for each business units. Scripts are developed to migrate data from Azure time series insights to Azure Data explorer. • Tickets are raised by the user if there is any data not loaded and any mismatch of data and we have to resolve that issue as well.
• The goal of the team was to migrate data of Delivery Insights data from IaaS to PaaS. Analyzed and developed the ways to ingest the data from CSV, Excel in Azure Synapse Analytics. The data flow in SSIS should be analyzed and according to it we developed the flow in PaaS. • With the help of Azure Storage Explorer created different folders for prod, test and dev files. • Developed External Polybase Tables in SQL Server to read data from CSV files stored in Azure Blob Storage. Developed different stored procedures based on the type of data in files to load history as well current data. • Developed various pipelines in Azure Data Factory V2 to call the stored procedures and perform other tasks like logging, Pause/Resume Synapse Analytics SQL Pool, lookup, copy activities to create a data flow like SSIS. • Debugging the pipeline if there are any errors and created triggers for the same. • After the data loaded in PaaS databases performed different testing on data to check the row count and if there is any mismatched in the data from IaaS data if there are any discrepancies made the changes wherever needed and create a document for each file. • In order to further test performed testing on Power Bi where imported data and created measures to test the data and also explored some visualizations.
Learned database management and web development through .NET programming language. Created sample web-based application on Visual Studio, handling end to end development both backend and frontend on our own using SQL server as backend, JavaScript as frontend and MVC Framework.