Post by Himanshi Yadav

🚀 Project Showcase: Azure Databricks End-to-End ETL & Visualization Project ☁️ Thrilled to share my recent project where I built a complete data engineering pipeline leveraging the power of Azure Databricks for ETL (Extract, Transform, Load) and data visualization using Power BI! 🔹 Key Highlights: • Developed an end-to-end data pipeline in Databricks for seamless data ingestion, transformation, and loading. • Implemented the Medallion Architecture (Bronze, Silver, and Gold layers) to ensure data quality and modularity. • Utilized Spark Streaming with Autoloader for real-time data processing. • Designed a Dimensional Data Model (Star Schema) for efficient analytics. • Built a Declarative Pipeline for Gold Layer data modeling. • Implemented Slowly Changing Dimensions (SCD Type 1 & Type 2) for maintaining historical data accuracy. • Leveraged PySpark for distributed data processing in a clustered environment. • Automated the entire workflow using Databricks Workflows for scheduling and orchestration. • Created a single unified pipeline capable of handling both historical and incremental data. • Visualized enriched Gold Layer data using Power BI dashboards. 💡 This project strengthened my understanding of modern data architecture, streaming data processing, and data orchestration in a real-world cloud environment. You can explore the full demo and code here: 🎥 Video Demo: https://lnkd.in/gwXkPQhs 📂 GitHub Repository: https://lnkd.in/ge2yTWe9 #Azure #Databricks #PySpark #ETL #DataEngineering #PowerBI #DataPipeline #BigData #CloudComputing #MedallionArchitecture #Autoloader #SparkStreaming