Post by Rajagopal Senthil Kumar

A query returns 1 million rows. But did Spark process only 1 million rows? Not necessarily. This is one of the most important concepts to understand when tuning Databricks and Spark workloads. Consider a simple join: * Customer Table → 1 Million rows * Sales Table → 10 Million rows * Final Result → 1 Million rows Most performance discussions focus on the output. However, Spark performance is primarily influenced by what happens before the result is produced: • How much data is read from storage • How partitions are distributed • Whether shuffle occurs • Whether sorting is required • How much data moves across the network • Which join strategy Spark selects This is why two queries returning exactly the same result can have dramatically different execution times. A Broadcast Join may complete in seconds because it avoids large-scale shuffle operations. A Sort Merge Join may take significantly longer because data must be redistributed, sorted, and merged across executors. The same principle applies to GROUP BY operations. The aggregation itself is usually not the expensive part. The expensive part is moving rows with the same key to the same executor so that Spark can perform the final aggregation. A useful way to think about Spark processing time is: Total Processing Time = Read Time + Shuffle Time + Sort Time + Compute Time + Write Time Where: Read Time = Data Size ÷ Storage Throughput Shuffle Time = Data Moved ÷ Network Throughput Sort Time = Data Volume × CPU/Memory Cost Compute Time = Join / Aggregation / Transformation Cost Write Time = Output Size ÷ Storage Throughput This explains why reducing shuffle volume often delivers a much bigger performance improvement than optimizing the join itself. In this infographic, I break down: • Broadcast Hash Join vs Sort Merge Join • How many records Spark actually reads • Why shuffle happens • Why GROUP BY triggers data movement • Partition pruning and its impact • Processing time calculation components • Practical optimization techniques used in Databricks One takeaway from large-scale Spark workloads: Performance is usually determined by how much data Spark moves, not how much data it returns. Read Less → Shuffle Less → Sort Less → Finish Faster References: Apache Spark SQL Performance Tuning https://lnkd.in/grX3UAWp Databricks Performance Best Practices https://lnkd.in/g4Vddz26 Databricks Photon Engine https://lnkd.in/g-9UWZmM #Databricks #ApacheSpark #SparkSQL #DataEngineering #DeltaLake #Lakehouse #PerformanceTuning #BroadcastJoin #Photon #BigData #DataArchitecture #AnalyticsEngineering #SENDAKIL