Post by Rajagopal Senthil Kumar

If a table contains 1 billion records, does Delta Sharing copy all 1 billion records to the consumer environment before processing? This is one of the biggest misconceptions I encounter when discussing Databricks Delta Sharing. Many people imagine: Producer Storage → Transfer 1 Billion Records → Consumer Storage → Process → Dashboard But that’s not how Delta Sharing works. Let’s consider a real scenario: • Sales table contains 1 Billion records • Partitioned by sales_country • Stored as Delta files in the producer account • Shared to another Databricks workspace using Delta Sharing Now a business user asks Genie: “Show total sales by country for the last 90 days.” Genie generates SQL and submits it to a Databricks SQL Warehouse in the consumer account. The interesting part is what happens next. The SQL Warehouse first reads the Delta transaction log and metadata: • Available partitions • File statistics • Min/Max values • File locations Only then does Spark determine which files are actually required. If the table contains 3 years of history and the query only needs the last 90 days, partition pruning and data skipping can dramatically reduce the amount of data read. Instead of scanning 1 billion records, Spark may read only a small fraction of the table. The consumer compute then: • Reads only required files directly from producer storage • Processes data in parallel across executors • Performs local aggregations in memory • Shuffles only aggregated results when required • Returns a small result set to Genie, Tableau, or dashboards One important point: The 1 billion records are not copied and stored in the consumer account. The data remains in the producer storage. The consumer compute reads the required files on demand, processes them in memory, and returns only the final result. This is why Delta Sharing can achieve performance very close to direct table access when the producer side is properly optimized. The biggest performance factors are usually: ✓ Partition Strategy ✓ File Size Optimization ✓ Liquid Clustering / Z-Order ✓ Data Skipping Statistics ✓ Query Design ✓ Shuffle Volume One lesson that stood out to me: A 1-billion-row table does not automatically mean a 1-billion-row scan. Good table design allows Spark to read less, shuffle less, and return results much faster. References: Databricks Delta Sharing https://lnkd.in/gh88K-DN Databricks-to-Databricks Sharing https://lnkd.in/g-P2k8nj Delta Lake Data Skipping https://lnkd.in/ggvNB2TX Databricks SQL Warehouse https://lnkd.in/gJ4uB5FE #Databricks #DeltaSharing #ApacheSpark #DataEngineering #Lakehouse #DeltaLake #BigData #Genie #AnalyticsEngineering #PerformanceTuning #SENDAKIL