Post by Prateek Jalgaonkar

🌱 Why does .show() run everything in Spark? Transformation v/s Execution When learning Apache Spark, this confuses at first. You write: filter(), select(), groupBy(), agg() …but nothing happens. That’s because these are #transformations. Spark is only planning, not running. Then you call .show() which is an #execution👇 Spark now needs actual data to display. To get that data, Spark must: ->Read the source ->Apply all transformations ->Compute the result So Spark executes the entire plan. Question: “If I remove .show() from your code, will Spark do any work?” Correct answer: “No. Without an action, Spark will not execute the DAG.” No action → no work. #ApacheSpark #LearningInPublic #SparkBasics #BigData #DataEngineering