Spark Adaptive Query Execution (AQE)

Adaptive Query Execution (AQE) reoptimizes and adjusts query plans based on runtime metrics collected during the execution of the query. As of Spark 3.0, there are three major features in AQE: coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and optimizing skew join.

06.12.2023

Sketchnotes of Spark's Adaptive Query Execution Spark Adaptive Query Execution (AQE)

Resources

Performance Tuning - Spark 3.5.0 Documentation, Apache Spark
Adaptive Query Execution: Speeding Up Spark SQL at Runtime, Databricks
Adaptive Query Execution Demo, Databricks

Spark Adaptive Query Execution (AQE) ​

Spark Adaptive Query Execution (AQE)