Appearance
Spark Adaptive Query Execution (AQE)
06.12.2023
Adaptive Query Execution (AQE) reoptimizes and adjusts query plans based on runtime metrics collected during the execution of the query. As of Spark 3.0, there are three major features in AQE: coalescing post-shuffle partitions, converting sort-merge join to broadcast join, and optimizing skew join.
Spark Adaptive Query Execution (AQE)
Resources
- Performance Tuning - Spark 3.5.0 Documentation, Apache Spark
- Adaptive Query Execution: Speeding Up Spark SQL at Runtime, Databricks
- Adaptive Query Execution Demo, Databricks