What you'll learn

Production Spark workloads often suffer from avoidable performance problems such as excessive shuffles, data skew, spill to disk, inefficient joins, poor file layouts, and improperly sized clusters. In this course, Optimize Apache Spark on Databricks, you'll gain the ability to diagnose and optimize Spark workloads running on Databricks. First, you'll learn how Spark executes workloads and how to use Spark UI, Databricks Run Details, and query plans to investigate performance issues. Next, you'll discover techniques for reducing shuffles, choosing effective join strategies, handling skew, and using caching appropriately. Finally, you'll learn how to optimize Delta Lake tables, select appropriate compute resources, leverage Photon acceleration, and evaluate cost-performance tradeoffs. When you're finished with this course, you'll have the skills and knowledge needed to systematically optimize Spark workloads in production Databricks environments.