What you'll learn

Processing data the moment it arrives—rather than in nightly batches—raises challenges that batch pipelines never face: ingesting new files incrementally, tracking progress reliably, choosing the right triggers and output modes, and correctly handling late or out-of-order events. In this course, Build Streaming Pipelines with Apache Spark on Databricks, you’ll gain the ability to design and operate fault-tolerant streaming pipelines using Spark Structured Streaming and Delta Lake. First, you’ll explore the core streaming and incremental processing patterns and how checkpoints make pipelines reliable. Next, you’ll discover how to ingest cloud data incrementally with Auto Loader and write results to Delta tables with the right triggers, output modes, and schema handling. Finally, you’ll learn how to apply event-time windows, watermarks, and stream-static and stream-stream joins to aggregate and enrich streaming data. When you’re finished with this course, you’ll have the skills and knowledge of stream processing on Databricks needed to build production-ready, real-time data pipelines.