Modeling Streaming Data for Processing with Apache Beam
The Apache Beam unified model allows us to process batch as well as streaming data using the same API. Several execution backends such as Google Cloud Dataflow, Apache Spark, and Apache Flink are compatible with Beam.
What you'll learn
Streaming data usually needs to be processed real-time or near real-time which means stream processing systems need to have capabilities that allow them to process data with low latency, high performance and fault-tolerance. In this course, Modeling Streaming Data for Processing with Apache Beam, you will gain the ability to work with streams and use the Beam unified model to build data parallel pipelines. First, you will explore the similarities and differences between batch processing and stream processing. Next, you will discover the Apache Beam APIs which allow one to define pipelines that process batch as well as streaming data. Finally, you will learn how windowing operations can be applied to streaming data. When you are finished with this course, you will have a strong grasp of the models and architectures used with streaming data and be able to work with the Beam unified model to define and run transformations on input streams.
Table of contents
- Introducing Apache Beam 6m
- Pipelines, PCollections, and PTransforms 5m
- Input Processing Using Bundles 4m
- Driver and Runner 3m
- Demo: Environment Set up and Default Pipeline Options 6m
- Demo: Filtering Using ParDo and DoFns 7m
- Demo: Aggregagtions Using Built-in Transforms 1m
- Demo: File Source and File Sink 8m
- Demo: Custom Pipeline Options 6m
- Demo: Streaming Data with the Direct Runner 7m
- Demo: Word Count 5m
- Stateless and Stateful Transformations 5m
- Types of Windows 7m
- Event Time and Processing Time 4m
- Watermarks and Late Data 3m
- Demo: Using Fixed Windows with in Memory Data 6m
- Demo: Using Fixed Windows with Input Files 7m
- Demo: Using Sliding Windows with in Memory Data 3m
- Demo: Using Sliding Windows with Input Files 3m
- Demo: Session Windows 3m
- Demo: Global Windows 1m
- Triggers 5m
- What, Where, When, and How in Stream Processing 4m
- Summary and Further Study 1m
About the author
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing ... more