Exploring the Apache Flink API for Processing Streaming Data
Flink is a stateful, tolerant, and large scale system which works with bounded and unbounded datasets using the same underlying stream-first architecture.
What you'll learn
Apache Flink is built on the concept of stream-first architecture where the stream is the source of truth. In this course, Exploring the Apache Flink API for Processing Streaming Data, you will perform custom transformations and windowing operations on streaming data.
First, you will explore different stateless and stateful transformations that Flink supports for data streams such as map, flat map, and filter transformations.
Next, you will learn the use of the process function and the keyed process function which allows you to perform very granular operations on input streams, get access to operator state, and access timer services.
Finally, you will round off your knowledge of the Flink APIs by performing transformations using the table API as well as SQL queries.
When you are finished with this course you will have the skills and knowledge to design Flink pipelines, access state and timers in Flink, perform windowing and join operations, and run SQL queries on input streams.
Table of contents
- Version Check 0m
- Prerequisites and Course Outline 3m
- Flink APIs for Stream Processing 4m
- Data Stream Connectors 2m
- Demo: Environment and Maven Project Setup 6m
- Demo: Configuring Dependencies in IntelliJ 3m
- Demo: Map and Filter Transformations 5m
- Demo: Streaming File Sink 3m
- Demo: Running Flink Applications on the Local Flink Cluster 5m
- Demo: Flatmap Transformations 4m
- Demo: Keyed Streams and Transforms on Keyed Streams 6m
- Demo: Using POJOs in Data Streams 4m
- Demo: Connected Streams 5m
- Async IO 5m
- Process Functions 3m
- Demo: Count Accumulators and Histogram Accumulators 6m
- Demo: Setting up a Custom Streaming Source 4m
- Demo: Transforming Data Using a Process Function 2m
- Demo: Accessing State and Registering Timers Using a Keyed Process Function 7m
- Demo: Processing Connected Streams Using the Keyed CoProcess Function 6m
- Demo: Processing Connected Streams Per Key Using a Keyed CoProcess Function 5m
- Side Outputs 2m
- Demo: Side Outputs 5m
- The Broadcast State Pattern 2m
- Demo: Broadcast State 8m
- Stateful Operations on Streams 2m
- Types of Windows 7m
- Event Time, Ingestion Time, and Processing Time 3m
- Watermarks and Late Data 3m
- Flink Triggers 3m
- Demo: Event Time Tumbling Windows 8m
- Demo: Custom Aggregate Functions with Tumbling Windows 3m
- Demo: Event Time Sliding Windows and Window All Operations 5m
- Demo: Processing Time Session Windows 4m
- Demo: Global Windows and Triggers 4m
- Window Joins 4m
- Interval Joins 3m
- Demo: Joins 8m
- Blink and Flink 3m
- Dynamic Tables and Continuous Queries 5m
- Demo: Introducing the Table API and SQL Queries 4m
- Demo: Creating Tables from Data Streams 5m
- Demo: Position-based and Name-based Mapping of Column Names 4m
- Demo: Selection, Projection, and Filtering Operations on Tables 4m
- Demo: Performing Selection, Projection, and Filtering Operations Using SQL Queries 3m
- Demo: Grouping Operations Using Dynamic Tables 3m
- Demo: Converting Tables to Data Streams Using Append Mode and Retract Mode 4m
- Demo: Performing Joins Using Tables 7m
- Demo: Performing Windowing Operations Using the Table API 5m
- Summary and Further Study 2m
About the author
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing ... more