Conceptualizing the Processing Model for Apache Flink

Flink is a stateful, tolerant, and large-scale system with excellent latency and throughput characteristics. It works with bounded and unbounded datasets using the same underlying stream-first architecture, focusing on streaming or unbounded data.
Course info
Level
Intermediate
Updated
Nov 20, 2020
Duration
3h 59m
Table of contents
Course Overview
Getting Started with Apache Flink
Executing and Monitoring Streaming Queries
Performing Stateless Transformations on Streams
Performing Stateful Transformations on Streams
Exploring the Checkpointing Architecture in Flink
Description
Course info
Level
Intermediate
Updated
Nov 20, 2020
Duration
3h 59m
Description

Apache Flink is built on the concept of stream-first architecture, where the stream is the source of truth. Flink offers extensive APIs to process both batch as well as streaming data in an easy and intuitive manner.

In this course, Conceptualizing the Processing Model for Apache Flink, you’ll be introduced to Flink Architecture and processing APIs to get started on your data analysis journey.

First, you’ll explore the differences between processing batch and streaming data, and understand how stream-first architecture works. You’ll study the stream-first processing model that Flink uses to process data at scale, and Flink’s architecture which uses JobManager, TaskManagers, and task slots to execute the operators and streams in a Flink application in a data-parallel manner.

Next, you’ll understand the difference between stateless and stateful stream transformations and apply these concepts in a hands-on manner in your Flink stream processing. You’ll process data in a stateless manner using the map(), flatMap(), and filter() transformations, and use keyed streams and rich functions to work with Flink state.

Finally, you’ll round off your understanding of the state persistence and fault-tolerance mechanism that Flink uses by exploring the checkpointing architecture in Flink. You’ll enable checkpoints and savepoints in your streaming application, see how state can be restored from a snapshot in the case of failures, and configure your Flink application to support different restart strategies.

When you’re finished with this course, you’ll have the skills and knowledge to design Flink pipelines performing stateless and stateful transformations, and you’ll be able to build fault-tolerant applications using checkpoints and savepoints.

About the author
About the author

A problem solver at heart, Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework.

More from the author
More courses by Janani Ravi
Section Introduction Transcripts
Section Introduction Transcripts

Course Overview
[Autogenerated] Hi, My name is Jonny Robbie, and welcome to the scores on conceptualizing the processing model for a party. Flink A little about myself. I have a master's degree in electrical engineering from Stanford and have worked at companies such as Microsoft, Google and Flip Cart at Google. A was one of the first engineers working on real time collaborative editing in Google Dogs and I hold four patterns for its online technologies. I currently work on my own startup Loony Con, a studio for high quality video content. In this course, you will be introduced to the Flink Architecture and Processing APIs to get you started on your data analysis a journey. First, you'll explore the differences between processing batch and streaming data on understand how the stream first architectural works. You will then explore the stream first processing model that Flink uses to process data at scale. You will also study flings architecture, which uses the job manager, task managers and task slots to execute the operators and streams in a Flink application in a data parallel manner. Next, you will understand the difference between stateless and state, full stream transformations and apply these concepts in a hands on manner in your Flink Stream processing URL. Process data in a stateless manner, using the map, flat map and filter transformations, and you'll use keyed streams and rich functions to work with feeling state. Finally, URL round off your understanding off the state persistence and fault tolerance mechanism that Flink uses by exploring the check pointing architecture in Flink URL enable checkpoints and save points in your streaming application and see how state can be restored from a snapshot in the case of failures. When you're finished with this course, you will have the skills and knowledge to design Flink pipelines, performing stateless and state ful transformations on you'll be able to build fault tolerant applications using checkpoints on save points.