Handling Streaming Data with AWS Kinesis Data Analytics Using Java
Kinesis Data Analytics is a service to transform and analyze streaming data with Apache Flink and SQL using serverless technologies. You'll learn to use the Amazon Kinesis Data Analytics service to process streaming data using Apache Flink runtime.
What you'll learn
Kinesis Data Analytics is part of the Kinesis streaming platform along with Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Video streams.
In this course, Handling Streaming Data with AWS Kinesis Data Analytics Using Java, you'll work with live Twitter feeds to process real-time streaming data. First, you'll create a developer account on the Twitter platform and generate authentication keys and tokens to access the Twitter streaming API. You'll then write code to access these tweets as streaming messages and publish them to Kinesis Data Streams which can be used as a source of streaming data in Kinesis Data Analytics.
Next, you'll run Kinesis Data Analytics applications using the Apache Flink runtime to process tweets. You'll deploy these applications using the web console as well as the command line. You'll set up the right permissions, and configure these applications to use cloud monitoring and logging, and see how you can use log messages to debug errors in your applications.
Finally, you'll perform a number of different processing operations on streaming tweets, windowing operations using tumbling and sliding windows. You'll apply global windows with count triggers, and continuous-time triggers. You'll implement join operations and create branching pipelines to sink some results to DynamoDB and other results to S3.
When you're finished with this course, you'll have the skills and knowledge to create and deploy streaming applications that process live streams such as Twitter messages.
Table of contents
- Version Check 0m
- Prerequisites and Course Outline 3m
- Kinesis Data Analytics and Connectors 4m
- Demo: Setting up the Local Environment 6m
- Demo: Setting up an Apache Maven Project 4m
- Demo: Setting up a Twitter Developer Account and Generating Keys 5m
- Demo: Publishing Twitter Messages to Kinesis Data Streams 5m
- Demo: Processing Tweets Using Apache Flink 4m
- Demo: Creating a Kinesis Data Analytics Application 4m
- Demo: Configuring Policies for the Kinesis Data Analytics Application 3m
- Demo: Running the Kinesis Data Analytics Application 4m
- Demo: Updating Starting and Stopping Application Using the CLI 5m
- Demo: Sinking Processed Results to a Kinesis Data Stream 5m
- Demo: Configuring Application Runtime Properties 4m
- Demo: Configuring a Kinesis Firehose Delivery Stream to Deliver Results to S3 6m
- Cloud Watch for Monitoring and Logging 2m
- Demo: Viewing Logs for a Kinesis Data Analytics Application 5m
- Demo: Querying Logs Using Logs Insights 3m
- Demo: Viewing Metrics Using Cloud Watch 3m
- Demo: Extract Hashtags, Screen Names, and Followers Count from Tweets 4m
- Demo: Configure the Log Group, Log Stream, and Permissions for a Kinesis Data Analytics Application 6m
- Demo: Configure Permissions to Allow the CLI to Associate a Role with an Application 2m
- Demo: Deploying an Application Using the CLI 6m
- Demo: Debugging Errors in Code Using Cloud Watch Logs 5m
- Demo: Viewing Custom Metrics in Cloud Watch 8m
- Demo: Extracting Event Time from Streaming Entities 8m
- Demo: Tumbling Windows Using Event Time 4m
- Demo: Allowing out of Order and Late Data 4m
- Demo: Sliding Windows with Allowed Lateness 4m
- Demo: Session Windows Using Event Time 4m
- Demo: Tumbling Processing Time Windows 5m
- Demo: Global Windows and Count Triggers 4m
- Demo: Global Windows and Continuous Event Time Triggers 4m
- Demo: Implementing a Branching Pipeline 7m
- Demo: Create an AWS Lambda to Write Processed Results to Dynamo DB 6m
- Demo: Deploying and Running a Branching Pipeline and Viewing Results 5m
- Demo: Implementing Joins on Data Streams 8m
- Demo: Perform Joining Operations Using Kinesis Data Analytics 4m
- Summary and Further Study 2m