-
Course
- Cloud
- Data
Designing Resilient AWS Data Pipelines
Pipelines often fail due to schema drift or retries gone wrong. This course will teach you how to design resilient AWS data pipelines that adapt to schema changes, support safe reprocessing, and recover from operational failures.
What you'll learn
Many AWS data pipelines fail to operate reliably as data models evolve or errors occur midstream. In this course, Designing Resilient AWS Data Pipelines, you’ll learn to build pipelines that can handle schema changes, reprocessing, and validation with confidence. First, you’ll explore how to partition data in S3 and define schema models that support analytics and data lifecycle management. Next, you’ll discover how to apply schema versioning and configure retry behavior and logging in pipeline components like Lambda and Glue. Finally, you’ll learn how to build idempotent processing logic and validate pipeline output using Athena queries. When you’re finished with this course, you’ll have the skills and knowledge of designing resilient AWS data pipelines needed to handle schema changes, ensure safe reprocessing, and build fault-tolerant systems.
Table of contents
About the author
Rupesh is an independent consultant with over 12 years of experience in software development. As a software architect Rupesh creates web applications for the various domains industries using JavaScript, Node, Angular, Typescript, C#, and .Net.
More Courses by Rupesh