- Course
Create Complex DAGs and Task Dependencies with Apache Airflow
Learn to build complex data pipelines with Apache Airflow. This course will teach you to create DAGs that read, validate, aggregate, and load data while managing task dependencies, using XComs for passing data between tasks.
Intermediate
- Course
Create Complex DAGs and Task Dependencies with Apache Airflow
Learn to build complex data pipelines with Apache Airflow. This course will teach you to create DAGs that read, validate, aggregate, and load data while managing task dependencies, using XComs for passing data between tasks.
Intermediate
Get started today
Access this course and other top-rated tech content with one of our business plans.
Try this course for free
Access this course and other top-rated tech content with one of our individual plans.
This course is included in the libraries shown below:
- Data
What you'll learn
Apache Airflow excels in making it easy for you to manage complex data pipelines.
In this course, Create Complex DAGs and Task Dependencies with Apache Airflow, you’ll gain the ability to design and implement intricate workflows in Apache Airflow.
First, you’ll explore how to create a DAG that reads a CSV file from a local directory using a BashOperator to check if the file exists.
Next, you’ll discover how to perform data validation and aggregation using PythonOperators.
Finally, you’ll learn how to load the transformed data using a SQLiteOperator and set up task dependencies with the bitshift operator and set_upstream()/set_downstream() methods, controlling the flow of execution by passing data between tasks using XComs.
When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to create and manage complex DAGs and task dependencies efficiently.
Create Complex DAGs and Task Dependencies with Apache Airflow
Intermediate
Table of contents
A problem solver at heart, Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework.