Building Pipelines for Workflow Orchestration Using Google Composer
Cloud Composer is a fully managed workflow orchestration service that allows creation, scheduling, and monitoring of pipelines. This course covers the architecture of Composer and creation of pipelines to run arbitrary shell scripts and Python code.
What you'll learn
Cloud Composer is a pipeline orchestration service on the GCP. Based on the Apache Airflow API, Composer was launched in May 2018 and is fast-emerging as a popular and versatile service for building and executing system pipelines. In this course, Building Pipelines for Workflow Orchestration Using Google Composer, you'll learn how Composer allows cloud users to quickly create pipelines with complex interconnected tasks. First, you'll discover where Composer fits in the taxonomy of GCP services and how it compares to Dataflow, which is another service for building and executing pipelines on the GCP. Next, you'll explore what a Composer environment is and how pipelines are specified, and run on these environments. Then, you'll develop an understanding of the powerful suite of operators made available for use within Composer pipelines by utilizing Airflow operators for executing shell scripts, executing arbitrary Python code, and implementing complex control flow. Finally, you'll learn how to use Airflow’s GCP-specific operators for sending email, working with BigQuery, and instantiating Dataproc clusters. When you're finished with this course, you'll have the skills and knowledge necessary to build and deploy complex pipelines built on the Apache Airflow API by utilizing Composer.
Table of contents
- Module Overview 2m
- Write an Apache Airflow DAG 5m
- Run a DAG on Cloud Composer 6m
- Explore Composer Web Server 4m
- Use the All_success Trigger Rule 7m
- Use the One_failed Trigger Rule 4m
- Use the All_done Trigger Rule 4m
- Specify Python Dependencies 5m
- Create a Dummy Operator 3m
- The BranchPythonOperator 7m
- Summary 1m
- Module Overview 1m
- Use Airflow Variables to Copy Files from One GCS Bucket to Another 7m
- Delete DAGs and Composer Environment 3m
- Use the Send Grid Email Integration in Airflow 6m
- Complex Pipeline Integrating BigQuery, Cloud Storage, and Send Grid 4m
- Writing DAG to Work with Big Query, GCS, and Send Grid 5m
- Use Cloud Dataproc Operators in Composer 6m
- Summary 2m
About the author
Vitthal has spent a lot of his life studying - he holds Masters Degrees in Math and Electrical Engineering from Stanford, an MBA from INSEAD, and a Bachelors Degree in Computer Engineering from Mumbai. He has also spent a lot of his life working - as a derivatives quant at Credit Suisse in New York, then as a quant trader, first with a hedge fund in Greenwich and then on his own, and finally at Google in Singapore and Flipkart in Bangalore. In all these roles, he has written a lot of code, and b... more