Building Pipelines for Workflow Orchestration Using Google Composer

Cloud Composer is a fully managed workflow orchestration service that allows creation, scheduling, and monitoring of pipelines. This course covers the architecture of Composer and creation of pipelines to run arbitrary shell scripts and Python code.
Course info
Level
Intermediate
Updated
Jan 15, 2019
Duration
1h 59m
Table of contents
Description
Course info
Level
Intermediate
Updated
Jan 15, 2019
Duration
1h 59m
Description

Cloud Composer is a pipeline orchestration service on the GCP. Based on the Apache Airflow API, Composer was launched in May 2018 and is fast-emerging as a popular and versatile service for building and executing system pipelines. In this course, Building Pipelines for Workflow Orchestration Using Google Composer, you'll learn how Composer allows cloud users to quickly create pipelines with complex interconnected tasks. First, you'll discover where Composer fits in the taxonomy of GCP services and how it compares to Dataflow, which is another service for building and executing pipelines on the GCP. Next, you'll explore what a Composer environment is and how pipelines are specified, and run on these environments. Then, you'll develop an understanding of the powerful suite of operators made available for use within Composer pipelines by utilizing Airflow operators for executing shell scripts, executing arbitrary Python code, and implementing complex control flow. Finally, you'll learn how to use Airflow’s GCP-specific operators for sending email, working with BigQuery, and instantiating Dataproc clusters. When you're finished with this course, you'll have the skills and knowledge necessary to build and deploy complex pipelines built on the Apache Airflow API by utilizing Composer.

About the author
About the author

An engineer and tinkerer, Vitthal has worked at Google, Credit Suisse, and Flipkart and studied at Stanford and INSEAD. He has worn many hats, each of which has involved writing code and building models. He is passionately devoted to his hobby of laughing at his own jokes.

More from the author
Leveraging Load Balancing Options on the GCP
Intermediate
2h 24m
Jan 17, 2019
More courses by Vitthal Srinivasan
Section Introduction Transcripts
Section Introduction Transcripts

Course Overview
Hi, my name is Vitthal Srinivasan, and I'd like to welcome you to this course on Building Pipelines for Workflow Orchestration Using Google Composer. First, a little bit about myself. I have master's degrees in financial math and electrical engineering from Stanford University and have previously worked in companies such as Google in Singapore and Credit Suisse in New York. I am now co-founder at Loonycorn, a studio for high quality video content based in Bangalore, India. Cloud Composer is a pipeline orchestration service on the GCP. Based on the Apache Airflow API, Composer was launched in mid-2018 and is fast emerging as a popular and versatile service for building and executing system pipelines. In this course, you will learn how Composer allows cloud users to quickly create pipelines with complex interconnected tasks. First, you will learn where Composer fits into the taxonomy of GCP services and how it compares to Dataflow, which is another service for building and executing pipelines. You will see how Composer focuses on pipelines of system tasks while Dataflow is all about building data and streaming pipelines. Next, you will explore what a Composer environment is and how pipelines are specified and run on these environments. The Composer service runs an entire implementation of Apache Airflow, including a Kubernetes cluster and several other infrastructure components. Finally, you will understand the powerful suite of operators made available for use within Composer pipelines. Pipelines can be thought of as directed acyclic graphs. These graphs can be used to monitor dependencies between steps in the pipeline. Each individual step is a task instance equivalent to an instance of an operator. You will use powerful Airflow operators for executing shell scripts, arbitrary Python code, implementing conditioners, branching, and complex control flow. When this course is complete, you will be very comfortable choosing Composer as the glue in your architecture and will have the skills and the knowledge to build and deploy complex pipelines built on the Apache Airflow API.