What you'll learn

Apache Airflow provides built-in operators to connect to external cloud services. In this course, Integrate External Services with Apache Airflow, you’ll gain the ability to connect and manage various external services within your Airflow workflows. You'll connect your Airflow workflow to Google Cloud Storage and Cloud BigQuery as well as to AWS S3, enabling data management and processing across cloud platforms. First, you’ll explore how to set up permissions on Google Cloud using a service account to enable Airflow to access Google Cloud Storage buckets and BigQuery datasets. Next, you'll discover how to read files from your local machine and upload to Google Cloud, create a BigQuery dataset, and create an external table in BigQuery. Then, you'll integrate your workflow with AWS S3 buckets to transfer data from S3 to Google Cloud Storage. Finally, you’ll use hooks to work with S3, Cloud Storage, and BigQuery - hooks give you much more control over how you access your data from external services. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to integrate and manage external services effectively within your data workflows.