Deploying Data Pipelines in Microsoft Azure
In this course, you will learn foundational knowledge needed to apply CI/CD methodologies to your pipeline creation process in Azure Data Factory to deploy robust and well-tested data pipelines to production.
What you'll learn
Data engineers working with Azure Data Factory can take advantage of Continuous Integration and Continuous Delivery practices to deploy robust and well-tested data pipelines to production.
In this course, Deploying Data Pipelines in Microsoft Azure, you will learn foundational knowledge to apply CI/CD methodologies to your data pipeline creation process.
First, you will learn to create the right environments to fall into the pit of success when creating data pipelines in ADF.
Next, you will discover how to deploy data pipelines using ADF visual tools and ARM templates.
Finally, you will explore how to create a release pipeline in Azure DevOps to automate the deployment process between three distinct environments: development, staging, and production.
When you are finished with this course, you will have the skills and knowledge to apply CI/CD practices to your data pipeline creation process, effortlessly.
Table of contents
- Introduction 1m
- Understanding Azure Data Factory Continuous Ingratiation Life Cycle 3m
- Creating a Feature Branch and Making Changes to the Pipeline 4m
- Creating a Pull Request and Publishing Changes 4m
- Exporting ARM Templates from the Collaboration Pipeline 2m
- Deploying to the Staging Environment 4m
- Deploying to the Production Environment 2m
- Summary 1m
- Introduction 1m
- Getting Started with Azure Data Factory and CI/CD Using Azure DevOps 1m
- Creating Variable Groups in Azure DevOps 1m
- Creating a Release Pipeline in Azure DevOps 3m
- Selecting an Artifact 1m
- Adding Tasks to the Pipeline Stage 5m
- Adding Pre and Post Deployment PowerShell Scripts 5m
- Cloning Stages, Linking Variable Groups, and Configuring Permissions 4m
- Enabling Continuous Deployment Trigger and Pre-deployment Conditions 1m
- Implementing a New Feature in the Development Environment 3m
- Deploying to the Staging Environment 2m
- Deploying to the Production Environment 2m
- Summary 0m
About the author
Marcelo has accumulated over two decades of expertise, adeptly aligning business goals with innovative technological solutions. Initiating his coding journey in the early 90s on an IBM 8086 XT, he became proficient in languages such as Clipper, Pascal and C++. By his mid-teens, Marcelo had already crafted and commercialized software for the DOS operating system. In the current tech landscape, his specialization encompasses technologies like .NET, C#, JavaScript, NextJS, SvelteKit, and cloud-base... more