- Course
Manage Data Movement with Dagster
Hardcoded file paths and connections make pipelines brittle and untestable. This course will teach you to use Dagster I/O managers and resources to separate storage from logic.
- Course
Manage Data Movement with Dagster
Hardcoded file paths and connections make pipelines brittle and untestable. This course will teach you to use Dagster I/O managers and resources to separate storage from logic.
Get started today
Access this course and other top-rated tech content with one of our business plans.
Try this course for free
Access this course and other top-rated tech content with one of our individual plans.
This course is included in the libraries shown below:
- Data
What you'll learn
When asset functions contain hardcoded file paths and database connections, your pipeline is locked to one environment and impossible to test without side effects. In this course, Manage Data Movement with Dagster, you'll gain the ability to control how and where assets read and write data. First, you'll explore I/O managers to decouple storage logic from transformation logic. Next, you'll discover how to define resources that manage database connections and external system access. Finally, you'll learn how to parameterize these components with configuration and environment variables so a single codebase runs cleanly in dev and production. When you're finished with this course, you'll have the skills and knowledge of Dagster data movement needed to build pipelines that are portable, testable, and environment-aware.