Pass Data Between Tasks with XComs in Apache Airflow 3

Table of Contents

1. Challenge

   Push a value to XCom

   Goal: Return a value from a task and confirm Airflow stores it as an XCom.

   You are a data engineer at Globomantics. The retail ETL Dag you inherited moves data between tasks with hand-written xcom_push and xcom_pull calls. Over three steps you will return values from tasks, pull them downstream, and finish by chaining the whole pipeline with the TaskFlow API.

   How to run the Dag. In the Terminal tab, click Run at the bottom right, or type bash run_dag.sh. Either one registers your latest file and runs the whole Dag in one process. A good run ends with state=success. Run it after every edit, because Airflow only sees changes when it runs again.

   Stuck? The solutions/ folder holds the finished xcom_lab_dag.py for every task.

   Airflow 3 spells it Dag in its own documentation and release notes, so this lab does too.

2. Challenge

   Pull a value in a downstream task

   Goal: Consume the extract task's XCom value inside a second task using the classic pull API.

   A task reads another task's XCom with ti.xcom_pull(task_ids=...). By default it reads the return_value key. To read a value that was pushed under another key, pass key=.... If you pull from a task that has not run, you get None instead of an error.

   Run bash run_dag.sh after each edit. If you get stuck, the matching file in solutions/ shows the finished Dag.

3. Challenge

   Chain a full extract, transform, load sequence

   Goal: Connect three tasks end to end with automatic XCom passing, then fix a task that returns more than one value.

   The TaskFlow API removes the manual calls. Decorate a function with @task and whatever it returns becomes its XCom. Call one task with the result of another, such as transform(count), and Airflow wires the dependency for you. A decorated function only becomes a task once you call it inside the with DAG(...) block.

   A returned tuple is stored as one value. @task(multiple_outputs=True) with a returned dict stores each key as its own XCom.

   You convert one function per task, and the Dag stays runnable after each one, so run bash run_dag.sh as you go. What you built: You took an ETL Dag that moved data with hand-written xcom_push and xcom_pull calls and turned it into a chained TaskFlow pipeline. A task now stores whatever it returns, the next task receives it as an argument, and multiple_outputs=True splits a returned dict into separately addressable XCom keys.

   Moving this to a deployment. The tasks, the XCom handoffs, and the dependencies carry over unchanged. What changes is the schedule, which becomes a cron string instead of None, and the file lives in the deployment's Dag bundle where the scheduler parses it.

   Where to go next

   • Return a file path instead of a large value. XCom is meant for small handoffs.
   • Replace the single transform with dynamic task mapping so each city becomes its own task instance.
   • Give the Dag a daily schedule and watch Airflow create a run for every day since start_date.