- Lab
-
Libraries: If you want this lab, consider one of these libraries.
- Cloud
- Data
Streaming to BigQuery with Dataflow Templates and DLQs
In this hands-on lab, you will create Pub/Sub resources to ingest e-commerce order events, deploy a Google-provided Dataflow template to stream records into BigQuery, and validate pipeline correctness by publishing both valid and malformed messages. You will inspect the dead letter table to confirm that failed records are captured with error details for debugging.
Lab Info
Table of Contents
-
Challenge
Create Pub/Sub resources for streaming ingestion
Create a Pub/Sub topic for order events and a pull subscription that the Dataflow pipeline uses to consume messages.
-
Challenge
Deploy a Dataflow template to stream data into BigQuery
Deploy a Google-provided Dataflow template that reads from the Pub/Sub subscription, writes valid records to the orders table, and automatically routes failed records to a dead letter table.
-
Challenge
Validate pipeline correctness and dead letter routing
Publish valid and malformed messages to the Pub/Sub topic, then query BigQuery to confirm valid records land in the orders table and failed records are captured in the orders_error_records dead letter table with error details.
Real skill practice before real-world application
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.
Learn by doing
Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.
Follow your guide
All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.
Turn time into mastery
On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.