Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Modern Data Engineering

Course Summary

The Modern Data Engineering training course presents data engineering concepts for teams who are moving to cloud-based services from legacy systems and want to take advantage of the increased security and availability that the cloud offers. This course will introduce students to the principles of working in the cloud, data warehousing best practices, and cloud-based tools for data engineering.

The course begins with a discussion of the fundamentals of data engineering in the cloud, including strategies for data collection, data storage, and structuring your data. The course then focuses on how to move and transform data in the cloud. The course concludes with data integration concepts and how to visually present data in the cloud.

While this course will focus primarily on the use of AWS to create and maintain data engineering and transformation infrastructure, other cloud providers can be substituted to allow for customization to your organization's specific needs.

Purpose
 Learn the basic concepts of data engineering and how to make the shift to the cloud.
Audience
 Software Engineers, Data Scientists and Data Engineers who want to learn about data engineering and building data pipelines in the cloud.
Role
Data Engineer - Data Scientist - Software Developer
Skill Level
Introduction
Style
Workshops
Duration
3 Days
Related Technologies
SQL | Cloud Computing Training | Machine Learning Training | Data Science

 

Productivity Objectives
  • Explain different cloud-based services used to build and operate data pipelines that can enable both ETL and ELT workflows.
  • Define strategies for collecting, storing, transforming, and visualizing data in the cloud.
  • Utilize AWS services to create and maintain data pipelines.
  • Understand the role of business intelligence tools.

What You'll Learn:

In the Modern Data Engineering training course, you'll learn:
  • Collecting Data From Multiple Sources
    • Overview of what ETL and ELT are and why they are important
    • Using APIs to get data
    • Using S/FTP to get data
    • Manually uploading data sources
    • Extracting data from SQL databases
  • Storing Data in the Cloud
    • Moving data to cloud storage services (S3/Buckets)
    • Using Amazon Redshift to store relational data in a data warehouse
    • NoSQL storage via DynamoDB
    • Elasticache/caching storage
    • File storage types (CSV, TSV, JSON, Avro, Parquet, Excel, etc)
  • Collecting and Structuring Data
    • File system recommendations
    • Data lake vs Data Warehouse
    • Distributed vs non-distributed
    • Data structures and filesystems
  • Extraction of Data
    • Overview of data extractions and their importance in overall data architecture
    • Moving from the data lake to structure in various tools (Dynamo, Spark, SQL, etc.)
    • Provisioning SQL databases around data patterns
    • Queueing services (AWS SQS, AWS Kinesis)
    • Creating listeners, pollers, and triggers in an extraction architecture
    • Lambdas, Terraform, and ethereal infrastructure
    • Creating a data extraction integration within an AWS VPC
  • Data Wrangling
    • Overview of data transformations and their importance in overall data architecture
    • Data transformations use-cases:
      • Transforming data types from XML to JSON
      • Executing queries to transform data for BI analysis
      • Ensuring robust copies of all data inflow
    • In-stream transformations in Dynamo
    • Uses of lambda for transforming/restructuring data
    • Kinesis in-stream transformations
    • Creating an integration with an endpoint
  • Data Analysis
    • Using AWS Quicksight to understand your data in the cloud
    • A brief introduction to Tableau, Looker, Metabase and the ecosystem of modern BI tools
    • What is data science and where does it fit with data engineering?
real world content

Real-World Content

Project-focused demos and labs using your tool stack and environment, not some canned "training room" lab.
expert practitioners

Expert Practitioners

Industry experts that bring their battle scars into the classroom.
experiential learning

Experiential Learning

More coding than lecture, coupled with architectural and design discussions.
tailored outlines

Tailored Outlines

One-size-fits-all doesn't apply to training teams. That's where we come in!
“I appreciated the instructor's technique of writing live code examples rather than using fixed slide decks to present the material.”

VMware

Dive in and learn more

When transforming your workforce, it's important to have expert advice and tailored solutions. We can help. Tell us your unique needs and we'll explore ways to address them.

Let's chat

By filling out this form and clicking submit, you acknowledge our privacy policy.