-
Course
- Data
Data Science & Hadoop Workflows at Scale With Scalding
Learn how to use Scalding and Algebird and join Twitter, Etsy, eBay, and others to efficiently extract value and process data at scale on Hadoop.
What you'll learn
This course teaches you how to use Scalding (a domain specific language) built on Scala and Cascading to build distributed applications on Hadoop. The course also focuses on the data science aspect using Algebird, an abstract algebra library for Scala, to solve real-world sketching/streaming problems on distributed systems. You will learn how to reason about a variety of problems, how to build and test locally, and how to deploy on Hadoop. You will also learn the algorithms used to solve problems at scale where performance, compute and memory resources, and the window of time you have to process streaming data are all challenges you'll have to overcome, and how you can use Scalding and Algebird to solve for these constraints. This course also covers some Scala basics to get you up to speed and looks into how you can monitor, visualize, and troubleshoot your application's workflow and performance problems. Watch this course if you were considering, or already know how to use Pig, Hive, or any other DSL for Hadoop and not only wanted more power over your workflows, but also a DSL that is actively being developed to support up and coming execution frameworks like Apache Tez and Apache Spark with all the flexibility that a full functional programming language like Scala has to offer. If you're serious about learning how to build enterprise-grade applications on Hadoop, data science, and Lambda architectures, then this course is for you.
Table of contents
- Introduction | 37s
- Why Scalding & Course Preview | 4m 1s
- Course Outline & Focus | 1m 25s
- Technologies Covered | 2m 27s
- Cascading | 1m 17s
- Scalding vs. Pig and Hive | 2m 8s
- Scalding APIs | 40s
- Scala (map) | 4m 21s
- Scalding (map) | 2m 7s
- Distributed Applications | 2m 10s
- Simple Pipeline | 4m 50s
- Pipe Transformations | 4m 39s
- Passing Arguments | 35s
- Scalding Sources | 1m 23s
- Installing the Scala IDE | 1m 46s
- Configuring Eclipse & Maven | 1m 43s
- Summary | 1m
- Introduction | 1m 9s
- Scalding REPL | 1m 35s
- Eclipse Scala Worksheet With Scalding | 5m 13s
- Word Count | 2m 15s
- FlatMap | 3m 36s
- Union | 2m 38s
- RITA Dataset - Exploring Data | 2m 2s
- Group Operations | 1m 24s
- Join Operations | 2m 9s
- Skew Joins | 1m 22s
- Joining Pipes | 2m 26s
- Reduction Operations | 1m 38s
- Fold and FoldLeft | 1m 52s
- FoldLeft in Practice | 6m 59s
- Summary | 1m 22s
About the author
Ahmad is a Data Architect specializing in the implementation of high-performance data warehouses and BI systems and enjoys speaking at various user groups and conferences.
More Courses by Ahmad