Developing Spark Applications Using Scala & Cloudera

By Xavier Morera
Apache Spark is one of the fastest and most efficient general engines for large-scale data processing. In this course, you'll learn how to develop Spark applications for your Big Data using Scala and a stable Hadoop distribution, Cloudera CDH.
Course info
Level
Beginner
Updated
May 3, 2018
Duration
5h 43m
Table of contents
Why Spark with Scala and Cloudera?
But Why Apache Spark? 2m Brief History of Spark 3m What We Will Cover in This Training 2m Picking a Spark Supported Language: Scala, Python, Java, or R 1m What Do You Need for This Course? 2m Takeaway 1m Why Spark with Scala and Cloudera? 1m
Getting an Environment and Data: CDH + StackOverflow
Getting an Environment & Data: CDH + StackOverflow 2m Prerequisites & Known Issues 2m Upgrading Cloudera Manager and CDH 6m Installing or Upgrading to Java 8 (JDK 1.8) 4m Getting Spark - There Are Several Options: 1.6 3m Getting Spark 2 Standalone 3m Installing Spark 2 on Cloudera 6m Data: StackOverflow & StackExchange Dumps + Demo Files 3m Preparing Your Big Data 4m Takeaway 2m
Refreshing Your Knowledge: Scala Fundamentals for This Course
Refreshing Your Knowledge: Scala Fundamentals for This Course 1m Scala's History and Overview 2m Building and Running Scala Applications 1m Creating Self-contained Applications, Including scalac & sbt 5m The Scala Shell: REPL (Read Evaluate Print Loop) 1m Scala, the Language 4m More on Types, Functions, and Operations 2m Expressions, Functions, and Methods 1m Classes, Case Classes, and Traits 1m Flow Control 1m Functional Programming 1m Enter spark2-shell: Spark in the Scala Shell 1m Takeaway 2m
Understanding Spark: An Overview
Understanding Spark: An Overview 3m Spark, Word Count, Operations, and Transformations 2m A Few Words on Fine Grained Transformations and Scalability 2m Word Count in "Not Big Data" 2m How Word Count Works, Featuring Coarse Grained Transformations 4m Parallelism by Partitioning Data 3m Pipelining: One of the Secrets of Spark's Performance 2m Narrow and Wide Transformations 4m Lazy Execution, Lineage, Directed Acyclic Graph (DAG), and Fault Tolerance 4m Time for the Big Picture: Spark Libraries 2m Takeaway 1m
Getting Technical with Spark
Getting Technical: Spark Architecture 3m Storage in Spark and Supported Data Formats 3m Let's Talk APIs: Low Level and High Level Spark APIs 5m Performance Optimizations: Tungsten and Catalyst 3m SparkContext and SparkSession: Entry Points to Spark Apps 4m Spark Configuration + Client and Cluster Deployment Modes 6m Spark on Yarn: The Cluster Manager 3m Spark with Cloudera Manager and YARN UI 4m Visualizing Your Spark App: Web UI and History Server 8m Logging in with Spark and Cloudera 2m Navigating the Spark and Cloudera Documentation 4m Takeaway 1m
Learning the Core of Spark: RDDs
Learning the Core of Spark: RDDs 2m SparkContext: The Entry Point to a Spark Application 4m Returning Data to the Driver, i.e. collect(), take(), first()... 4m Takeaway 2m RDD and PairRDD - Resilient Distributed Datasets 4m Creating RDDs with Parallelize 4m Partitions, Repartition, Coalesce, Saving as Text, and HUE 3m Creating RDDs from External Datasets 10m Saving Data as ObjectFile, NewAPIHadoopFile, SequenceFile, ... 6m A Little Bit More on Lineage and Dependencies 1m Creating RDDs with Transformations 3m
Going Deeper into Spark Core
Going Deeper into Spark Core 1m A Quick Look at Map, FlatMap, Filter, and Sort 5m Set Operations: Join, Union, Full Right, Left Outer, and Cartesian 5m Combining, Aggregating, Reducing, and Grouping on PairRDDs 9m ReduceByKey vs. GroupByKey: Which One Is Better? 1m Caching and Data Persistence 2m Shared Variables: Accumulators and Broadcast 5m What's Needed for Developing Self-contained Spark Applications 2m Disadvantages of RDDs - So What's Better? 1m Takeaway 2m Functional Programming: Anonymous Functions (Lambda) in Spark 2m How Can I Tell It Is a Transformation 1m Why Do We Need Actions? 1m Partition Operations: MapPartitions and PartitionBy 6m Sampling Your Data 2m Grouping Data into Buckets with Histogram 3m
Increasing Proficiency with Spark: DataFrames and Spark SQL
Increasing Proficiency with Spark: DataFrames & Spark SQL 1m "Everyone" Uses SQL and How It All Began 3m Hello DataFrames and Spark SQL 3m DataFrames to RDDs and Vice Versa 3m Loading DataFrames: Text and CSV 2m More Data Loading: Parquet and JSON 4m Rows, Columns, Expressions, and Operators 2m Working with Columns 2m Takeaway 2m SparkSession: The Entry Point to the Spark SQL / DataFrame API 2m Creating DataFrames 2m Schemas: Inferred and Programatically Specified + Option 5m More Columns, Expressions, Cloning, Renaming, Casting, & Dropping 4m User Defined Functions (UDFs) on Spark SQL 3m
Continuing the Journey on DataFrames and Spark SQL
Querying, Sorting, and Filtering DataFrames: The DSL 5m What to Do with Missing or Corrupt Data 4m Saving DataFrames 5m Spark SQL: Querying Using Temporary Views 4m Loading Files and Views into DataFrames Using Spark SQL 2m Saving to Persistent Tables + Spark 2 Known Issue 2m Hive Support and External Databases 5m Aggregating, Grouping, and Joining 5m The Catalog API 1m Takeaway 2m
Working with a Typed API: Datasets
Understanding a Typed API: Datasets 1m The Motivation Behind Datasets 5m What's a Dataset? 3m What Do You Need for Datasets? 1m Creating Datasets 3m Dataset Operations 3m RDDs vs. DataFrames vs. Datasets: A Few Final Thoughts 1m Takeaway 2m
Final Takeaway and Continuing the Journey with Spark
Final Takeaway 6m Continuing the Journey with Spark, Scala, and Cloudera 5m
Course Overview
Course Overview 2m
Description
Course info
Level
Beginner
Updated
May 3, 2018
Duration
5h 43m
Description

At the core of working with large-scale datasets is a thorough knowledge of Big Data platforms like Apache Spark and Hadoop. In this course, Developing Spark Applications Using Scala & Cloudera, you’ll learn how to process data at scales you previously thought were out of your reach. First, you’ll learn all the technical details of how Spark works. Next, you’ll explore the RDD API, the original core abstraction of Spark. Then, you’ll discover how to become more proficient using Spark SQL and DataFrames. Finally, you'll learn to work with Spark's typed API: Datasets. When you’re finished with this course, you’ll have a foundational knowledge of Apache Spark with Scala and Cloudera that will help you as you move forward to develop large-scale data applications that enable you to work with Big Data in an efficient and performant way.

About the author
Xavier Morera
About the author

Xavier is very passionate about teaching, helping others understand search and Big Data. He is also an entrepreneur, project manager, technical author, trainer, and holds a few certifications with Cloudera, Microsoft, and the Scrum Alliance, along with being a Microsoft MVP.

More from the author
Programming Python Using an IDE
Intermediate
2h 0m
Jun 26, 2019
Understanding Databases with SQLAlchemy: Python Data Playbook
Beginner
1h 23m
Mar 16, 2019
Microsoft Azure Developer: Implementing Data Lake Storage Gen2
Intermediate
1h 9m
Dec 11, 2018
More courses by Xavier Morera
Section Introduction Transcripts
Section Introduction Transcripts

Course Overview
Hello and welcome to this Pluralsight course, Developing Spark Applications Using Scala and Cloudera. I am Xavier Morera and I help developers understand enterprise search and big data. Did you know that Spark as a big data processing engine is at least 10 to 100 times faster than Hadoop MapReduce? And on top of that it is easier to learn, widely adopted, and used for a diverse range of applications? In this course we're going to learn how to create Spark applications in the very language in which Spark was created; Scala, a language in which most Spark samples are available. Also, because infrastructure is important, we will leverage the first and one of the most widely used Hadoop distributions, CDH, which stands for Cloudera's Distribution including Hadoop. Some of the major topics that we will cover include getting an environment set up with Spark and some interesting data, namely CDH plus StackOverflow, understanding Spark and overview, and getting technical with Spark. We will also learn how to work with RDDs or resilient distributed datasets, the original core abstraction of Spark, and then we will cover DataFrames and Spark SQL, which helps us become proficient with Spark quicker, and finally we will learn about datasets, the higher-level typed API in Spark. By the end of this course you will be able to create Spark applications using Scala and Cloudera. Before beginning the course, you should be familiar with programming, preferably with Scala, but I include a refresher module to jumpstart your journey. Additionally, you will need a cluster, but I will explain how to get your infrastructure set up in multiple different ways. I hope you will join me on this journey to learn about Spark with the Developing Spark Applications with Scala and Cloudera course at Pluralsight.

You're the smartest person in the room

PROVE IT

Your team won't just close the skills gap

THEY'LL HURDLE IT