Developing Spark Applications with Python & Cloudera

By Xavier Morera
Apache Spark is one of the fastest and most efficient general engines for large-scale data processing. In this course, you will learn how to develop Spark applications for your Big Data using Python and a stable Hadoop distribution, Cloudera CDH.
Course info
Level
Beginner
Updated
Feb 27, 2018
Duration
5h 42m
Table of contents
Why Spark with Python and Cloudera?
Why Spark with Python and Cloudera? 1m Brief History of Spark 3m What We Will Cover in This Training 2m Picking a Spark Supported Language: Python, Scala, Java, or R 1m What Do You Need for This Course? 2m Takeaway 1m But Why Apache Spark? 2m
Getting an Environment & Data: CDH + StackOverflow
Getting an Environment and Data: CDH + StackOverflow 2m Prerequisites and Known Issues 2m Upgrading Cloudera Manager and CDH 6m Installing or Upgrading to Java 8 (JDK 1.8) 4m Getting Spark - There Are Several Options: 1.6 3m Getting Spark 2 Standalone 3m Installing Spark 2 on Cloudera 6m Bonus -> IPython with Anaconda: Supercharge Your PySpark Shell 7m Data: StackOverflow and StackExchange Dumps + Demo Files 3m Preparing Your Big Data 4m Takeaway 2m
Refreshing Your Knowledge: Python Fundamentals for This Course
Refreshing Your Knowledge: Python Fundamentals for This Course 1m Python's History, Philosophy, and Paradigm 3m The Python Shell: REPL 3m Syntax, Variables, (Dynamic) Types, and Operators 7m Compound Variables: Lists, Tuples, and Dictionaries 5m Code Blocks, Functions, Loops, Generators, and Flow Control 5m Map, Filter, Group, and Reduce 2m Enter PySpark: Spark in the Shell 2m Takeaway 2m
Understanding Spark: An Overview
Understanding Spark: An Overview 3m Spark, Word Count, Operations, and Transformations 2m A Few Words on Fine Grained Transformations and Scalability 2m Word Count in "Not Big Data" 2m How Word Count Works, Featuring Coarse Grained Transformations 4m Parallelism by Partitioning Data 3m Pipelining: One of the Secrets of Spark's Performance 2m Narrow and Wide Transformations 4m Lazy Execution, Lineage, Directed Acyclic Graph (DAG), and Fault Tolerance 4m The Spark Libraries and Spark Packages 2m Takeaway 1m
Learning the Core of Spark: RDDs
Learning the Core of Spark: RDDs 2m SparkContext: The Entry Point to a Spark Application 3m RDD and PairRDD - Resilient Distributed Datasets 4m Creating RDDs with Parallelize 4m Returning Data to the Driver, i.e. collect(), take(), first()... 4m Partitions, Repartition, Coalesce, Saving as Text, and HUE 3m Creating RDDs from External Datasets 10m Saving Data as PickleFile, NewAPIHadoopFile, SequenceFile, ... 5m Creating RDDs with Transformations 3m A Little Bit More on Lineage and Dependencies 1m Takeaway 2m
Increasing Proficiency with Spark: DataFrames & Spark SQL
Increasing Proficiency with Spark: DataFrames & Spark SQL 1m "Everyone" Uses SQL and How It All Began 3m Hello DataFrames and Spark SQL 3m SparkSession: The Entry Point to the Spark SQL and DataFrame API 2m Creating DataFrames 3m DataFrames to RDDs and Viceversa 3m Loading DataFrames: Text and CSV 2m Schemas: Inferred and Programatically Specified + Option 5m More Data Loading: Parquet and JSON 4m Rows, Columns, Expressions, and Operators 2m Working with Columns 2m More Columns, Expressions, Cloning, Renaming, Casting, & Dropping 4m User Defined Functions (UDFs) on Spark SQL 3m Takeaway 2m
Understanding a Typed API: Datasets Works with Scala, Not Python
Understanding a Typed API: Datasets; Works with Scala, Not Python 1m Got Scala? 1m A Quick Look at Datasets 3m Takeaway 1m
Final Takeaway and Continuing the Journey with Spark
Final Takeaway 6m Continuing the Journey with Spark 5m
Getting Technical with Spark
Getting Technical: Spark Architecture 3m Storage in Spark and Supported Data Formats 3m Let's Talk APIs: Low-level and High-level Spark APIs 4m Performance Optimizations: Tungsten and Catalyst 3m SparkContext and SparkSession: Entry Points to Spark Apps 4m Spark Configuration + Client and Cluster Deployment Modes 6m Spark on Yarn: The Cluster Manager 3m Spark with Cloudera Manager and YARN UI 4m Visualizing Your Spark App: Web UI and History Server 8m Logging in Spark and with Cloudera 6m Navigating the Spark and Cloudera Documentation 4m Takeaway 1m
Going Deeper into Spark Core
Takeaway 2m Disadvantages of RDDs - So What's Better? 1m Developing Self-contained PySpark Application, Packages, and Files 1m Shared Variables: Accumulators and Broadcast Variables 5m Grouping Data into Buckets with Histogram 3m Caching and Data Persistence 2m Why Do We Need Actions? 1m Partition Operations: MapPartitions and PartitionBy 7m Sampling Your Data 2m Set Operations: Join, Union, Full Right, Left Outer, and Cartesian 5m ReduceByKey vs. GroupByKey: Which One Is Better? 1m Combining, Aggregating, Reducing, and Grouping on PairRDDs 8m Going Deeper into Spark Core 1m Functional Programming: Anonymous Functions (Lambda) in Spark 1m A Quick Look at Map, FlatMap, Filter, and Sort 5m How I Can Tell It Is a Transformation 1m
Continuing the Journey on DataFrames and Spark SQL
What to Do with Missing or Corrupt Data 4m Saving DataFrames 6m Spark SQL: Querying Using Temporary Views 4m Loading Files and Views into DataFrames Using Spark SQL 2m Saving to Persistent Tables + Spark 2 Known Issue 2m Hive Support and External Databases 5m Aggregating, Grouping, and Joining 5m The Catalog API 1m Takeaway 2m Querying, Sorting, and Filtering DataFrames: The DSL 5m
Course Overview
Course Overview 2m
Description
Course info
Level
Beginner
Updated
Feb 27, 2018
Duration
5h 42m
Description

At the core of working with large-scale datasets is a thorough knowledge of Big Data platforms like Apache Spark and Hadoop. In this course, Developing Spark Applications with Python & Cloudera, you’ll learn how to process data at scales you previously thought were out of your reach. First, you’ll learn all the technical details of how Spark works. Next, you’ll explore the RDD API, the original core abstraction of Spark. Finally, you’ll discover how to become more proficient using Spark SQL and DataFrames. When you’re finished with this course, you’ll have a foundational knowledge of Apache Spark with Python and Cloudera that will help you as you move forward to develop large-scale data applications that enable you to work with Big Data in an efficient and performant way.

About the author
Xavier Morera
About the author

Xavier is very passionate about teaching, helping others understand search and Big Data. He is also an entrepreneur, project manager, technical author, trainer, and holds a few certifications with Cloudera, Microsoft, and the Scrum Alliance, along with being a Microsoft MVP.

More from the author
Programming Python Using an IDE
Intermediate
2h 0m
Jun 26, 2019
Understanding Databases with SQLAlchemy: Python Data Playbook
Beginner
1h 23m
Mar 16, 2019
Microsoft Azure Developer: Implementing Data Lake Storage Gen2
Intermediate
1h 9m
Dec 11, 2018
More courses by Xavier Morera
Section Introduction Transcripts
Section Introduction Transcripts

Course Overview
Hello, and welcome to this Pluralsight course, Developing Spark Applications with Python and Cloudera. I am Xavier Morera and I help developers understand Enterprise Search and Big Data. Did you know that Spark as a big data processing engine is at least 10 to 100 times faster than Hadoop or MapReduce and that on top of that, it is easier to learn, widely adopted, and used for a diverse range of applications? In this course we're going to learn how to create Spark applications in a very popular and also easy-to-use language, Python, and because infrastructure is important, we will leverage the first and one of the most widely used Hadoop distributions, CDH, which stands for Cloudera's Distribution including Hadoop. Some of the major topics that we will cover include getting an environment set up with Spark and some interesting data, namely CDH plus StackOverflow, understanding Spark, an overview and getting technical with Spark. Then we will learn how to work with the original core abstraction of Spark, the RDD or resilient distributed dataset. Next we will cover data frames and Spark SQL, which helps us become more proficient with Spark quicker. And finally we will talk at a high level about datasets, which are not to be used with Python because it's dynamically typed and we'll also cover a few related topics. By the end of this course you will be able to create Spark applications with Python and Cloudera, but before beginning the course you should be familiar with programming, preferably with Python, but I include also a small refresher module in case you need a jumpstart. Additionally, you will need a cluster, but I will explain how to get your infrastructure set up in multiple different ways. I hope you will join me on this journey to learn about Spark with the Developing Spark Applications with Python and Cloudera course at Pluralsight.

You're the smartest person in the room

PROVE IT

Your team won't just close the skills gap

THEY'LL HURDLE IT