Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.
  • Labs icon Lab
  • A Cloud Guru

Running a Pyspark Job on Cloud Dataproc Using Google Cloud Storage

This hands-on lab introduces how to use Google Cloud Storage as the primary input and output location for Dataproc cluster jobs. Leveraging GCS over the Hadoop Distributed File System (HDFS) allows us to treat clusters as ephemeral entities, so we can delete clusters that are no longer in use, while still preserving our data.


Path Info

Clock icon Intermediate
Clock icon 30m
Clock icon Aug 15, 2019

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Table of Contents

  1. Challenge

    Prepare Our Environment

    1. First, we need to enable the Dataproc API:
    gcloud services enable
    1. Then create a Cloud Storage bucket:
    gsutil mb -l us-central1 gs://$DEVSHELL_PROJECT_ID-data
    1. Now create the dataproc cluster:
    gcloud dataproc clusters create wordcount --region=us-central1 --zone=us-central1-f --single-node --master-machine-type=n1-standard-2
    1. And finally, download the file that will be used for the pyspark job:
    gsutil cp -r gs://acg-gcp-labs-resources/data-engineer/dataproc/* .
  2. Challenge

    Submit the Pyspark Job to the Dataproc Cluster

    In Cloud Shell, type:

    gcloud dataproc jobs submit pyspark --cluster=wordcount --region=us-central1 -- 
  3. Challenge

    Review the Pyspark Output

    1. In Cloud Shell, download output files from the GCS output location:
    gsutil cp -r gs://$DEVSHELL_PROJECT_ID-data/output/* .

    Note: Alternatively, we could download them to our local machine via the web console.

  4. Challenge

    Delete the Dataproc Cluster

    1. We don't need our cluster any longer, so let's delete it. In the web console, go to the top-left menu and into BIGDATA > Dataproc.

    2. Select the wordcount cluster, then click DELETE, and OK to confirm.

    Our job output still remains in Cloud Storage, allowing us to delete Dataproc clusters when no longer in use to save costs, while preserving input and output resources.

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Provided environment for hands-on practice

We will provide the credentials and environment necessary for you to practice right within your browser.

Guided walkthrough

Follow along with the author’s guided walkthrough and build something new in your provided environment!

Did you know?

On average, you retain 75% more of your learning if you get time for practice.

Start learning by doing today

View Plans