Table of Contents

1. Challenge

   Prepare Our Environment

   First, we need to enable the Dataproc API:

   gcloud services enable dataproc.googleapis.com
   

   Then create a Cloud Storage bucket:

   gsutil mb -l us-central1 gs://$DEVSHELL_PROJECT_ID-data
   

   Now we'll first enable the Cloud Resource Manager API , enable private IP Google access, then create the ephemeral dataproc cluster.

   gcloud services enable cloudresourcemanager.googleapis.com
   
   gcloud compute networks subnets update default --region=us-central1 --enable-private-ip-google-access
   
   gcloud dataproc clusters create wordcount --region=us-central1 --single-node --master-machine-type=n1-standard-2
   

   And finally, download the wordcount.py file that will be used for the pyspark job:

   gsutil cp -r gs://acg-gcp-labs-resources/data-engineer/dataproc/* .
   

2. Challenge

   Submit the Pyspark Job to the Dataproc Cluster

   In Cloud Shell, type:

   gcloud dataproc jobs submit pyspark wordcount.py --cluster=wordcount --region=us-central1 -- 
   gs://acg-gcp-labs-resources/data-engineer/dataproc/romeoandjuliet.txt 
   gs://$DEVSHELL_PROJECT_ID-data/output/
   

3. Challenge

   Review the Pyspark Output

   1. In Cloud Shell, download output files from the GCS output location:

   gsutil cp -r gs://$DEVSHELL_PROJECT_ID-data/output/* .
   

   Note: Alternatively, we could download them to our local machine via the web console.

4. Challenge

   Delete the Dataproc Cluster

   1. We don't need our cluster any longer, so let's delete it. In the web console, go to the top-left menu and into BIGDATA > Dataproc.

   2. Select the wordcount cluster, then click DELETE, and OK to confirm.

   Our job output still remains in Cloud Storage, allowing us to delete Dataproc clusters when no longer in use to save costs, while preserving input and output resources.