Running a Pyspark Job on Cloud Dataproc Using Google Cloud Storage

First, we need to enable the Dataproc API:

gcloud services enable dataproc.googleapis.com

Then create a Cloud Storage bucket:

gsutil mb -l us-central1 gs://$DEVSHELL_PROJECT_ID-data

Now we'll first enable the Cloud Resource Manager API , enable private IP Google access, then create the ephemeral dataproc cluster.

gcloud services enable cloudresourcemanager.googleapis.com

gcloud compute networks subnets update default --region=us-central1 --enable-private-ip-google-access

gcloud dataproc clusters create wordcount --region=us-central1 --single-node --master-machine-type=n1-standard-2

And finally, download the wordcount.py file that will be used for the pyspark job:

gsutil cp -r gs://acg-gcp-labs-resources/data-engineer/dataproc/* .

In Cloud Shell, type:

gcloud dataproc jobs submit pyspark wordcount.py --cluster=wordcount --region=us-central1 -- 
gs://acg-gcp-labs-resources/data-engineer/dataproc/romeoandjuliet.txt 
gs://$DEVSHELL_PROJECT_ID-data/output/

1. In Cloud Shell, download output files from the GCS output location:

gsutil cp -r gs://$DEVSHELL_PROJECT_ID-data/output/* .

Note: Alternatively, we could download them to our local machine via the web console.

1. We don't need our cluster any longer, so let's delete it. In the web console, go to the top-left menu and into BIGDATA > Dataproc.

2. Select the wordcount cluster, then click DELETE, and OK to confirm.

Our job output still remains in Cloud Storage, allowing us to delete Dataproc clusters when no longer in use to save costs, while preserving input and output resources.