Table of Contents

1. Challenge

   Exploring and Running the Application (Starting, Nonparallel)

   STEPS

   Exploring and Running the App

   Understanding the App

   This App is intended to verify cryptographic signatures which come batched together in a large file (data.csv)

   Open data.csv - you'll notice 128 lines of numeric CSV data.

   This data was created with the slow hash function found in src/etc.js

   import { pbkdf2Sync } from "crypto"
   
   export function slow_hash(value, iterations = 99999, secret = "secret") {
       return pbkdf2Sync(value, secret, iterations, 32, "sha256").toString("hex")
   }
   

   The code above implements the pbkdf2Sync algorithm, which is provided in a method from the crypto library, Node's built in tool for hashes like the one above.

   Note that due to the large amount of iterations, it takes a significant amount of time to verify every entry in the .csv file.

   How can you ensure all the entries are verified while making the work go as quickly as possible?

   Naively Verifying the Data

   As this lab is opened, the file src/lab/index.js contains a naive solution for verifying the data.

   function verify_document_nonparallel(path) {
       let start = performance.now()
       let document = readFileSync(path, "utf8")
       let lines = document.trim().split("\n")
   
       for (let i = 0; i < lines.length; i++) {
           let line = lines[i]
           let [a, b, c] = line.split(",")
           let actual_c = slow_hash(`${a},${b}`)
   
           if (actual_c !== c) {
               throw new Error(`Line ${i + 1} failed verification`)
           }
       }
   
       let end = performance.now()
       let seconds = ((end - start) / 1000).toFixed(2)
   
       console.log(`Verified ${lines.length} lines in ${seconds}s`)
   }
   

   The code above defines the verify_document_nonparallel method, which:

   • Loads the spec data from the provided path using fs, a commonly used built in Node library
   • Reads the document, one line at a time
   • Runs the slow-hash function. Each line of the document contains three entries - the first two must be concatenated, and then hashed. If the hash matches the third entry, the document is valid.

   As you can see, the script runs a simple, single-threaded for loop with no optimizations.

   You can run the script with npm run test-lab.

   It should output something like:

   Verified 128 lines in 9.91s
   

   Note, how in total, it takes about 10 seconds to verify all 128 lines.

2. Challenge

   Creating a Worker

   A worker is a small unit of JavaScript code that can be executed outside of the main thread.

   In order to make the cryptographic process parallel, you'll define a worker whose unit of work is to verify a single cryptographic statement.

   Create a new file, src/demo/worker.js

   // src/demo/worker.js
   import { parentPort, workerData } from "worker_threads"
   import { slow_hash } from "@pluralsight/workers-lab/etc"
   
   let { lines, start } = workerData
   
   for (let i = 0; i < lines.length; i++) {
       let [a, b, c] = lines[i].split(",")
   
       if (slow_hash(`${a},${b}`) !== c) {
           throw new Error(`Line ${start + i + 1} failed verification`)
       }
   }
   
   parentPort.postMessage("done")
   

   This worker accepts any number of lines to cryptographically verify, and verifies them in sequence. By calling the method, parentPort.postMessage, data can be returned to the caller thread (in this case, indicating that all entries have been verified.) To verify entries in parallel, the worker will need to be called from the main thread.

3. Challenge

   Using Available Parallelism

   Available parallelism is a built-in method in JavaScript indicating how many worker instances a workload can support. This varies from many on more powerful machines to just a few for lightweight hardware.

   Create a new file, src/demo/index.js

   // import to read the documents
   import { readFileSync } from "fs"
   
   // os's availableparallelism lets the workload see parallel processing information
   import { availableParallelism } from "os"
   
   // Worker is a built in method provided by JavaScript used to instantiate a worker.
   import { Worker } from "worker_threads"
   
   function verify_document_parallel(path) {
       let worker_count = availableParallelism()
   
       console.info({
         worker_count
       })
   }
   
   await verify_document_parallel("data.csv")
   

   Now, you can inspect the workload's available parallelism by running

   npm run test-demo
   

   Which outputs a value as follows:

   { worker_count: 4 }
   

4. Challenge

   Parallelizing Code with Worker Threads

   A worker can be created to address a workload with the new Worker() constructor, which accepts a path to the file containing the worker code.

   In order to divide the work among the workers we have available, you must divide the raw data into chunks, with one chunk for each worker.

   Update src/demo/index as follows, in order to create a worker for each chunk and assign it to them.

   import { readFileSync } from "fs"
   import { availableParallelism } from "os"
   import { Worker } from "worker_threads"
   
   function verify_document_parallel(path) {
     let lines = readFileSync(path, "utf8")
       .trim()
       .split("\n")
   
     console.info("Verifying lines", lines.length)
     let promises = []
   
     let worker_count = availableParallelism()
   
     let chunk_size = Math.ceil(lines.length / worker_count)
   
     for (let i = 0; i < worker_count; i++) {
       let start = i * chunk_size
       let end = start + chunk_size
   
       let worker = new Worker("./src/solution/worker.js", {
         workerData: {
           start,
           lines: lines.slice(start, end)
         }
       })
   
       promises.push(new Promise((resolve, reject) => {
         worker.once("message", resolve)
         worker.once("error", reject)
       }))
     }
   
     return Promise.all(promises)
   }
   
   let data = await verify_document_parallel("data.csv")
   console.info("Verified data")
   

   The above code does the following:

   • It defines the verify_document_parallel method, which:
     • Loads the spec/demo file defined in path
     • Computes the amount of work each worker should be given by dividing the total workload by the available parallelism
     • Initializes one worker per prescribed chunk and runs them
     • All the worker's processes are abstracted into one Promise, which is awaited for with Promise.all, allowing for neat and tidy code.

   You can test your script with npm run test-demo, output should look as follows:

   Verifying lines 128
   Verified data
   

5. Challenge

   Measuring Performance

   You now have a functional script which runs a cryptographic process in parallel using several workers. Although it's already functionally complete, the point of using parallelism is to conserve time, so in this step, you'll add a log which measures how long the task took.

   Update src/demo/index.js as follows (it should now match src/solution/index.js)

   import { readFileSync } from "fs"
   import { availableParallelism } from "os"
   import { Worker } from "worker_threads"
   
   function verify_document_parallel(path) {
       let start_time = performance.now()
   
       let lines = readFileSync(path, "utf8")
           .trim()
           .split("\n")
   
       let worker_count = Math.min(availableParallelism(), lines.length)
       let chunk_size = Math.ceil(lines.length / worker_count)
   
       let promises = []
   
       for (let i = 0; i < worker_count; i++) {
           let start = i * chunk_size
           let end = start + chunk_size
   
           let worker = new Worker("./src/demo/worker.js", {
               workerData: {
                   start,
                   lines: lines.slice(start, end)
               }
           })
   
           promises.push(new Promise((resolve, reject) => {
               worker.once("message", resolve)
               worker.once("error", reject)
           }))
       }
   
       return Promise.all(promises).then(() => {
           let seconds = ((performance.now() - start_time) / 1000).toFixed(2)
           console.log(`Verified ${lines.length} lines in ${seconds}s`)
       })
   }
   
   await verify_document_parallel("data.csv")
   

   The above has been updated from the sample in the previous step to also do the following:

   • The time the process starts is measured by performance.now
   • When all the tasks are done, a time delta is created by calling performance.now again and subtracting the two values.

   Now, you can test your finished App with npm run test-demo, which should output something like:

   Verified 128 lines in 3.12s
   

   If you'd like to review your progress, you can try again running the original script - npm run test-lab Which should output

   Verified 128 lines in 10.08s
   

   Thanks to Workers, you've reduced the time it takes to verify 128 lines by 70%! Great job.