Solving the Duplication Crisis

Table of Contents

1. Challenge

   Step 1: Getting started

   You have inherited a Node.js customer-data pipeline that runs fine on a few hundred records but freezes the server on the production-volume 100,000-row dataset.

   Every slow function shares the same shape: a filter callback that calls Array.includes or Array.indexOf on a second array, turning each pass into an O(n²) scan.

   In this lab you'll refactor the offending code to use Set and Map, so each membership check becomes O(1) on average and the same machine can serve production traffic without freezing. ### Check the baseline performance

   Before changing any code, run the test suite once so you can see the baseline behavior.

   In the terminal, make sure you’re in the application directory, then run:

   ./runTest.sh
   

   The small-fixture correctness checks pass, but every production-volume check fails its performance budget.

   Over the next three steps you'll make each production-volume check pass by reaching for Set and Map in the six source files under src/. > Stuck on a task? Peek at the matching solution/stepN/ folder for the step you're on to see a working implementation. Try the refactor yourself first; the solution is there as a safety net. info> This lab experience was developed by the Pluralsight team using Forge, an internally developed AI tool utilizing Gemini technology. All sections were verified by human experts for accuracy prior to publication. For issue reporting, please contact us.

2. Challenge

   Step 2: Replace membership checks with a Set

   The blocklist filter is the most direct example of the performance bug. Array.includes walks the blocklist from the start every time it is called, so filtering 100,000 customers against a 10,000-id blocklist can cost up to one billion comparisons.

   Building a Set once turns every subsequent lookup into O(1) average time, while the rest of the function shape stays the same.

3. Challenge

   Step 3: Deduplicate primitives with a Set

   When you need to deduplicate primitive values, such as strings, numbers, and booleans, a Set is the right tool.

   Built-in equality means you don’t need to write a custom key function, and a single pass is enough: check whether each value has been seen, and if it hasn’t, add it.

4. Challenge

   Step 4: Deduplicate objects with a Map and intersect with a Set

   When the items you want to deduplicate are objects, a plain Set won't help because two objects with the same email are still distinct references. The right tool is a Map keyed by whatever derived value defines uniqueness for your domain.

   When you need to compute the intersection of two collections, building one of them into a Set once turns the whole operation into O(n + m).

5. Challenge

   Lab complete

   Confirm the full suite is green

   Before finishing the lab, run the full test suite to confirm that every refactor works at both small-fixture and production volume. Nicely done. You took a customer-data pipeline that froze on 100,000 records and made it complete the same workload inside a tight performance budget without scaling up the hardware.

   You practiced three reusable patterns:

   • Build a Set once before a hot membership check.
   • Use a Set to deduplicate primitives in one pass.
   • Use a Map keyed by a derived value to deduplicate objects.

   Next, try spotting the same shape in your own code. Any loop or filter callback that calls .includes(...), .indexOf(...), or .some(...) against another collection may be a candidate for the same refactor.