Solving the Duplication Crisis

Table of Contents

1. Challenge

   Step 1: Getting Started

   You have inherited a Node.js customer-data pipeline that runs fine on a few hundred records but freezes the server on the production-volume 100,000-row dataset. Every slow function shares the same shape — a filter callback that calls Array.includes or Array.indexOf on a second array, turning each pass into an O(n²) scan. In this lab you'll refactor the offending code to use Set and Map, so each membership check becomes O(1) on average and the same machine can serve production traffic without freezing. ### Check the baseline performance

   Before changing any code, run the test suite once so you can see the baseline behavior. In the terminal, make sure you’re in the application directory, then run:

   ./runTest.sh
   

   The small-fixture correctness checks pass, but every production-volume check fails its performance budget.

   Over the next three steps you'll make each production-volume check pass by reaching for Set and Map in the six source files under src/. > Stuck on a task? Peek at the matching solution/stepN/ folder for the step you're on to see a working implementation. Try the refactor yourself first; the solution is there as a safety net. info> This lab experience was developed by the Pluralsight team using Forge, an internally developed AI tool utilizing Gemini technology. All sections were verified by human experts for accuracy prior to publication. For issue reporting, please contact us.

2. Challenge

   Step 2: Replace Membership Checks with a Set

   The blocklist filter is the most direct example of the performance bug. Array.includes walks the blocklist from the start every time it is called, so filtering 100,000 customers against a 10,000-id blocklist can cost up to one billion comparisons. Building a Set once turns every subsequent lookup into O(1) on average — and the rest of the function shape stays the same.

3. Challenge

   Step 3: Deduplicate Primitives with a Set

   When the values you want to deduplicate are primitives — strings, numbers, booleans — a Set is the right tool. Built-in equality means you don't write a custom key function, and a single pass is enough: ask whether each value has been seen, and if not, add it.

4. Challenge

   Step 4: Deduplicate Objects with a Map and Intersect with a Set

   When the items you want to deduplicate are objects, a plain Set won't help — two objects with the same email are still distinct references.

   The right tool is a Map keyed by whatever derived value defines uniqueness for your domain. And when you need to compute the intersection of two collections, building one of them into a Set once turns the whole operation into O(n + m).

5. Challenge

   Lab Complete

   Nicely done. You took a customer-data pipeline that froze on 100,000 records and made it sail through the same workload inside a tight per-step performance budget — without scaling up the hardware. Three patterns now sit in your toolbox:

   • Build a Set once before a hot membership check.
   • Reach for a Set to deduplicate primitives in one pass.
   • Reach for a Map keyed by a derived value to deduplicate objects.

   Next, try spotting the same shape in your own code: any filter whose callback contains .includes(...), .indexOf(...), or .some(...) against another collection is a candidate for the same refactor.