Fixing a C# Application with C# Failure Stack Traces

Table of Contents

1. Challenge

   Step 1: Meet OrderPulse and its failing overnight batch

   Meet OrderPulse

   OrderPulse is the overnight batch job that a mid-size logistics company runs after the warehouse closes. Every night it reads the day's order lines, applies each customer's discounts, calls out to a third-party pricing assembly for tax, updates a shared inventory count across a small pool of worker threads, and writes a settlement report that finance reads first thing in the morning. It has run unattended for two years. Last week it started failing in four different ways on the same night, and you are the first person to actually open the code. ### The pipeline, stage by stage

   1. Parse — each raw order line is turned into an Order record. Malformed lines should be reported, not fatal.
   2. Discount — a per-order discount pipeline adjusts the order's price based on the customer's active discounts, then rounds it for settlement.
   3. Price — a third-party pricing assembly computes tax for the order; an adapter in OrderPulse translates its result into the batch's own numbers.
   4. Inventory — each order's line items decrement a shared stock count, read and written by whichever worker thread picked up that order.
   5. Settle — a report is written combining totals, defects, and final stock counts.

   Each stage is a place a defect can hide, and each defect in this lab hides in exactly one of them. ### Four failure reports on the same night

   The on-call rotation collected four symptoms before escalating:

   • A crash. The batch process exits mid-run with an unhandled exception and a full stack trace pointing into the order-line parser.
   • Drifting totals. The settlement report's totals are subtly wrong, but only after several orders have been discounted in a row — the first order or two always look right.
   • An intermittent failure. Expedited orders fail sometimes, not always, and only the on-call engineer who happened to have a backorder-heavy batch that night saw it.
   • Inventory drift. Stock counts are wrong at the end of the run, but only when the batch is configured to run with more than one worker thread.

   You will fix these in the order they were reported: crash first, because it's loudest and best-documented, concurrency last, because it needs everything else already working before it's worth chasing. ### Run the batch before you change anything

   Open the Terminal tab and run the whole batch:

   dotnet run --project src/Lab.csproj
   

   Each task in this lab has its own scenario, and the run ends with a summary of all of them:

   === Summary ===
     2.1  CRASH  Parse the order feed
     2.2  WRONG  Classify rejected lines
     3.1  WRONG  Discount consecutive orders
     ...
     5.3  WRONG  Settle the night's batch
   

   CRASH means the scenario threw an exception, WRONG means it finished with the wrong numbers, and OK means it matches what finance expects. Right now nothing is OK.

   While you work on a task, run only its scenario by passing the task number:

   dotnet run --project src/Lab.csproj -- 2.1
   

   A single scenario isn't wrapped in a try/catch, so a crash prints the real unhandled exception and its full stack trace, exactly as last night's batch did. Run dotnet run --project src/Lab.csproj -- list to see every scenario. ### Checkpoints: observe, change, compare

   Every task follows the same loop, and each one ends with a checkpoint:

   1. Observe. Run the task's scenario and read its output before you open the code.
   2. Gather evidence. Add a probe, read a stack trace, or inspect an assembly to test one hypothesis.
   3. Change. Make the smallest fix the evidence supports.
   4. Compare. Rerun the same scenario and compare the output with your first run. The task's result line should now read OK.

   Only then select Check for the task. The check runs the task's tests, so it confirms what you've already seen for yourself. ### Debugging from the terminal

   This lab's editor has no attached debugger, so you'll use terminal equivalents of the debugger features each failure calls for.

   They answer the same questions:

   | Debugger feature | What you'll use in this lab | |---|---| | Exception and call stack window | The unhandled exception's stack trace, with file and line numbers | | DataTips and the Watch window | Probe.Watch("label", value), which prints a value every time the line runs | | Tracepoints with conditions and hit counts | Probe.Trace("id", message, when: ..., fromHit: ...), which logs without stopping the run | | Symbol loading and decompilation | dotnet run --project src/Lab.csproj -- inspect, which shows an assembly's symbol-file availability, API, and IL | | Hot Reload | dotnet watch run, which applies your edits to the process while it runs |

   Probe lives in src/Probe.cs. Add a probe where you would set a watch or a tracepoint, run one scenario, and read the values in the terminal. Remove it once you have your answer, the same way you'd clear a breakpoint. ### The stance this lab teaches

   The instinct under pressure is to open the file that sounds guilty and start editing. Resist it.

   For each failure, you will:

   • Read the evidence you already have (a stack trace, a watched value, a tracepoint log, an assembly's IL) before touching any code.
   • Form one hypothesis that evidence could actually disprove.
   • Pick the tool that tests that specific hypothesis.

   Every task in this lab hands you evidence first and a fix second, in that order, because that is the order that actually works on a codebase you didn't write.

2. Challenge

   Step 2: Reproduce the crash and classify the defect

   Reproduce the crash

   The first report is the easiest to start from: a full stack trace.

   Somewhere in last night's run, OrderLineParser.ParseLine threw an exception that nothing caught, and the process exited.

   Before changing anything, reproduce it:

   dotnet run --project src/Lab.csproj -- 2.1
   

   The scenario parses one good line, then crashes on the second:

     Parsing "ORD-1002|SKU-07||9.50|true|false"
   Unhandled exception. System.FormatException: The input string '' was not in a correct format.
      at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
      at System.Int32.Parse(String s, IFormatProvider provider)
      at Lab.OrderLineParser.ParseLine(String rawLine) in /home/ps-user/workspace/src/OrderLineParser.cs:line 65
      at Program.<<Main>$>g__RunParse|0_1() in /home/ps-user/workspace/src/Program.cs:line 128
      at Program.<Main>$(String[] args) in /home/ps-user/workspace/src/Program.cs:line 63
   

   Read a stack trace from the top down:

   • The first line names the exception type and its message. An empty string ('') failed to parse as a number.
   • The frames in System.* are where .NET detected the problem, not where it was caused. Skip past them.
   • The first frame in your own code, Lab.OrderLineParser.ParseLine ... OrderLineParser.cs:line 65, is where to start looking.
   • The frames below it are the callers: they tell you how execution got there.

   Open src/OrderLineParser.cs at that line. The bad value came from the input line itself, the quantity field is empty, so this is a bad input defect.

   That classification tells you where the fix belongs: in validation inside the parser, not in a try/catch wrapped around the caller. ### Classify before you hypothesize

   A parser that no longer crashes still needs somewhere to put what went wrong. Not every validation failure is the same kind of problem: a missing field, an unparsable number, and an unrecognized SKU are three different shapes of defect, and the settlement report is supposed to tell finance which is which. Classifying failures into named categories, before you go hunting for root causes, is what turns "something failed" into a hypothesis you can actually test.

   The 2.2 scenario runs the classifier over the reasons in last night's rejected-lines log and marks each one that lands in the wrong bucket:

   dotnet run --project src/Lab.csproj -- 2.2
   

3. Challenge

   Step 3: Trace corrupting state through the discount pipeline

   Trace state instead of guessing at it

   The second report has no exception and no stack trace, just totals that are wrong, and only after several orders. That pattern (correct at first, wrong later) suggests state may be leaking between calls.

   In an IDE you'd hover over the variable for a DataTip, or add it to the Watch window and step from call to call. Here, Probe.Watch gives you the same view in the terminal: it prints a value every time execution reaches that line.

   For example, a watch on the first line of a method:

   Probe.Watch("_runningDiscount on entry", _runningDiscount);
   

   prints one line per call, right above that call's output:

    [watch] _runningDiscount on entry = 0
   ORD-3001 $100.00 with 10% off -> $90.00 (expected $90.00)
    [watch] _runningDiscount on entry = 0.10
   ORD-3002 $100.00 with 10% off -> $80.00 (expected $90.00)   <-- wrong
   

   The second watch line is the evidence: a fresh order starts with the previous order's discount already applied. You've found the exact call where the value stops matching what you expect, instead of working backward from a wrong number at the end.

   Start with the 3.1 scenario:

   dotnet run --project src/Lab.csproj -- 3.1
   ``` ### Order of operations matters when it's silent
   
   Stacked discounts fixed, the totals still don't land on the cent. Rounding and discounting are each simple on their own, but applying them in the wrong order loses precision with no error message: the code runs fine, it just runs on the wrong number.
   
   Run the `3.2` scenario and look at how many decimal places the results have:
   
   ```bash
   dotnet run --project src/Lab.csproj -- 3.2
   

   Before you assume the rounding logic is wrong, watch the value flowing into the rounding call itself. ### Stale state across an order boundary

   Even with prices correct, an inventory adjustment can still read a value cached from the order processed just before it.

   Watching that cached field next to the real stock count, across the exact moment one order ends and the next begins, shows you directly whether it refreshed. There's no need to guess from the final stock count.

   dotnet run --project src/Lab.csproj -- 3.3
   

4. Challenge

   Step 4: Corner the intermittent expedited-order failure

   Corner an intermittent failure with tracepoints

   The third report only shows up at specific points in a long run, which makes a plain breakpoint useless: you'd step through hundreds of unrelated orders waiting for the one that matters. A tracepoint logs a message every time execution reaches it without stopping the run, and a hit-count or conditional expression lets it fire only when it matters.

   Probe.Trace is the terminal version:

   Probe.Trace("batch-counter", $"currentIndex={currentIndex}", fromHit: 248);
   

   • Every call counts as one hit for its id ("batch-counter" here), whether or not it prints.
   • fromHit: 248 is a hit-count condition: nothing prints until the 248th hit.
   • when: <expression> is a conditional tracepoint: the line prints only when the expression is true.

   Each printed line shows the id and hit number, so you can tell exactly which pass through the code it came from:

   [trace batch-counter hit 248] currentIndex=247
   

   Start with the batch loop:

   dotnet run --project src/Lab.csproj -- 4.1
   ``` ### Narrowing by condition, not by luck
   
   The expedited-order failure is narrower still. The batch handles dozens of expedited orders without trouble, then crashes on one of them:
   
   ```bash
   dotnet run --project src/Lab.csproj -- 4.2
   

   The stack trace tells you which line threw a NullReferenceException, but not which order reached it or what state it was in. A conditional tracepoint, one that only fires when an expression you write is true, logs the state for exactly the combination you suspect, in a single run.

5. Challenge

   Step 5: Cross the assembly boundary and close the inventory race

   Step into code you don't own

   The tax amounts are wrong, and the bug isn't obviously in OrderPulse's own code. OrderPulse calls TaxCalculator.Calculate in Contoso.Pricing, a third-party assembly that ships as a compiled DLL (lib/Contoso.Pricing.dll.b64, unpacked at build time). You don't have its source, and you don't have its symbols.

   In an IDE, you could use matching symbols to debug the assembly's original source or decompilation to investigate code without the source.

   In this lab, the inspect command provides assembly information, IL, and runtime evidence from the terminal:

   dotnet run --project src/Lab.csproj -- inspect
   

   It prints four kinds of evidence:

   • Identity: the assembly's name, version, and where it was loaded from.
   • Symbols: whether a matching .pdb file was found. Without one, the assembly's stack frames have no file or line numbers.
   • API and IL: each public method's signature, and the IL instructions it actually runs, which is what a decompiler reads to rebuild C#.
   • Behavior: the results of calling the method with known inputs, and a stack trace thrown from inside it.

   Treat all of it as evidence about the contract, not as source code. IL has lost the original constant names and comments, and a decompiler's C# is a reconstruction of it. The fix belongs entirely on your side of the boundary. ### The last symptoms: races only workers can trigger

   With pricing correct, what's left is inventory drift when several workers run at once, the one kind of failure that never reproduces with a single thread. That shape points at unsynchronized access to shared state: two workers both read the same stock count, both compute their own reservation, and both write back, so one write silently overwrites the other.

   Race conditions can be difficult to reproduce because their behavior depends on execution timing. Running the application under load can help reveal them. So you'll verify these fixes in a process that keeps running.

   The soak scenario repeats Tasks 5.2 and 5.3 every two seconds, and dotnet watch applies each edit you save to that running process with Hot Reload:

   dotnet watch run --project src/Lab.csproj -- soak
   

   Every round prints the reservations and settlement figures for that round, and when you save a file, dotnet watch reports the change being applied:

   dotnet watch ⌚ File updated: ./src/InventoryTracker.cs
   dotnet watch 🔥 C# and Razor changes applied in 308ms.
   --- Round 22 at 21:15:11 ---
   

   The rounds before that line ran your old code, and the rounds after it run your new code, in the same process with no restart.

   Note: If dotnet watch can't apply a change without restarting, follow the prompt to restart the application. However, the goal of this exercise is to apply and verify the fix in the same running process using Hot Reload. ### Verify under load, not just in theory

   A synchronization fix that you only check by reading the code isn't verified. With the soak run still going, the stock line is now steady, but the revenue line in the same rounds still changes from round to round. That's the same signature: some other shared state is being updated from several threads at once.

6. Challenge

   Step 6: Run the application

   Every task is done, and each one was graded as you completed it. Now run the whole batch once more, the same way you did at the start of the lab:

   dotnet run --project src/Lab.csproj
   

   Compare the summary with your first run. Every scenario that crashed or produced wrong numbers should now read OK:

   === Summary ===
     2.1  OK     Parse the order feed
     2.2  OK     Classify rejected lines
     ...
     5.3  OK     Settle the night's batch
   ``` ### Run the checks yourself
   
   The lab ran each task's test for you as you went, but they are ordinary tests and
   you can run them whenever you like. The whole suite:
   
   ```bash
   dotnet test tests/Lab.Tests.csproj --nologo --verbosity quiet
   

   Or one task at a time:

   bash runTest.sh Task_2_1
   

   Each failure names the task it belongs to, so the output tells you which step to go back to rather than which line to stare at. Two things worth trying before you finish: break something on purpose and watch the matching scenario and check fail, then put it back. Knowing what a failure looks like is worth as much as knowing what a pass looks like, and it costs you nothing here.

   When you are done, everything you wrote is still in the editor tabs. The lab is yours to keep reading.