- Lab
-
Libraries: If you want this lab, consider one of these libraries.
- Core Tech
Tracing Failures in an ASP.NET Core Application
The warehouse platform runs a nightly job that ingests batches of warehouse order records through an ASP.NET Core API and renders their status on an internal dashboard. Lately the import silently swallows exceptions instead of surfacing them, the dashboard sometimes shows a wrong order total, a status stylesheet renders incorrectly, and roughly one batch a week fails outright with no indication of which record caused it. You are the engineer picking up the pager: you will restructure how the service handles and logs failures, trace the HTTP traffic and served assets behind the dashboard bugs, and build a tool that can pinpoint the exact poisoned record in a batch of hundreds.
Lab Info
Table of Contents
-
Challenge
Step 1: Meet the on-call rotation
Meet the on-call rotation
The team runs a warehouse platform: every night a batch-import job reads order records into an ASP.NET Core API, and an internal dashboard renders each batch's status and total for the operations team to review in the morning. You are the engineer who just picked up the pager. For the last two weeks the rotation notes have piled up with the same handful of complaints: the import sometimes finishes green even though records inside it failed, one status code or another comes back wrong when something does go wrong, the dashboard's total is occasionally off, a status color stopped rendering, and roughly once a week an entire batch aborts with no indication of which record caused it. ## The pieces you'll be working across
Three layers make up the service you're inheriting:
OrderParserturns raw lines of a batch file intoOrderRecordobjects. It's the first thing that touches untrusted data, and it's currently the reason failures vanish instead of surfacing.- Middleware and
Program.cswrap every HTTP request. Right now the exception-handling middleware exists but reports every failure the same way, and it's wired into the pipeline in the wrong place to even catch what it's meant to. - The dashboard's supporting endpoints and assets compute a batch total, serialize it as JSON, and serve a small stylesheet that colors each order's status. Each of those has a specific, findable defect. ## What you'll fix, in order
- Step 2 — rewrite
OrderParserso a failing line throws with its original context intact instead of disappearing into a swallowedcatch, and so a stream is never left undisposed when parsing fails partway through. - Step 3 — give the exception-handling middleware the ability to map specific exception types to the right status code, replace an interpolated log message with structured fields, and fix the one line in
Program.csthat currently registers that middleware too late to catch anything. - Step 4 — chase a wrong dashboard total back to a broken accumulator, follow the HTTP response to a JSON field name mismatch, and fix a stylesheet selector that no longer matches the markup it's meant to style.
- Step 5 — build a bisection routine that halves a failing batch of hundreds of records down to the single one that's poisoned, wire it into the batch endpoint so a failure produces a diagnosis instead of a crash, and diff that isolated record against a known-good reference. ## The habits this lab builds
Each step leans on a different diagnostic habit you'll use for the rest of your on-call career: preserving exception context instead of discarding it, guaranteeing cleanup with
finally/usingregardless of how a method exits, ordering middleware so it actually wraps what it's supposed to protect, logging with structured fields instead of baked-in strings, tracing an HTTP exchange and a served asset the way browser dev tools would show them, and bisecting a large input to isolate a single bad record. None of these are exotic techniques — they're the ones that turn "it's broken somewhere in there" into a precise, fixable diagnosis.info> If you get stuck, you can refer to the provided solution code for each task, available in the
solutionfolder. -
Challenge
Step 2: Restore trustworthy exception context in the order parser
Restore trustworthy exception context in the order parser
A batch is a stream of lines;
OrderParserturns each one into anOrderRecord. Right now, when a line fails to parse,ParseOrderLinecatches the exception and returnsnull— the batch keeps going, and nobody downstream ever learns which line was bad or why. The fix is not to stop catching the exception; it's to catch it, decorate it with the context only this method has (which line, what data), and let it keep going up. That's exception chaining: wrapping the low-levelFormatExceptionin a higher-levelOrderParseExceptionwhile keeping the original attached asInnerException, so nothing about the original failure is lost.ParseOrderLinenow throws instead of swallowing — but the caller,ParseBatch, currently catches that exception just to dothrow ex;. That single line looks harmless and is one of the most common exception-handling mistakes in C#:throw ex;resets the stack trace to this catch block, so by the time anyone reads the exception, the frames that show where insideParseOrderLinethings actually broke are gone.throw;(no argument) rethrows without disturbing the trace; wrapping the exception in a new one with the original passed asinnerExceptiondoes the same while also letting you add context, like which position in the batch failed. One more failure mode hides inLoadBatchFromStream: it opens aStreamReaderover the batch's stream, reads and parses lines one at a time, and disposes the reader only after the loop finishes. Now that a bad line throws instead of returning null, an exception partway through the loop skips straight past thatDispose()call — the stream is left open for as long as the process runs.finallyblocks (and theusingstatement that expands to one) exist for exactly this: code inside them runs whether the protected block returns normally or throws, which is the only guarantee that a resource acquired at the top of a method is released no matter how the method exits. -
Challenge
Step 3: Instrument the pipeline with structured logging and ordered middleware
Instrument the pipeline with structured logging and ordered middleware
With
OrderParserthrowing informative exceptions, the next question is what the HTTP layer does with them. ASP.NET Core exception-handling middleware sits in the request pipeline, wraps the call to the next piece of middleware (and eventually the endpoint) in a try/catch, and turns whatever exception comes out into an HTTP response.ExceptionHandlingMiddlewarealready does that much, but it treats every exception identically: everything becomes a 500. A batch that failed because of bad input data (anOrderParseException) and a batch that failed because of a genuine server bug should not look the same to the client calling the API. Status codes tell the caller what happened; logs tell you what happened when the caller isn't looking.BatchLogger.LogFailurecurrently builds its log message by interpolatingbatchIdandorderIddirectly into a string. That reads fine in a console, but once logs are shipped to any aggregator, an interpolated string is just text — there's no way to query "every failure for batch B-4471" without a text search.ILogger's message-template overloads (LogError(ex, "Batch {BatchId} failed on order {OrderId}", batchId, orderId)) keepBatchIdandOrderIdas separate structured fields on the log entry, queryable independently of the rendered message. The middleware now knows how to classify exceptions correctly — but only for requests that actually reach it. InProgram.cs,CreateAppcallsapp.MapGetto register the batch endpoints, and only afterward callsapp.UseMiddleware<ExceptionHandlingMiddleware>(). Minimal API pipelines are built in the order theirUse.../Map...calls appear: a middleware registered after an endpoint is defined does not wrap that endpoint, because the endpoint's terminal handling was already spliced into the pipeline first. An exception thrown inside one of thoseMapGethandlers currently propagates past the middleware entirely. -
Challenge
Step 4: Chase a bad total through the debugger and the network tab
Chase a bad total through the debugger and the network tab
The dashboard occasionally shows a batch total that doesn't match the sum of the orders in it. This is the kind of bug that a debugger session (or a careful read of the loop) settles quickly once you form the right hypothesis: is the total wrong because a value is missing from the sum, double-counted, or is the accumulator itself never accumulating?
OrderTotalCalculator.CalculateBatchTotalwalks every order in a batch — check what its loop body actually does to the running total on each pass before assuming the bug is anywhere fancier than that. A correct total can still show up wrong on screen if the JSON carrying it doesn't match what the browser's JavaScript reads. This is exactly what you'd find by opening a browser's Network tab, inspecting the batch-summary response, and comparing its field names against the client code that consumes it: the dashboard'sfetchhandler readsresponse.totalandresponse.batchId, in camelCase.DashboardApi.SerializeSummaryserializes the sameOrderSummaryDtowith the defaultJsonSerializerOptions, which emits C#'s PascalCase property names as-is — a mismatch invisible on the server, obvious the moment you trace the actual bytes on the wire. The last dashboard symptom is visual: an order's status is supposed to render green for ok and red for error, and it currently renders in neither. The dashboard's markup applies a class likeorder-status--ok(a double hyphen, the BEM naming convention this codebase uses for state modifiers) to each status badge.DashboardAssets.GetStatusStylesheetserves the CSS rules meant to style those badges — open it and compare its selectors, character for character, against the class name the markup actually renders, the same comparison you'd make by inspecting the element in a browser's Elements panel and reading its computed styles. -
Challenge
Step 5: Bisect a poisoned batch against a known-good reference
Bisect a poisoned batch against a known-good reference
The hardest recurring failure is the vaguest one: once a week, an entire batch of hundreds of order records aborts, and the only fact available is that one of them is bad. Checking every record by hand doesn't scale, but you don't need to: if you have a way to test whether a sublist still fails, you can split the batch in half, test each half, and recurse into whichever half still fails. Each split halves the number of suspects; a batch of 512 records narrows to the single poisoned one in about nine rounds instead of up to 512 individual checks. This is binary search applied to "which input causes the crash" instead of "where is this value in a sorted list," and it's the fastest way to isolate one bad element out of many when you can only ask yes/no of a group. A bisection routine is only useful if the system actually calls it when things go wrong. Right now
BatchProcessingService.ProcessBatchdetects a failing batch and just throws — the exact crash-with-no-diagnosis behavior this step exists to replace. Once a batch fails, the response to build is not an exception but a result: which record was isolated, so the on-call engineer opens the result instead of a stack trace pointing at whichever record happened to be processed last. Isolating the poisoned record only tells you which one is bad, not why. The last step is comparing it field-by-field against a known-good reference record — the same shape of order that processes successfully every night — so the output names exactly what diverged: a mismatchedOrderId, an unexpectedTotal, or both.RecordDiffer.Diffis where that comparison happens; it should walk each field it knows about and report every one that differs, not just detect that a difference exists somewhere. -
Challenge
Step 6: Watch the import diagnose its own failures
Watch the import diagnose its own failures
Every task is done, and each one was graded as you finished it. Now run the service and see the pieces work together: the dashboard receiving the data it expects, a bad batch coming back with an honest status code, and a 512-order batch that names its own poisoned record.
Start the service
In the Terminal tab, start the service:
bash startApi.shThe script builds the app, starts it in the background, and waits until it answers. When it prints
The order import service is up at http://localhost:5000, the terminal is free for the requests below. The service's logs go to/tmp/order-import.log.Every request in this step goes through a second script,
callApi.sh. Give it a path, and it sends a GET request to the service and prints the response body followed by the HTTP status code. For example,bash callApi.sh /healthcallshttp://localhost:5000/healthand printsokand thenHTTP 200. ### See what the dashboard receivesRequest batch
B-1001's summary, the same request the dashboard'sfetchhandler makes, then the stylesheet it loads:bash callApi.sh /batches/B-1001/summary bash callApi.sh /assets/status.cssThe summary reads
{"batchId":"B-1001","total":50.00,"count":3}. The total is 19.99 + 5.01 + 25.00, because your+=inCalculateBatchTotal(Task 4.1) adds every order. Before that fix, it reported25.00, the last order alone. The field names are camelCase (batchId,total) because of your naming policy inSerializeSummary(Task 4.2). The broken version sentBatchIdandTotal, which the dashboard's JavaScript reads asundefined. The stylesheet's selectors are now.order-status--okand.order-status--error(Task 4.3), so they match the classes the markup renders. All three bugs were invisible on the server and obvious in the bytes on the wire. ### Send a bad batch and an unknown oneBatch
B-1002hasA-3,twelveon its third line. Ask for its summary, then ask for a batch that doesn't exist:bash callApi.sh /batches/B-1002/summary bash callApi.sh /batches/nope/summaryThe bad batch returns
HTTP 422(Unprocessable Entity). Its body names the failing line's position (the solution's wording isFailed to parse batch at position 3.). That position exists becauseParseOrderLinestopped returningnull(Task 2.1) andParseBatchwrapped the failure instead of resetting it withthrow ex;(Task 2.2). Before those fixes, this request failed with500andObject reference not set to an instance of an object., which says nothing about which order was bad. The unknown batch returnsHTTP 400(Bad Request) withUnknown batch nope. Your type mapping inExceptionHandlingMiddleware(Task 3.1) is what tells the two apart: bad data in the batch is a 422, a bad request from the caller is a 400, and anything else stays a 500. ### Diagnose a poisoned 512-order batchThe diagnosis endpoint builds a batch of 512 orders, all
10.00except one, and runs it through yourProcessBatch:bash callApi.sh /batches/B-2001/diagnosisThe response is a diagnosis, not a crash:
"success":false,"poisonedOrderId":"A-338","checks":10. YourProcessBatch(Task 5.2) handed the failure toFindPoisonedRecord(Task 5.1), which halved the batch until one order was left: 1 check of the whole batch plus 9 halvings, instead of up to 512. Thedifferenceslist comes from yourRecordDiffer.Diff(Task 5.3) comparing the poisoned order against the batch's first, known-good order. It reports every field that diverges:OrderId differs: expected A-1, got A-338andTotal differs: expected 10.00, got -4.00.Now read the log entry the failure left behind:
tail -n 1 /tmp/order-import.logIts
Stateholds"BatchId":"B-2001","OrderId":"A-338"as separate fields next to the{OriginalFormat}template. Those are the structured properties from yourBatchLogger.LogFailure(Task 3.2). A log aggregator can filter on them without a text search.When you're done, stop the service with
bash stopApi.sh. You now have an import that fails loudly, names the record at fault, and leaves a log entry you can query.
About the author
Real skill practice before real-world application
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.
Learn by doing
Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.
Follow your guide
All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.
Turn time into mastery
On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.