Guided: Instrumenting an ASP.NET Core 10 App with OpenTelemetry

Table of Contents

1. Challenge

   Introduction

   Welcome to the Instrumenting an ASP.NET Core 10 App with OpenTelemetry Code Lab. In this hands-on lab, you add OpenTelemetry to an existing Web API --- configuring distributed tracing, metrics collection, and structured logging with trace context correlation. By the end of this lab, you will have a fully observable application with correlated traces ready for production monitoring.

   What You Will Learn

   1. Tracing configuration --- Configure distributed tracing with automatic instrumentation for incoming HTTP requests and outgoing dependencies.
   2. Metrics collection --- Collect framework-level HTTP metrics and custom application counters using OpenTelemetry's Meter API.
   3. Logging integration --- Connect ILogger to OpenTelemetry so every log entry carries TraceId and SpanId for trace-to-log correlation.
   4. Telemetry export and validation --- Configure exporters to emit telemetry and validate the instrumentation end-to-end.

   The Scenario

   You are a platform engineer at CarvedRock building a payment processing API. The operations team needs visibility into request latency, dependency failures, and error rates to maintain SLAs. Your task is to instrument the application with OpenTelemetry.

   The Application (Click to expand)

   This lab uses an ASP.NET Core 10 Web API that processes payments:

   • PaymentsController --- endpoints for processing payments, retrieving history, and health checks
   • PaymentProcessingService --- business logic where you add custom tracing and metrics
   • ExternalPaymentService --- simulated external gateway using HttpClient for dependency instrumentation
   • RequestLoggingMiddleware --- logs request details with trace context for correlation

   Familiarizing with the Program Structure (Click to expand)

   Key files:

   1. src/Configuration/TelemetrySettings.cs --- Telemetry constants (you complete this)
   2. src/Program.cs --- Tracing, metrics, logging, and exporter configuration (you complete this)
   3. src/Services/PaymentProcessingService.cs --- Custom ActivitySource instrumentation (you complete this)
   4. src/Controllers/PaymentsController.cs --- API controller (pre-built)
   5. src/Services/ExternalPaymentService.cs --- External dependency via HttpClient (pre-built)
   6. src/Middleware/RequestLoggingMiddleware.cs --- Request logging middleware (pre-built)
   7. tests/validate-telemetry.sh --- Validation script (you complete this)

   Build the solution using: dotnet build src/CarvedRockPayments.csproj

   Run with: dotnet run --project src/CarvedRockPayments.csproj

   The API starts on http://localhost:4200.

   Your Mission

   1. Define service identity constants that tag all telemetry data
   2. Configure the OpenTelemetry resource builder and tracing pipeline
   3. Add ASP.NET Core instrumentation for tracing and metrics
   4. Integrate OpenTelemetry with the logging pipeline for trace correlation
   5. Instrument outgoing HTTP dependencies and register custom trace sources
   6. Configure console exporters across all three pipelines
   7. Add custom ActivitySource spans with business attributes to payment processing
   8. Validate the complete instrumentation end-to-end

   Note: Complete tasks in order. Build frequently with dotnet build src/CarvedRockPayments.csproj to catch errors early.

   info> If you get stuck on a task, you can find solution files with the completed code for each task in the solution folder of your filetree.

2. Challenge

   Configuring the OpenTelemetry Foundation

   Understanding Resources and Tracing

   In OpenTelemetry, a Resource is a set of attributes that identify the entity producing telemetry --- typically service.name, service.version, and service.namespace. You build the resource once with ResourceBuilder and pass it to each pipeline so traces, metrics, and logs share the same identity. CreateDefault() includes SDK and runtime metadata; CreateEmpty() produces a blank resource.

   The tracing pipeline is registered via AddOpenTelemetry().WithTracing(). Inside it, you add instrumentation libraries that hook into framework pipelines to create spans automatically. AddAspNetCoreInstrumentation() creates a span per incoming HTTP request. SetResourceBuilder() on each pipeline ties telemetry data back to your service.

3. Challenge

   Collecting Metrics and Integrating Logging

   Understanding Metrics and Logging in OpenTelemetry

   The .WithMetrics() pipeline collects numerical measurements. AddAspNetCoreInstrumentation() captures request duration histograms, request counts, and active connections. AddMeter() registers a custom Meter so OpenTelemetry collects its instruments --- without registration, counter and histogram data is silently discarded.

   OpenTelemetry logging enhances ILogger by enriching each log entry with the active TraceId and SpanId. IncludeScopes preserves contextual data from logging scopes, and IncludeFormattedMessage renders the full message text rather than just the template.

4. Challenge

   Instrumenting Dependencies and Configuring Exporters

   Understanding Dependencies and Exporters

   AddHttpClientInstrumentation() hooks into HttpClient to create child spans for outgoing calls, linked to the parent via W3C trace context. This completes the request flow picture: incoming request, business logic, and downstream dependencies all appear in one trace.

   Registering a custom ActivitySource with AddSource() is required for OpenTelemetry to record spans from that source. Without registration, StartActivity() returns null.

   An exporter sends telemetry to a backend. The console exporter writes to stdout for development visibility. In production, you would swap in AddOtlpExporter() for backends like Jaeger or Grafana Tempo. Each pipeline needs its own exporter call.

5. Challenge

   Custom Instrumentation and End-to-End Validation

   Understanding Custom Instrumentation

   Automatic instrumentation captures HTTP boundaries but cannot see inside business logic. ActivitySource is the .NET API for creating custom spans. You declare it as a static readonly field, register its name with AddSource(), and call StartActivity() to create spans linked to the current trace context. Tags added via SetTag() become searchable attributes in tracing backends.

   StartActivity() returns null when no listener is registered (e.g., in unit tests), so use the null-conditional operator when calling SetTag().

6. Challenge

   Conclusion

   Congratulations on completing the Instrumenting an ASP.NET Core 10 App with OpenTelemetry lab! You have implemented a complete observability solution using the industry-standard OpenTelemetry framework.

   What You Have Accomplished

   1. Defined service identity constants — Centralized service.name, service.version, and service.namespace across all telemetry pipelines.
   2. Configured the resource builder and tracing pipeline — Built a resource with CreateDefault().AddService() and attached it to the tracing pipeline with SetResourceBuilder().
   3. Added tracing and metrics instrumentation — Configured AddAspNetCoreInstrumentation() on both pipelines and registered the custom Meter for payment counters and histograms.
   4. Integrated logging with trace context — Connected ILogger to OpenTelemetry with scope preservation, formatted messages, and log-level filtering.
   5. Instrumented outgoing dependencies — Added HttpClient instrumentation and registered the custom ActivitySource to complete the request flow picture.
   6. Configured console exporters — Added exporters to all three pipelines for development-time visibility.
   7. Added custom business tracing — Created ActivitySource spans with payment.id, payment.amount, and payment.currency tags for searchable business telemetry.
   8. Validated end-to-end — Ran a test suite confirming correlated traces, metrics, and logs across all code paths.

   Key Takeaways

   • Automatic instrumentation (AddAspNetCoreInstrumentation(), AddHttpClientInstrumentation()) captures HTTP boundaries without code changes.
   • Custom instrumentation (ActivitySource, Meter) fills the gaps for business-specific operations.
   • OpenTelemetry logging enriches ILogger with TraceId/SpanId automatically — no changes to existing log statements needed.
   • Exporters are pluggable: swap AddConsoleExporter() for AddOtlpExporter() when moving to production backends.

   Experiment Before You Go

   • Send multiple requests and observe unique TraceId values in the console
   • Add a second span for a "fraud check" operation in PaymentProcessingService
   • Create a new counter that tracks payment amounts by currency
   • Set activity?.SetStatus(ActivityStatusCode.Error) on payment failure and observe the span status
   • Examine how RequestLoggingMiddleware log entries include TraceId values matching trace spans