Guided: Implement OpenTelemetry for Python Observability

In this lab, you'll be working with the backend API for a bicycle parts manufacturer. The API is built using Python's FastAPI library and provides routes to manage various bike parts as they are removed from or added to the company's inventory.

Start by getting familiar with the code for the FastAPI application.

Review the main.py file. Make a note of the routes the application exposes for requests and the HTTP port the API server will use to listen for requests.

The routes include:

• /parts: This route returns all the parts in the database.
• /part/{PART_NUMBER}: This route returns or updates details for a specific part.

The lab includes a virtual environment directory named venv. FastAPI applications typically use an additional layer of infrastructure to run properly. The bike parts API depends on Uvicorn, a Python web server that can be used to run FastAPI applications with minimal configuration and maximum performance.

Uvicorn supports the Asynchronous Server Gateway Interface (ASGI) standard, which ensures compatibility with a wide range of Python web frameworks, including FastAPI. Now that the lab environment is configured to run the API for the bike parts manager, the next step is to configure OpenTelemetry.

With the FastAPI application in place, the next step is to set up OpenTelemetry to collect observability data. OpenTelemetry provides distributed tracing and metrics capabilities, allowing you to monitor application performance and diagnose issues effectively.

The OpenTelemetry SDK consists of multiple components, including the core SDK, instrumentors, and exporters. This lab requires the following OpenTelemetry dependencies:

• opentelemetry-sdk: The core OpenTelemetry SDK
• opentelemetry-instrumentation-fastapi: Library for instrumenting FastAPI applications
• opentelemetry-exporter-otlp-proto-grpc: Library for exporting telemetry data using the OpenTelemetry Protocol (OTLP)

In the next task, you will install each of these dependencies using pip. Before integrating OpenTelemetry into the FastAPI application, it's important to verify that the SDK is correctly installed and understand how OpenTelemetry applications are instrumented.

Open and review the file examples/2_opentelemetry_sdk.py, which demonstrates a basic OpenTelemetry setup:

1. It initializes a TracerProvider and assigns a service name.
2. It configures a ConsoleSpanExporter to print trace data to the terminal.
3. It registers a SimpleSpanProcessor to process spans.
4. It generates a test span with a simple print statement. You have confirmed that OpenTelemetry is installed and observed how an application is instrumented with OpenTelemetry. Now apply these concepts in the next step by instrumenting the FastAPI application.

Instrumentation enables OpenTelemetry to capture trace data for incoming requests, helping you monitor request flow and performance.

Now that you've seen basic instrumentation, use a similar approach to to instrument the FastAPI application by updating the main.py file to include OpenTelemetry tracing.

Modify main.py to include OpenTelemetry instrumentation. At this point, the FastAPI application is instrumented to allow traces to be reported for incoming requests. The next step is to collect custom metrics.

Now that your FastAPI application is instrumented for tracing, the next step is to collect custom metrics. While traces help monitor request flows and performance bottlenecks, metrics provide quantitative insights into application behavior over time.

Before modifying the FastAPI application, review how a custom metric is defined and collected using OpenTelemetry. Now that you've seen how a counter metric works, apply the concept to the FastAPI application. Use these steps to observe the custom metric:

1. In the first Terminal, start the application.

2. In the second Terminal, run the following curl command a few times to update the "Brake Lever" inventory:

   curl -X 'PUT' 'http://127.0.0.1:9000/part/1?count=50'
   

3. Check the first Terminal for metric data to be printed every 15 seconds.

4. Stop the application using Control + C.

5. In the final telemetry output, look for the metrics report for update_count. The value for "part_name": "Brake Lever" should match the number of times you ran curl. In this step, you explored how OpenTelemetry collects custom metrics and applied that knowledge to the FastAPI application.

While exporting metrics to the console is useful for quick validation, visualizing OpenTelemetry data in a dedicated observability tool provides deeper insights. A structured interface allows you to explore traces and metrics more effectively, making it easier to analyze application performance and identify issues.

In the next step, you'll configure the FastAPI application to export telemetry data to Jaeger, a powerful tool for visualizing and analyzing trace data.

Now that the FastAPI application is collecting traces and custom metrics, the next step is to export telemetry data to Jaeger, a distributed tracing platform.

Jaeger allows you to visualize traces and diagnose performance bottlenecks in a structured way. In addition, the graphical interface makes working with traces much easier compared to using the Terminal. ### View the Example Traces in Jaeger

1. Open the Jaeger UI by selecting the lab's Web Browser tab and ensure the URL is localhost:16686. Then select the Open in a new browser tab button.

2. In the Jaeger UI, look for Service and select example-export from the drop-down menu. Then select the button labeled Find Traces.

   The display should update with the following:

   • Dots on a graph with axes for Duration and Time
   • A list of traces below the graph

3. Select one of the dots or one of the items from the list. This opens a timeline view for the trace and the spans it includes.

4. Near the top, left-side of the view, make a note of the information displayed including:

   1. Trace Start
   2. Duration
   3. Services
   4. Depth
   5. Total Spans

5. Under Service & Operation, select the trace name, example-export example-span. Under the newly displayed text example-span, expand the items under Tags: and Process:.

6. Using the drop-down menu on the far right side of the interface, experiment with different views of the trace by selecting:

   1. Trace Graph
   2. Trace Statistics
   3. Trace Spans Table
   4. Trace Flamegraph
   5. Trace JSON (will open in a new tab)
   6. Trace JSON (unadjusted) (will open in a new tab)

Update the main.py script to send traces to Jaeger.

Send Traces to Jaeger from the FastAPI Application

After main.py is configured to export OpenTelemetry traces to Jaeger, start the application and verify that traces are being sent.

1. Select the file tab for main.py.
2. Select the first Terminal and then select the Run button.
3. Confirm the application starts and is running on http://127.0.0.1:9000.
4. In the second Terminal, use curl to send several requests to http://localhost:9000/parts.
5. Go to the Web Browser tab where the Jaeger interface is open and select Search in the top navigation bar.
6. Under Service, select parts-manager and then select Find Traces. Confirm the display updates with traces for the Parts Manager API. Select one of the traces from the list.
7. Expand the various parts of the timeline view under Service & Operation. In this step, you configured the FastAPI application to export traces to Jaeger using the OTLP exporter. With traces now visible in Jaeger's UI, you can analyze request flow, detect performance bottlenecks, and better understand how the application behaves under different conditions.

Exporting traces to a structured observability tool provides a significant advantage over viewing raw logs, allowing for deeper insights into latency, dependencies, and request patterns.

The next step is to use Jaegar to analyze telemetry data and optimize the application based on the collected traces.

Now that the FastAPI application is exporting traces to Jaeger, it's time to analyze telemetry data and make optimizations based on what you observe. In this step, you will identify a performance bottleneck, resolve the issue, and verify that the application performance has improved.

Generate Traces by Updating a Part

Send multiple PUT requests to update the inventory count for the Brake Lever part (ID: 1).

These requests will generate traces in Jaeger, allowing you to identify a performance issue.

Analyze Traces in Jaeger

1. Navigate to the Jaeger UI to inspect the traces you just generated.
2. Select parts-manager from the service dropdown.
3. Click Find Traces.
4. Open one of the traces for the PUT /part/1 request.
5. Notice that the PUT request duration is between 1 and 3 seconds.
6. Review the details on the Jaegar interface to observe the request's Duration. The file parts_manager.py contains the backend logic for managing parts.

A time.sleep() call has been added to the backend to simulate network latency in the application.

Optimize the application by removing the artificial latency from the backend script.

Confirm Improved Performance

1. If the FastAPI application is still running in the first Terminal, stop it by typing Control + C.

2. Restart the application by pressing the Run button.

3. In the second Terminal, repeat the command to update a part:

   curl -X PUT "http://127.0.0.1:9000/part/1?count=10"
   

4. Return to the Jaeger UI and select:

   1. Search
   2. Service: parts-manager
   3. Operation: PUT /part/{part_id}
   4. Find Traces

5. Select the checkbox next to two traces, one that was created before the code was updated and another that was created after the update.

   Tip: Use the timestamp on the far right side of each trace to help determine when the trace was created.

6. Select Compare Traces.

7. On the updated display, compare the duration for the traces labeled as A and B. The trace representing the updated parts-manager.py should have a much shorter duration than the previous trace. In this step, you analyzed trace data using Jaeger to identify a bottleneck in the FastAPI application. By removing the delay, you improved the performance of the application's update operation.