Guided: Logging and Monitoring in Microservices

Microservices architectures bring flexibility, scalability, and independent deployability, but they also introduce new operational challenges. In a traditional monolithic system, logs, metrics, and application state are often centralized and easier to trace. With microservices, the same visibility requires deliberate design and tooling. ---

Why Logging and Monitoring Are Critical

In a distributed system:

• A single request may involve multiple services
• Failures can be hidden within isolated components
• Identifying performance bottlenecks or bugs becomes harder without proper tracing

Without proper observability, you're essentially debugging in the dark.

Challenges in Microservices Observability

• Cross-service request tracing
  Requests span multiple services, making them difficult to follow without consistent context propagation.

• Silent or partial failures
  Failures may not surface clearly and can leave downstream systems in inconsistent states.

• Volatile infrastructure
  Containerized and auto-scaling services scatter logs across short-lived instances.

• Asynchronous communication
  Messaging systems delay visibility and make real-time debugging more complex.

• Sensitive data exposure
  Logs and metrics can unintentionally leak personal or confidential information if not carefully handled.

In this lab, you will walk through a series of steps to make a simple two-service system observable. Starting with minimal logs, you will progressively add:

• Structured logs with levels
• Propagate correlation IDs across services
• Log request metadata including latency
• Capture custom metrics for downstream calls
• Implement health checks for dependency reachability

If you get stuck on a task, you can view the solution in the solution folder in your Filetree, or click the Task Solution link at the bottom of each task after you've attempted it.

The system currently consists of two simple Spring Boot services working together:

order-service

• Runs on port 8081 and handles incoming order requests (product ID and quantity) and delegates stock updates to inventory-service.

inventory-service

• Runs on port 8082 and processes inventory updates and returns the remaining stock for the specified product.

Request Flow

A client request to the order-service indirectly triggers a request to inventory-service.

• order-service (retrieves productId)

• inventory-service (checks stock for productId)

• order-service (combines and returns result)

Next, you will start both the services and make a REST API call to place an order usibg order-service. This will help you observe the current logging behavior and identify what information is missing.

To begin, start the order-service and inventory-service in the first and second Terminal tabs respectively.

In first Terminal start order-service:

cd order-service
mvn spring-boot:run

In second Terminal start inventory-service:

cd inventory-service
mvn spring-boot:run

Now you will send a request to Order service for Product ID : P123 and Qty : 2 You should receive the following response in the third Terminal :

Inventory updated for product P123. New stock: 98

Failed to place order for product P12300: 400 : "Product not found: P12300"

You can observe the logs for both services.

Order Service ( First Terminal ) :

Calling inventory service for product id: P123 for URL: http://localhost:8082/api/inventory
Received response from inventory service: Inventory updated for product P123. New stock: 98
Calling inventory service for product id: P12300 for URL: http://localhost:8082/api/inventory
Error calling inventory service for product id: P12300 - 400 : "Product not found: P12300"

Inventory Service ( Second Terminal ) :

Reducing stock for productId P123
Inventory updated for product P123. New stock: 98
Reducing stock for productId P12300
Product ID P12300 not found in inventory.

Observability Gaps

While the services functionally work, there are significant blind spots when it comes to operations and debugging:

• No request tracking across services
  When an issue occurs, there's no way to link related requests between services to see what actually happened.

• Lack of timing information
  If the system is slow, there is no visibility into which service caused the delay or how long downstream calls took.

• No error monitoring
  Failures during communication with other services are not tracked, making it difficult to diagnose intermittent issues.

• Missing service health indicators
  There's no visibility into whether a service is up, degraded, or failing to reach its dependencies.

• Unstructured logs
  Logs exist, but are plain text with inconsistent formats, making them hard to query or aggregate across services.

This current state reflects a common starting point in microservices-based systems. In the following steps, you will address these gaps and gradually add capabilities that make the system observable, diagnosable, and production-ready.

Why Structured Logging?

By default, logs are plain text messages that may vary in format and structure across services. This makes it difficult to search, filter, or aggregate logs in production environments, especially when dealing with distributed microservices.

Structured logging adds consistent fields like timestamp, thread, log level, logger name, and request context (such as correlation ID). This makes logs machine-readable and helps tools like ELK, Loki, or Splunk parse them effectively.

Problem Without Structured Logs

• Logs are hard to query due to inconsistent formats
• It’s difficult to extract useful metadata (e.g., request path, response time)
• Correlation between logs across services is nearly impossible without a shared structure

Now you will configure the log pattern in logback-spring.xml file to show the below information in logs:

• Timestamp
• Thread name
• Log level
• Logger name (shortened to 36 characters) Similarly update the logback-spring.xml file for inventory-service. ---

Logging frameworks support multiple log levels to indicate the severity or purpose of each log entry:

| Level | Description | |----------|-----------------------------------------------------------------------------| | TRACE | Very fine-grained debugging information, typically not enabled in production | | DEBUG | Detailed application behavior useful during development or troubleshooting | | INFO | General runtime events (e.g., startup, key business actions) | | WARN | Unexpected situations that do not stop execution but may require attention | | ERROR | Errors that prevent normal flow or functionality |

Choosing the right log level helps developers filter logs efficiently in production systems. Both services currently log messages using only log.info. Next, you will enhance the logging by applying more appropriate log levels such as log.debug, log.warn, and log.error based on the context of each message. Similarly you will also update the InventoryController.java to have logging using appropriate log level. --- You can test by restarting the applications and running the curl command in the third Terminal.

curl -X POST http://localhost:8081/api/order -H "Content-Type: application/json" -d '{"productId": "P123", "quantity": 2}'

Example Output:

order-service

2025-09-02 18:42:20.834 [http-nio-8081-exec-1] INFO  c.p.order.controller.OrderController - Received response from inventory service: Inventory updated for product P123. New stock: 98

inventory-service

2025-09-02 18:42:20.822 [http-nio-8082-exec-1] INFO  c.p.i.controller.InventoryController - Inventory updated for product P123. New stock: 98

--- In this step, you configured structured logging and improved the quality of log messages by applying appropriate log levels like debug, warn, and error.

In the next step, you will introduce correlation IDs to make it possible to trace a single request across multiple microservices. This will help you link logs end-to-end, identify where delays or failures occur, and debug distributed workflows more effectively.

In a microservices architecture, a single client request may involve multiple service calls. Without a shared identifier across these calls, tracing the complete flow of a request becomes nearly impossible.

A correlation ID is a unique identifier generated for each incoming request and passed along when the request travels between services. By including this ID in your log entries, you can correlate logs from different services and trace the end-to-end flow of a request.

In this step, you will:

• Generate a correlation ID when a request enters the system

• Store the ID in the logging context (MDC) so it appears in all logs

• Update the log pattern in logback-spring.xml to include the correlation ID

• Propagate the ID to downstream services via HTTP headers

• Extract the ID in the receiving service and continue the trace Similarly update the logback-spring.xml for inventory-service. Next, you will update the request filter named OrderCorrelationIDFilter to populate the correlation id. Once the correlation ID is available, you have store it in two places:

• The MDC (Mapped Diagnostic Context) so it gets included automatically in all log statements.

• The HTTP response header, allowing clients or upstream systems to retrieve the ID for tracing. --- After the request is processed, you must remove the correlation ID from the MDC. This prevents the ID from accidentally leaking into unrelated log entries in thread pools or reused threads. ---

For simplicity, the similar changes for inventory-service are already done in the file InvCorrelationIdFilter.java . --- You can test by restarting the applications and running the curl command in the third Terminal.

curl -X POST http://localhost:8081/api/order -H "Content-Type: application/json" -d '{"productId": "P123", "quantity": 2}'

Example Output:

order-service

2025-09-02 18:48:36.882 [http-nio-8081-exec-1] INFO  c.p.order.controller.OrderController - cid=0941fbb3-ce38-458f-8db6-c633dc6e4c1f Received response from inventory service: Inventory updated for product P123. New stock: 98

inventory-service

2025-09-02 18:48:36.869 [http-nio-8082-exec-2] INFO  c.p.i.controller.InventoryController - cid=0453333e-0a69-46ab-be2e-6f25e918d139 Inventory updated for product P123. New stock: 98
``` In this step, you added correlation ID support to trace requests across services. The ID is logged, returned in the response, and cleaned up after each request. 

In the next step, you will log request method, path, status, and latency to improve visibility into request performance.

While correlation IDs help trace a request through services, they don’t provide enough detail about how each service handled the request.

Capturing request metadata like the HTTP method, path, response status, and time taken to process the request gives critical insight into the behavior and performance of your APIs.

This is especially useful for:

• Detecting slow or failing endpoints
• Understanding traffic patterns
• Troubleshooting user-reported issues

You will use the OrderRequestLoggingFilter to log the latency.

This filter captures metadata about each request and logs it when the request completes. Latency is calculated by measuring the time before and after the filter chain is executed. Perform the similar changes in InvRequestLoggingFilter.

You can test by restarting the applications and running the curl command in the third Terminal.

curl -X POST http://localhost:8081/api/order -H "Content-Type: application/json" -d '{"productId": "P123", "quantity": 2}'

Example Output:

order-service

2025-09-02 18:55:18.777 [http-nio-8081-exec-1] INFO  c.p.order.controller.OrderController - cid=35070021-2e91-492d-86ef-a3221a2dbaae Received response from inventory service: Inventory updated for product P123. New stock: 98
2025-09-02 18:55:18.785 [http-nio-8081-exec-1] INFO  c.p.o.f.OrderRequestLoggingFilter - cid=35070021-2e91-492d-86ef-a3221a2dbaae method=POST path=/api/order status=200 latencyMs=272 

inventory-service

2025-09-02 18:55:18.764 [http-nio-8082-exec-1] INFO  c.p.i.controller.InventoryController - cid=f1f015dc-85dd-4f57-9a15-e41ee80388e0 Inventory updated for product P123. New stock: 98
2025-09-02 18:55:18.773 [http-nio-8082-exec-1] INFO  c.p.i.f.InvRequestLoggingFilter - cid=f1f015dc-85dd-4f57-9a15-e41ee80388e0 method=POST path=/api/inventory status=200 latencyMs=77 
``` In this step, you logged method, path, status, and latency for each request to improve traceability.

Next, you will enable actuator endpoints and add custom metrics to track latency and failures more effectively.

Logging is helpful for diagnosing issues, but it’s not enough for tracking long-term performance, health, or trends. This is where metrics come in.

Metrics are numeric indicators collected over time , such as request counts, error rates, and response times and are ideal for alerting and dashboarding.

In Spring Boot, Micrometer is the de facto library for collecting application metrics, and it integrates seamlessly with monitoring systems like Prometheus, Grafana, and others.

In this step, you will:

• Enable monitoring endpoints using Spring Boot Actuator in application.properties

• Use Micrometer to record custom metrics in your service

• Track latency and failure counts for downstream calls After restarting the applications you can verify actuator endpoint by visiting:

• Order Service : http://localhost:8081/actuator

With actuator enabled, your service now exposes built-in system metrics. However, for business-specific insights, you need to instrument custom metrics.

In the last step, you logged request latency for individual requests. Now, using Micrometer, you'll track aggregated metrics like inventory call latency and failure counts, which are more useful for dashboards and alerts.

Next, you will enhance the order-service by adding a Timer to measure inventory call duration and a Counter to track how often those calls fail. Restart both services and invoke the order-service by sending multiple test orders:

curl -X POST http://localhost:8081/api/order -H "Content-Type: application/json" -d '{"productId": "P123", "quantity": 2}'

Then, open the following URLs in your browser or use curl to inspect the metrics:

{{localhost:8081}}/actuator/metrics
{{localhost:8081}}/actuator/metrics/inventory_latency
{{localhost:8081}}/actuator/metrics/inventory_failures

These endpoints show live metrics that you can graph or monitor using tools like Prometheus or Grafana.

Sample output

{{localhost:8081}}/actuator/metrics

Returns the list of all available metrics (built-in and custom):

{
  "names": [
    "jvm.memory.used",
    "jvm.gc.pause",
    "process.cpu.usage",
    "http.server.requests",
    "inventory_latency",
    "inventory_failures",
    "logback.events",
    "system.cpu.usage",
    ...
  ]
}

{{localhost:8081}}/actuator/metrics/inventory_latency

Returns measurement details for the custom timer inventory_latency:

{
  "name": "inventory_latency",
  "description": "Latency for inventory calls",
  "baseUnit": "seconds",
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 4.0
    },
    {
      "statistic": "TOTAL_TIME",
      "value": 6.82
    },
    {
      "statistic": "MAX",
      "value": 2.45
    }
  ],
  "availableTags": []
}

This shows that 4 calls were made (COUNT) with a total latency of ~6.82 seconds and a maximum single-call latency of 2.45 seconds.

Now send a request with invalid productId as P12300 and then check the inventory_failures metric.

curl -X POST http://localhost:8081/api/order -H "Content-Type: application/json" -d '{"productId": "P12300", "quantity": 2}'

{{localhost:8081}}/actuator/metrics/inventory_failures

Returns the failure counter for inventory calls:

{
  "name": "inventory_failures",
  "description": "Number of failures when calling inventory",
  "baseUnit": null,
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 2.0
    }
  ],
  "availableTags": []
}

This means 2 failed attempts have been recorded when calling the inventory service.

In this step, you enabled actuator endpoints and added custom Micrometer metrics to track latency and failures in inventory calls. These metrics help you monitor service performance over time.

In the next step, you will create a custom health indicator to check the availability of the inventory service and report it via the /actuator/health endpoint.

Spring Boot's Actuator provides a built-in /actuator/health endpoint that reports on application status. By default, it includes checks for components like the database, disk space, and readiness probes.

However, in a microservices environment, it’s just as important to verify the health of external dependencies, such as downstream services, APIs, or internal services your app relies on.

With a custom HealthIndicator, you can add your own logic to the /actuator/health report. This lets you programmatically check the reachability and status of critical dependencies like inventory-service. --- In this step, you will:

• Implement a custom HealthIndicator in order-service
• From order-service attempt to reach the inventory-service from within the health check and return Available or Unavailable. Next, if the inventory-service is not reachable, return Unavailable. --- Stop both the services and start only the order-service.

Open the page : {{localhost:8081}}/actuator/health

You wlil receive a output similar to below:

{
  "status": "DOWN",
  "components": {
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 212313567232,
        "free": 19724337152,
        "threshold": 10485760,
        "path": "\\logging-monitoring-microservices\\.",
        "exists": true
      }
    },
    "inventory": {
      "status": "DOWN",
      "details": {
        "inventory-service": "Unavailable"
      }
    },
    "ping": {
      "status": "UP"
    }
  }
}

Observe the inventory status:

  "inventory": {
    "status": "DOWN",
    "details": {
      "inventory-service": "Unavailable"
    }

Now once you test this endpoint again with inventory-service started you will get a response like:

    "inventory": {
      "status": "UP",
      "details": {
        "inventory-service": "Available"
      }

In this step, you added a custom health indicator to the order-service to check if the inventory-service is reachable. This helps surface dependency health through the /actuator/health endpoint, improving service reliability and visibility.

Summary of What You Implemented

Throughout this lab, you enhanced observability in a two-service microservices setup (order-service and inventory-service) by progressively implementing logging and monitoring techniques:

• Identified observability gaps in a basic microservices system
• Added structured logging with proper log levels (info, debug, warn, error)
• Implemented correlation IDs to trace requests across services
• Logged request method, path, status, and latency to capture runtime behavior
• Enabled actuator endpoints and added custom metrics using Micrometer
• Created a custom health indicator to monitor dependency availability

These improvements help make the system more traceable, resilient, and easier to diagnose during failures.

Next Steps and Recommendations

Here’s how you can extend your observability stack further:

• Centralized Logging
  Use tools like ELK (Elasticsearch, Logstash, Kibana) or Loki + Grafana to collect and visualize logs across services.

• Distributed Tracing
  Integrate with tools like OpenTelemetry, Zipkin, or Jaeger to trace requests across service boundaries with timing information.

• Alerting and Dashboards
  Use Prometheus + Grafana to build real-time dashboards and configure alerts based on metrics like latency, error rates, or service health.

You now have a solid foundation in implementing logging and monitoring in Spring Boot microservices!