Guided: Monitoring and Logging in Node.js Microservices

Objective

Set up a basic Express server, which will serve as the foundational environment for integrating advanced monitoring and logging features. ### 1: Project Initialization (offline setup)

info> Tests in the tests folder are for verification of the tasks. Do not edit these. You should not need to edit the package.json file either.

This Code Lab already includes a package.json file with the dependencies you will need. But if you were starting a project from scratch in your local environment, you would use npm init to create a package.json file with default values, it looks something like this:

{
  "name": "my-express-server",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

The package.json file is crucial as it manages project dependencies, scripts, and versions. This file will be updated as you add dependencies and scripts to your project. You can also modify the values manually if needed. We'll use the ES6 module syntax in this project. To enable this, you need to add the following line to the package.json file.

{
  "type": "module"
}
``` ### 2: Install Express (offline setup)

The Code Lab environment already comes with Express installed (you can take a look at the `package.json` file). But if you were running this in a local project, you would install Express as a project dependency using npm.
```sh
npm install [email protected] --save-exact

This command installs Express and saves it as a project dependency in the package.json file. Using --save-exact pins the dependency to version 4.18.2, ensuring that all installations of this project use the same version, which enhances reproducibility and reduces bugs related to version discrepancies.

About Express

Express is a fast, unopinionated, minimalist web framework for Node.js. It simplifies routing, middleware integration, and many other web server features, making it a popular choice for web applications.

info> The solution folder contains the code as it should look after you're done with the codelab. If you are stuck at some point, check the solution. 1. Run the Express server using the following command. sh node index.js You can also use the start script defined in the package.json file: sh npm run start 2. Open your browser and navigate to http://localhost:8000. You should see a message indicating that the server is running. But since you haven't set up any routes yet, you'll see an error message: Cannot GET /. 1. Restart the server by stopping the current process (press Ctrl + C in the terminal) and running the node index.js command again.

2. Refresh the browser at http://localhost:8000/hello. You should now see the message Hello World!. Congratulations! You've successfully set up a basic Express server that listens for incoming requests and responds with a simple message. This foundational environment will serve as the basis for integrating more advanced features in the upcoming steps.

Objective

Integrate the Winston library into the Express server to implement structured logging. This module will teach you how to configure different log levels and transports, ensuring logs are both informative and useful for debugging and monitoring.

Introduction

Logging is a critical aspect of any application. It allows developers to track the flow of their application, understand the root causes of issues, and maintain a record of application states and events. Structured logging is a way of formatting logs consistently and predictably, enhancing their utility in automated analysis tools. In this module, you will configure Winston to manage your logs effectively. ### 1: Install Winston (offline setup) Again, this is not necessary in this Code Lab, but in an existing project you would install Winston as a project dependency using npm.

npm install [email protected] --save-exact

This command installs Winston and saves it as a project dependency in the package.json file. As before, use --save-exact to pin the dependency to version 3.13.0, to ensure consistency.

About Winston

Winston is a versatile logging library for Node.js applications. It provides a simple and consistent API for logging information at different levels, such as info, warn, error, and debug. Winston also supports multiple transports, allowing logs to be stored in various locations, such as the console, files, databases, or third-party services.

4: Test the Logging Functionality

1. Run the Express server using the following command:

node index.js

or run it using the npm run start script. Notice how the console message indicating that the server is listening on port 8000 is now JSON.

2. Use the Web Browser tab and navigate to http://localhost:8000. You should see the "Hello World!" message from the previous module. Check the terminal where the server is running to see the JSON log messages generated by Winston.

3. Perform additional requests to the server, such as refreshing the page or navigating to other routes. Observe how Winston logs the incoming requests and the server's responses.

4. Experiment with different log levels (e.g. debug, warn, error) and formats in the Winston configuration and the app.js file, to understand how they affect the output of the logs. Remember to reset these files to how they were when they passed the tests before proceeding.

Conclusion

Congratulations! You've successfully integrated Winston into your Express server to implement structured logging. By configuring different log levels and transports, you can manage logs effectively and gain insights into your application's behavior. For example, in a production environment you could use tools such as Filebeat or the Winston Elasticsearch transport to send the logs from all instances of your microservices to an ELK cluster for better visualization and analysis. Using structured JSON format is crucial for efficient querying of these logs.

This foundational knowledge will help you build more robust and maintainable applications in the future.

Objective

In this module, you will integrate Morgan, a popular HTTP request middleware logger for Node.js, with their Express server. This setup will automatically log details of all incoming HTTP requests, enhancing visibility into server traffic and aiding in troubleshooting and monitoring.

Introduction

Understanding the flow of HTTP requests through your server is crucial for both development and production diagnostics. Morgan is a middleware for Express that simplifies logging of every HTTP request, including method, path, response status, and response time, which are vital for real-time monitoring and historical analysis.

1: Install Morgan (offline setup)

As with the other dependencies, you don't need to run this in the Code Lab, but for a local project you woudl install Morgan as a project dependency using npm.

npm install [email protected] --save-exact

This should be familiar by now, if not, refer to the previous modules for a detailed explanation.

About Morgan

Morgan is a popular HTTP request logger middleware for Node.js. It provides a simple yet powerful way to log incoming requests, including details such as the request method, URL, response status, and response time. Morgan supports various logging formats and can be customized to suit different requirements.

2: Integrate Morgan with Express

1. Open the app.js file and add the following code to import Morgan and use it as middleware in the Express server.

import morgan from 'morgan'

And after the const app = express() line:

// Choose 'combined' for detailed Apache-style logs, or 'tiny' for minimal output
app.use(morgan('combined'))

This code imports Morgan and adds it as middleware to the Express application. The morgan function takes a string parameter to specify the log format. The 'combined' format provides detailed logs similar to Apache logs (e.g. Client IP, request date, HTTP method, URL, HTTP version, response status, response length and response time), while 'tiny' offers a more concise output.

2. Remove the middleware function that logs incoming requests, if you added it in a previous module. Morgan will handle this functionality more effectively.

3. Restart the app using node index.js or npm run start and make some requests to localhost:8000/hello and other endpoints, and check the console logs.

4. Experiment with different log formats by changing the parameter passed to the morgan function. For example, try using 'tiny', 'short' or 'dev' instead of 'combined' to see the difference in log output.

This is how simple it is to set up morgan. But the real power comes when you combine it with winston, as you'll do next. ### 4: Test the Morgan Logger

1. Restart the app and make some requests using the Web Browser. Verify that logs are now sent to the console and the combined.log file using JSON. ## Conclusion

In this module, you learned how to integrate Morgan with an Express server to log incoming HTTP requests. Morgan provides detailed logs that can be customized to suit different requirements, making it a valuable tool for monitoring server traffic and diagnosing issues. By leveraging Morgan's capabilities, you can gain insights into your server's performance and improve its reliability and security.

Objective

In this module, you will enhance your Express server by setting up error handling that logs errors with your Winston logger. This setup will improve the robustness of the application by enabling quick diagnosis and resolution of issues.

Introduction

Error handling is a critical aspect of application development. In Node.js, unhandled errors can cause the server to crash and lead to downtime. Proper error handling combined with detailed error logging helps maintain application stability and aids in troubleshooting by providing insights into what went wrong and where. ## About Error Handling in Express

Express provides a robust error handling mechanism through middleware functions. These functions can catch errors thrown during request processing and respond with appropriate error messages. By centralizing error handling in middleware, you can ensure consistent error responses across your application. It's essential to log errors effectively to understand the root cause of issues and take corrective actions promptly. Logging errors with detailed information, including stack traces, request context, and metadata, can help in diagnosing and resolving problems efficiently. In this module, you'll integrate error handling middleware with Express and configure it to log errors using Winston. This setup will enable you to capture and log errors effectively, enhancing the application's reliability and maintainability.

3: Test the Error Handler

1. Restart the service using npm run start or node index.js
2. Make a request to http://localhost:8000/error using the Web Browser
3. Verify that now you get a JSON message with the exception in the logs and the combined file ## Conclusion You have now seen how combining Express middleware with Winston can improve the maintainability of your Node.js microservices, by making sure errors in the application are correctly logged to a file, including useful information for debugging and trouble shooting, such as detailed stack traces.

Objective

In this step, you will enhance your microservice’s observability by integrating Prometheus metrics. Observability is key in a microservices architecture, especially for internal monitoring, as it helps you understand the state of your systems. By exposing various metrics such as HTTP request durations and error rates, you can gain insights into your application's performance and reliability.

Introduction

Prometheus is an open-source monitoring and alerting toolkit that collects and stores time-series data. It provides a powerful query language, PromQL, for analyzing metrics and generating alerts based on predefined rules. By instrumenting your microservices with Prometheus metrics, you can monitor various aspects of your application, such as request rates, latencies, error rates, and resource usage.

1: Install Prometheus Client (offline setup)

1. As before, this is not required for the Code Lab, but in your local environment you would install the prom-client package, which is a Prometheus client for Node.js. This package will allow you to create various metrics within your application.

npm install [email protected] --save-exact

As before, use the --save-exact flag for fixing the version.

About Prometheus

Prometheus is an open-source monitoring and alerting system that collects metrics from various sources, stores them in a time-series database, and provides a powerful query language for analyzing the data. Prometheus uses a pull-based model where it scrapes metrics from targets (e.g., applications, services) at regular intervals. It supports multiple exporters and client libraries for instrumenting applications and exposing metrics in a standardized format. ### 3: Test the Metrics Endpoint

1. Restart the application
2. Make a request to the /metrics endpoint using the Web Browser.
3. Verify that the page contains the metrics in prometheus format, and find some default metrics, and the custom ones you defined (text color might be white so make sure you select the text with the mouse or use Ctr+F to find metrics)
4. Make some requests to the /hello endpoint and see how the metrics change if you query the /metrics endpoint again
5. You can define a helper sleep function if you want to add more variation to the request duration:

/* Helper function to simulate a delay */
async function sleep (ms) {
  return new Promise(resolve => setTimeout(resolve, ms))
}

And in the /hello route, before calling res.send():

await sleep(Math.random() * 30)

6. See how this affects the results in the metrics. ## Conclusion You have now instrumented your Node.js microservice to expose some default and custom metrics in prometheus format. You haven't configured prometheus, as it runs as a separate application, but after you configure it to scrape your microservice, you should be able to see these metrics, and also combine it with tools such as Grafana to visualize them.