Guided: Build a Real-time Dashboard with Node.js and MongoDB

Welcome to the Guided: Build a Real-time Dashboard with Node.js and MongoDB Lab

You have been given a feature rich meal planning application built using Node.js, Express, React, and MongoDB. The meal planning application supports user login and registration in addition to create, read, update, and delete (CRUD) operations for its data collections: recipes, households, and meal plans.

In this scenario, a user would like to see changes made to the meal plan dashboard they are viewing in real time without refreshing the page. Changes can be made to a meal plan by other users who are part of the meal plan’s household, through database operations via the terminal, or by the same user in another window.

In order to support real-time data updates, you will need to incorporate Server-sent Events (SSE) and MongoDB Change Streams on the backend. You will need to receive multiple responses to the initial request on the frontend using a browser interface called EventSource.

This lab provides a hands-on environment to help you practice some of the concepts learned in the Server-sent Events and WebSockets in Node.js course.

You are encouraged to reference the course if you have any questions regarding the concepts covered in this lab. Information necessary to complete the lab will be given at a high level. This lab will not cover WebSockets to handle bidirectional communication (client to server and server to client), but this lab will review primary differences between Server-sent Events and WebSockets.

A basic understanding of Node.js, Express, and routes using these technologies will be useful to complete this lab. This application's database is MongoDB, and the application uses mongoose. Familiarity with these technologies will help you understand data operations and how MongoDB Change Streams work. However, details around MongoDB Change Streams will be provided in this lab. The frontend is a React application, and you will be responsible to add an EventSource to a useEffect hook on a React component. Specific knowledge around any of these technologies that is required to complete this lab will be provided.

Meal Planning Application Details and Start Instructions

Start the application by starting the client in one terminal and starting the server in another terminal. In one terminal, start the client by following the directions below:

1. Change directories to the client directory.
   • Run cd client.
2. Run npm run start to start the client.

In another terminal, start the server by following the directions below:

1. Change directories to the server directory.
   • Run cd server.
2. Run npm run dev to start the server.

Once the client and server are running, the application can be seen in the Web Browser tab if you visit http://localhost:3000 or by clicking the following link: {{localhost:3000}}. Take some time to start and visit the application now.

When the application has been started, you will see the photo below. You can login with the seeded user “Jon Doe”. His email address is [email protected] and his password is jonpassword. Luckily this is not production and you don't need to be concerned about the complexity of Jon’s password.

Once logged in, you will see the recipes list. Visit Jon's households and view the “Doe’s Household”’s meal plan by following the photos below.

The meal planning application you are provided with has been seeded with recipes, households, and users. Every household is given an empty weekly meal plan to start. You can add recipes to the meal plan by assigning a recipe to a specific day and meal. This is done by logging in, accessing a household’s meal plan, and filling out the Add Recipe form. You can remove recipes by deleting them from the meal plan.

Helpful Tips While Completing This Lab

There is a solution directory that you can refer to if you are stuck or want to check your implementations at any time. Keep in mind that the solution provided in this directory is not the only solution, so it is acceptable for you to have a different solution so long as the application functions as intended.

All dependencies you will need to complete this lab have already been installed for you using Node Package Manager, npm. If you are curious about what these dependencies are, feel free to look at the package.json file in the server and client directories. You will learn about any dependency that are relevant to this lab as you complete it.

Just a reminder, you can stop running any foreground process in the Terminal with the command, Ctrl C.

What are Server-sent Events (SSE)?

Server-Sent Events (SSE) is a one-way communication method where the server pushes updates to the client over an HTTP connection. How it Works:

• The server keeps an open connection with the client.
• Updates are sent as events over this connection when MongoDB detects changes using MongoDB Change Streams
• It's lightweight compared to WebSockets and better suited for simple, one-way real-time communication.

What is Event Source

EventSource is a built-in JavaScript API that allows a client (typically a web browser) to receive real-time updates from a server using Server-Sent Events (SSE). Strengths of using EventSource include:

• One-way communication (server to client only).
• Automatic reconnection in case of disconnection.
• Text-based messages, usually in JSON format.
• Lightweight alternative to WebSockets when only server-to-client updates are needed.
• Uses standard HTTP protocol, making it firewall and proxy-friendly.

Use EventSource to Create an SSE Request for Real-time Data Updates

In this step, you will configure the meal plan frontend to connect with an SSE server endpoint and process incoming SSE events to display updated meal plans. Since the SSE endpoint will not be implemented until a future step, you will not be able to see the frontend’s EventSource object receive responses yet. However, you will be able to see the frontend fire a request to the future SSE endpoint by the end of this step. This step is half of the work necessary to accomplish the first bullet point for how SSE works as it will mean the client makes a request to the server that the server can keep open and the client will be capable of updating data that is being displayed as it parses future responses from the server.

In the MealPlanPage component, found in client/src/pages/mealplan.jsx, locate the connectSSE lambda function inside of the useEffect hook. Inside the connectSSE method, follow the steps below to setup an SSE connection and parse response data for real-time data updates on the frontend.

1. Instantiate a new EventSource object to open a persistent connection with an SSE server endpoint located at /api/mealplans/updates/${id}. id is already provided for you at the top of the MealPlanPage component. Store this new EventSource object in the variable eventSource that is already defined for you at the top of the useEffect hook.
2. Set eventSource’s onmessage event handler property to be a function that takes an event as its parameter.
3. Inside the body of the onmessage function, add a try catch block that catches an error and logs it to the console. If an error is encountered, you will know that a problem was encountered when parsing the SSE event’s data.
4. Inside the try block, store JSON.parse(event.data) in a constant variable called, updatedMealPlan. In a future step, you will write the SSE server endpoint and it will set the response’s data to be the updated meal plan object.
5. Next, add a console log to log the updated meal plan for future verification.
6. Lastly, use the setMealPlan state setter to set mealPlan to updatedMealPlan.

const connectSSE = () => {
  // Setup SSE connection for real-time updates
  eventSource = new EventSource(`/api/mealplans/updates/${id}`); //Instruction 1

  eventSource.onmessage = (event) => {
    try {
      const updatedMealPlan = JSON.parse(event.data);
      console.log("Received meal plan update:", updatedMealPlan);
      setMealplan(updatedMealPlan);
    } catch (error) {
      console.error("Error parsing SSE data:", error);
    }
  };
};

Verify the SSE Request is Being Sent

With connectSSE partly implemented, you can now verify that your client starts a request with the SSE server endpoint.

1. Visit the application in the Web Browser. It will be easier to complete the latter steps and open Developer Tools if you open the Web Browser in a new browser tab.
2. Login as Jon and go to a meal plan.
3. Open Developer Tools. Developer tools are opened by right clicking in the window using your mouse and clicking Inspect.
4. Go to the Network tab in Developer Tools and you should see two pending request to ”/api/mealplans/updates/${id}” like the photo below.

Gracefully Handle EventSource Errors and Reconnect to SSE Endpoint

Now that you have an eventSource, it is important to also specify what to do when the connection cannot be opened. The error event of the EventSource API is fired when a connection with an event source fails to be opened. In this step, you will set an event handler property for the error event (like you did for the message event in the previous step). You will be responsible for closing eventSource and reconnecting to the SSE server endpoint after five seconds.

1. In the connectSSE method, Set eventSource’s onerror event handler property to be a function that takes an error as its parameter.
2. In the body of the onerror function, log the error to the console with a message indicating that you will be reconnecting to the SSE server endpoint.
3. Add eventSource.close() to the onerror method to close the open EventSource object.
4. Use Javascript’s setTimeout method to execute connectSSE after five seconds (5000 milliseconds). This will call the connectSSE method you have implemented over the past two steps after five seconds and that will reconnect the SSE server endpoint and client.

const connectSSE = () => {
  // Setup SSE connection for real-time updates
  eventSource = new EventSource(`/api/mealplans/updates/${id}`);

  // Handle and Parse SSE Responses
  eventSource.onmessage = (event) => {
    try {
      const updatedMealPlan = JSON.parse(event.data);
      console.log("Received meal plan update:", updatedMealPlan);
      setMealplan(updatedMealPlan);
    } catch (error) {
      console.error("Error parsing SSE data:", error);
    }
  };

  //Gracefully Handle Event Source Errors and Reconnect
  eventSource.onerror = (error) => {
    console.error(
      "EventSource failed. Reconnecting... The error was: ",
      error
    );
    eventSource.close();
    setTimeout(connectSSE, 5000); // Reconnect after 5 seconds
  };
};

Close the SSE Request When the Component Unmounts

In React, a component will unmount for a variety of reasons including to free up memory and prevent memory leaks. This means that if the page is left open for long periods of time, the component may unmount and remount as the user interacts with the page and takes breaks. You want to close the component’s eventSource whenever the component unmounts to prevent the EventSource object from listening for updates from the server and attempting to update an unmounted component.

In this step, you will close the eventSource connection when the MealPlanPage component unmounts.

1. The correct way to perform cleanup actions when a component unmounts or when the dependencies of the effect change is to return a function from the useEffect. The function returned from the useEffect will be ran before the associated component unmounts and before the useEffect is reran due to a change in the useEffect's dependencies list. At the bottom of the useEffect in the MealPlanPage component, return a lambda function that takes no parameters and has an empty body.
2. Inside the body of the function you made above, run eventSource.close() if eventSource is defined.

useEffect(() => {
  let eventSource;

  const connectSSE = () => {
    // Setup SSE connection for real-time updates
    eventSource = new EventSource(`/api/mealplans/updates/${id}`);

    // Handle and Parse SSE Responses
    eventSource.onmessage = (event) => {
      try {
        const updatedMealPlan = JSON.parse(event.data);
        console.log("Received meal plan update:", updatedMealPlan);
        setMealplan(updatedMealPlan);
      } catch (error) {
        console.error("Error parsing SSE data:", error);
      }
    };

    //Gracefully Handle Event Source Errors and Reconnect
    eventSource.onerror = (error) => {
      console.error(
        "EventSource failed. Reconnecting... The error was: ",
        error
      );
      eventSource.close();
      setTimeout(connectSSE, 5000); // Reconnect after 5 seconds
    };
  };

  const fetchMealplan = async () => {
    const receivedMealplan = await getData(`/api/mealplans/${id}`);
    setMealplan(receivedMealplan);
    setmealplanLoading(false);
  };

  const fetchRecipes = async () => {
    const allRecipes = await getData(`/api/recipes`);
    setRecipes(allRecipes);
    setLoading(false);
  };

  fetchRecipes();
  fetchMealplan();

  connectSSE();

  // Close the Event Source Connection when the Component Unmounts
  return () => {
    if (eventSource) {
      eventSource.close();
    }
  };
}, [id]);

This completes what is necessary to setup the connection to the SSE server endpoint from the frontend and process the multiple SSE responses as they come in. In the next step, you will start implementing the SSE server endpoint to actually send multiple responses as MongoDB Change Streams detects data changes.

Verify You Only See One Pending SSE Request

Now that connectSSE is fully implemented, you will only see one live pending request to the server. Closing the eventSource object with the component unmounts prevents duplicate requests.

1. Visit the application in the Web Browser in a new browser tab.
2. Login as Jon.
3. Go to a meal plan.
4. Open Developer Tools.

Verify using the Network tab that you only see one pending request to ”/api/mealplans/updates/${id}” like the photo below.

SSE Response Headers

In this step, you will begin implementing the mealplans/updates/:id SSE server endpoint in server/routes/mealplans.js. The endpoint will be responsible for responding to the SSE request whenever a change is detected to a meal plan matching the id url param. The response will contain the updated meal plan as its data. Start implementing this endpoint by specifying the response headers.

1. A SSE server endpoint will set the response’s Content-Type to ”text/event-stream”. This way, the client knows it is an event stream it will be parsing when when it receives the response. Execute res.setHeader("Content-Type", "text/event-stream")after theconsole.logand before thetrycatchblock to set this header. Use this function to set the next two headers.
2. Set the Cache-Control header to ”no-cache”. This is standard with SSE responses to ensure that the SSE connection remains fresh and receives real-time updates from the server, rather than serving potentially outdated data from caches (such as browser cache, proxy servers, or intermediate caching layers).
3. Set the Connection header to ”keep-alive”. This is necessary with SSE responses to prevent the client from closing the request once it receives a response. This is how the server can continue to send multiple responses to the same request.
4. Call res.flushHeaders(); after setting the headers above. This will send the headers of the response to the client without waiting for the body to be complete. This is helpful in preventing time outs which would cause the connection to break. However, it is not helpful or necessary for every implementation of SSE.

router.get("/updates/:id", authorizeRoute, async (req, res) => {
  const mealPlanId = req.params.id;
  const id = ObjectId.createFromHexString(mealPlanId);

  console.log(`Client connected for meal plan updates: ${mealPlanId}`);

  // SSE Response Headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  try {
  } catch (err) {
    res.status(500).send("Internal Server Error");
  }
});

Note: Whenever you make a change to server/routes/mealplan.js the server hot reloads using nodemon. This means the session will be reset and you will need to login again to the application in the Web Browser after any change.

What are MongoDB Change Streams?

MongoDB Change Streams is a feature that provides real-time notifications of changes (inserts, updates, deletes, etc.) happening in a MongoDB collection, database, or an entire deployment.

MongoDB Change Streams can be created by calling watch on the respective Mongoose model. The watch() method in MongoDB's Change Streams API, as used in Mongoose (or the MongoDB Node.js driver), accepts an aggregation pipeline as its first parameter. This pipeline allows you to filter and process the stream of changes, ensuring that only relevant events are sent to your application. This way you only respond to the SSE request when a document of interest changes.

This creation of a MongoDB Change Stream looks something like:

MongooseModel.watch([
  {
    $match: {
      "documentKey._id": id,
    },
  },
]);

If nothing was passed to watch, the change stream would simply kick off anything changed within the collection it is watching.

MongoDB Change Streams and How to Send Multiple Responses with SSE

Now, you will be creating a MongoDB Change Stream on a document that matches id from the url parameters. Then, you will use the MongoDB Change Stream to perform a callback on any change to the document being watched. In the callback, you will write a response to the SSE client request with the updated meal plan as data. Lastly, you will report any errors to the console.

1. Inside the try block, create a MongoDB Change Stream using the Mongoose MealPlanModel already created for you. Pass a parameter to the watch function so that the change stream is only triggered when the "documentKey._id" matches the ObjectId stored in id at the top of the endpoint. Assign this change stream to a constant variable called changeStream.
2. Underneath the instantiation of changeStream, write changeStream.on(“change, async (change) => { }); to list for changes in the MongoDB collection document (in specific update or delete). Whenever a change occurs, the callback function you are about to implement will be executed. The change parameter contains the details of the database operation that triggered the event.
3. Inside the on change callback function, log change and the mealPlanId it refers to.
4. Next, in the callback function, find the meal plan and await the result. Store the result in a constant variable called, result. This will be done with the following line: const result = await MealPlanModel.findById(id);.
5. Use the helper method provided, userCanSeeMealPlan, to determine if the user with the active session can preview that meal plan. If the user can see the meal plan, write to the response a string that matches: data: ${JSON.stringify(result)}\n\n. This should look like the code block below. Note: The two new lines denote that the server can send the response and it is not waiting for more to the response message. If you ever see one new line, that means that the server needs to wait until it sees two consecutive new lines to send the response’s message.
6. In the catch block before responding with a status of 500, log the error to the console.

//Use the Change Stream to Detect Changes to the Document
changeStream.on("change", async (change) => {
  //Log that a Change was Detected
  console.log(`Change detected for meal plan ${mealPlanId}:`, change);
  //Find the Updates Meal Plan Document
  const result = await MealPlanModel.findById(id);
  //If the User can Still See the Meal Plan - write the new meal plan as
  //stringified JSON to the SSE response. End the Response with two new lines
  //to denote that the message is complete and the server can send the response
  if (userCanSeeMealPlan(result, req.session.user.id)) {
    res.write(`data: ${JSON.stringify(result)}\n\n`);
  }
});

router.get("/updates/:id", authorizeRoute, async (req, res) => {
  const mealPlanId = req.params.id;
  const id = ObjectId.createFromHexString(mealPlanId);

  console.log(`Client connected for meal plan updates: ${mealPlanId}`);

  // SSE Response Headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  try {
    //Setup a MongoDB Change Stream on the Mongoose MealPlanModel
    const changeStream = MealPlanModel.watch([
      {
        $match: {
          "documentKey._id": id,
        },
      },
    ]);

    //Use the Change Stream to Detect Changes to the Document
    changeStream.on("change", async (change) => {
      //Log that a Change was Detected
      console.log(`Change detected for meal plan ${mealPlanId}:`, change);
      //Find the Updates Meal Plan Document
      const result = await MealPlanModel.findById(id);
      //If the User can Still See the Meal Plan - write the new meal plan as
      //stringified JSON to the SSE response. End the Response with two new lines
      //to denote that the message is complete and the server can send the response
      if (userCanSeeMealPlan(result, req.session.user.id)) {
        res.write(`data: ${JSON.stringify(result)}\n\n`);
      }
    });
  } catch (err) {
    //Log the error that would have occured in the change stream
    console.error("Error in change stream:", err);
    res.status(500).send("Internal Server Error");
  }
});

In your case, you did not need to dig through the change event to figure out what the new meal plan is. You were able to just query the database and pass the new meal plan to the client since you knew it had changed.

Verify SSE Response is Being Sent By Server

Now that the SSE server endpoint is implemented to write a response whenever a change is detected to the associated meal plan, you can verify that the server is sending SSE responses.

1. Visit the application in the Web Browser in a new browser tab.
2. Log into the application as Jon.
3. Visit a meal plan.
4. Open Developer Tools, using the Network tab.
5. Inspect the pending request's response Headers and Event Stream.

Once you have Developer Tools open, make a change to the weekly meal plan by adding a recipe and verify that a response to the SSE request comes in. You will see something like the picture below.

You can also verify changes are being detected via the MongoDB Change Stream by looking for logs in the terminal that runs your server. You may see a message like the photo below.

Gracefully Handle Request Closures

Whenever the SSE request is closed, you will want to close the MongoDB Change Stream. This will prevent writing responses to a request that is already closed when data changes.

1. Underneath changeStream.on(...) implemented in the previous step, write req.on(“close”, () => {}); to define a callback to call when the SSE request closes.
2. In the callback for the close event, add a log to the console indicating the SSE connection has closed for the mealPlanId.
3. Lastly, close the MongoDB Change Stream in the callback by writing changeStream.close();.

router.get("/updates/:id", authorizeRoute, async (req, res) => {
  const mealPlanId = req.params.id;
  const id = ObjectId.createFromHexString(mealPlanId);

  console.log(`Client connected for meal plan updates: ${mealPlanId}`);

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  try {
    const changeStream = MealPlanModel.watch([
      {
        $match: {
          "documentKey._id": id,
        },
      },
    ]);

    changeStream.on("change", async (change) => {
      console.log(`Change detected for meal plan ${mealPlanId}:`, change);
      const result = await MealPlanModel.findById(id);
      if (userCanSeeMealPlan(result, req.session.user.id)) {
        res.write(`data: ${JSON.stringify(result)}\n\n`);
      }
    });

    //Gracefully Handle Request Closures by Closing the Change Stream
    //This prevents going rogue and sending responses that are no longer being listened for
    req.on("close", () => {
      console.log(`SSE connection closed for meal plan ${mealPlanId}`);
      changeStream.close();
    });
  } catch (err) {
    console.error("Error in change stream:", err);
    res.status(500).send("Internal Server Error");
  }
});

Send Periodic Heartbeats to Keep Connection Alive

By default, MongoDB connections may close after a period of inactivity, and the default timeout is 10 minutes. In order to prevent your SSE connection from timing out, you will add code to periodically send a heartbeat to keep the connection open and connected to MongoDB.

1. Directly above the req.on(“close”, () => {...} code in the SSE endpoint, use JavaScript’s setInterval method to run a callback function every thirty seconds. Store the return value in a constant variable called, heartbeat.
2. Inside the callback for the interval, write a response with a custom event called ”ping” and no data. Custom events start with ”event: <Event Name>\n” in the response. Custom events may also contain id’s, but this one will not.

router.get("/updates/:id", authorizeRoute, async (req, res) => {
  const mealPlanId = req.params.id;
  const id = ObjectId.createFromHexString(mealPlanId);

  console.log(`Client connected for meal plan updates: ${mealPlanId}`);

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  try {
    const changeStream = MealPlanModel.watch([
      {
        $match: {
          "documentKey._id": id,
        },
      },
    ]);

    changeStream.on("change", async (change) => {
      console.log(`Change detected for meal plan ${mealPlanId}:`, change);
      const result = await MealPlanModel.findById(id);
      if (userCanSeeMealPlan(result, req.session.user.id)) {
        res.write(`data: ${JSON.stringify(result)}\n\n`);
      }
    });

    // Periodically send a heartbeat to prevent timeout - Step 8
    const heartbeat = setInterval(() => {
      res.write(`event: ping\ndata: {}\n\n`);
    }, 30000); // Send every 30 seconds

    req.on("close", () => {
      console.log(`SSE connection closed for meal plan ${mealPlanId}`);
      changeStream.close();
    });
  } catch (err) {
    console.error("Error in change stream:", err);
    res.status(500).send("Internal Server Error");
  }
});

Verify SSE Heartbeat Responses are Being Sent By Server

Login to the application and visit a meal plan. Get Developer Tools ready and go to the Network tab. You should see your pending request receive a heartbeat response every thirty seconds. This may look like the photos below.

Conclusion

In conclusion, you have successfully implemented MongoDB Change Streams and Server-sent Events (SSEs) in a React frontend and Node.js backend. These technologies together support real-time data updates.

Take the time to do one last experiment.

1. Login to the application on separate Web Browser windows.
2. Visit the same meal plan in each window.
3. In each window, inspect the pending SSE request through the Developer Tools Network tab.
4. Make a change to a meal plan under one window.
5. Visit the other window and observe that the change is already reflected there without needing to refresh and refetch any data.
6. In both windows, look at the pending request and see the pending requests received identical responses for the change to the meal plan.

This experiment confirms that real-time data updates using MongoDB Change Streams and SSEs send responses to all requests subscribed to the specific SSE server endpoint. This allows updates to happen across windows.

In the end, you should feel confident in your ability to use MongoDB Change Streams and Server-sent Events to develop real-time data dashboards after completing this course.