Lab
A Cloud Guru

Configuring Prometheus Alerts

Prometheus alerts allow you to issue real-time notifications triggered by your Prometheus metric data. In this lab, you will work directly with Prometheus alerts. You will create an alert in an existing Prometheus Server by modifying the Prometheus configuration.

Try for free Contact sales

Path Info

Level

Intermediate

Duration

45m

Published

Jun 12, 2020

Challenge

Create an Alerting Rule in Prometheus to Alert When the Server Goes Down
1. Log in to the Prometheus server.
2. Edit the Prometheus config file:
```
sudo vi /etc/prometheus/prometheus.yml
```
3. Add a path for rules files to the Prometheus config:
```
rule_files:
  - "/etc/prometheus/rules/*.yml"
```
4. Create the rules directory:
```
sudo mkdir -p /etc/prometheus/rules/
```
5. Create a new rules file for your alerting rule:
```
sudo vi /etc/prometheus/rules/limedrop-web.yml
```
6. Implement an alerting rule to issue an alert when the server goes down:
```
groups:
- name: limedrop-web
  rules:
  - alert: WebServerDown
    expr: up{instance="limedrop-web:9100"} == 0
    labels:
      severity: critical
    annotations:
      summary: Web Server Down
```
7. Restart Prometheus to reload the configuration:
```
sudo systemctl restart prometheus
```
8. Access Prometheus in a browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.
9. Click Alerts. You should see your WebServerDown alert listed.
Challenge

Configure a Routing Tree and Receiver in Alertmanager
1. Edit the Alertmanager configuration file:
```
sudo vi /etc/alertmanager/alertmanager.yml
```
2. Add a new node to the routing table:
```
route:

  ...

  routes:
  - receiver: 'web.hook'
    group_wait: 30s
    match_re:
      alertname: WebServerDown
```
3. Restart Alertmanager to reload the configuration:
```
sudo systemctl restart alertmanager
```
4. You can test your alert by shutting down the limedrop-web exporter to simulate the server going down. To do this, stop the limedrop_web_exporter service:
```
sudo systemctl stop limedrop_web_exporter
```
5. After a few moments, check the Alertmanager page in your browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. If the alert is firing, you should see the WebServerDown alert appear.
6. If you wish, you can start limedrop_web_exporter again and watch the alert disappear in Alertmanager:
```
sudo systemctl start limedrop_web_exporter
```

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.