- Lab
-
Libraries: If you want this lab, consider one of these libraries.
- Cloud

Advanced Configuration for Prometheus Alerts
Prometheus Alertmanager provides some additional useful features around the management of alerts. These features allow you to customize and tweak your alerts so they are more useful in real-world situations. In this lab, you will have the opportunity to practice using some of these Alertmanager features, including alert grouping, inhibitions, and silences.

Lab Info
Table of Contents
-
Challenge
Combine the Web Server Down Alerts into a Single Group
-
Log in to the Prometheus server.
-
Edit the Alertmanager configuration file:
sudo vi /etc/alertmanager/alertmanager.yml
-
Add a new node to routing tree to combine the
WebServer.*Down
alerts:route: ... routes: - receiver: 'web.hook' group_by: ['service'] match_re: alertname: 'WebServer.*Down'
-
Load the new configuration:
sudo killall -HUP alertmanager
-
Check Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093
. You should see the Web Server alerts grouped together under the groupservice="webserver"
.
-
-
Challenge
Create an Inhibition to Stop the `WebBadGateway` Alert When a `WebServerDown` Alert Is Already Firing
-
Edit the Alertmanager configuration file:
sudo vi /etc/alertmanager/alertmanager.yml
-
Add a new inhibit rule:
inhibit_rules: ... - source_match_re: alertname: 'WebServer.*Down' target_match: alertname: 'WebBadGateway'
-
Load the new configuration:
sudo killall -HUP alertmanager
-
Check Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093
. TheWebBadGateway
should no longer appear. You can click the Inhibited box to make it appear again.
-
-
Challenge
Silence the `WebServer1Down` Alert
-
Access Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093
. -
Expand the
service="webserver"
group. -
Locate the alert with
alertname="WebServer1Down"
, and click the Silence button for that alert. -
Fill out the Creator and Comment fields, and then click Create.
-
If you return to the main Alertmanager page, the
WebServer1Down
should no longer appear.
-
About the author
Real skill practice before real-world application
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.
Learn by doing
Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.
Follow your guide
All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.
Turn time into mastery
On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.