Table of Contents

1. Challenge

   Create an Alerting Rule in Prometheus to Alert When the Server Goes Down

   1. Log in to the Prometheus server.

   2. Edit the Prometheus config file:

      sudo vi /etc/prometheus/prometheus.yml
      

   3. Add a path for rules files to the Prometheus config:

      rule_files:
        - "/etc/prometheus/rules/*.yml"
      

   4. Create the rules directory:

      sudo mkdir -p /etc/prometheus/rules/
      

   5. Create a new rules file for your alerting rule:

      sudo vi /etc/prometheus/rules/limedrop-web.yml
      

   6. Implement an alerting rule to issue an alert when the server goes down:

      groups:
      - name: limedrop-web
        rules:
        - alert: WebServerDown
          expr: up{instance="limedrop-web:9100"} == 0
          labels:
            severity: critical
          annotations:
            summary: Web Server Down
      

   7. Restart Prometheus to reload the configuration:

      sudo systemctl restart prometheus
      

   8. Access Prometheus in a browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.

   9. Click Alerts. You should see your WebServerDown alert listed.

2. Challenge

   Configure a Routing Tree and Receiver in Alertmanager

   1. Edit the Alertmanager configuration file:

      sudo vi /etc/alertmanager/alertmanager.yml
      

   2. Add a new node to the routing table:

      route:
      
        ...
      
        routes:
        - receiver: 'web.hook'
          group_wait: 30s
          match_re:
            alertname: WebServerDown
      

   3. Restart Alertmanager to reload the configuration:

      sudo systemctl restart alertmanager
      

   4. You can test your alert by shutting down the limedrop-web exporter to simulate the server going down. To do this, stop the limedrop_web_exporter service:

      sudo systemctl stop limedrop_web_exporter
      

   5. After a few moments, check the Alertmanager page in your browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. If the alert is firing, you should see the WebServerDown alert appear.

   6. If you wish, you can start limedrop_web_exporter again and watch the alert disappear in Alertmanager:

      sudo systemctl start limedrop_web_exporter