Table of Contents

1. Challenge

   Add a Rules File

   1. Add a rules file configuration to the Prometheus config:

      sudo $EDITOR /etc/prometheus/prometheus.yml
      
      rule_files:
        - "rules.yml"
      

   2. Save and exit.

   3. Create and open the rules.yml file:

      sudo $EDITOR /etc/prometheus/rules.yml
      

2. Challenge

   Add an Alert to Track Uptime

   1. Before creating the alert itself, create a recording of the desired metric:

      groups:
        - name: uptime
          rules:
            - record: job:uptime:average:ft
              expr: avg without (instance) (up{job="forethought"})
      

   2. Create an alert based on this recording:

      groups:
        - name: uptime
          rules:
            - record: job:uptime:average:ft
              expr: avg without (instance) (up{job="forethought"})
            - alert: ForethoughtApplicationDown
              expr: job:uptime:average:ft < .75
              for: 30s
              labels:
                severity: page
                team: devops
      

   3. Save and exit.

   4. Restart Prometheus:

      sudo systemctl restart prometheus
      sudo systemctl status prometheus
      

3. Challenge

   Configure Alertmanager to Use an SMTP Smarthost

   1. Open the Alertmanager configuration file:

      sudo $EDITOR /etc/alertmanager/alertmanager.yml
      

   2. Define the global settings:

      global:
        resolve_timeout: 5m
        smtp_smarthost: 'localhost:25'
        smtp_from: 'prometheus'
      

4. Challenge

   Set up Alertmanager Routing

   1. Set up the backup route:

      route:
        receiver: 'email_backup'
        group_by: ['alertname']
        group_wait: 10s
        group_interval: 10s
        repeat_interval: 1m
      

   2. Set up the route for critical alerts:

      route:
        receiver: 'email_backup'
        group_by: ['alertname']
        group_wait: 10s
        group_interval: 10s
        repeat_interval: 1m
        routes:
          - match:
              severity: 'critical'
            group_by: ['team']
            receiver: 'email_pager'
      

   3. Set up the route for team alerts:

      route:
        receiver: 'email_backup'
        group_by: ['alertname']
        group_wait: 10s
        group_interval: 10s
        repeat_interval: 1m
        routes:
          - match:
              severity: 'critical'
            group_by: ['team']
            receiver: 'email_pager'
            routes:
              - match:
                  team: devops
                  receiver: 'email_devops'
      ``

5. Challenge

   Create the Needed Receivers

   1. Create the receivers:

      receivers:
      - name: 'email_backup'
        email_configs:
          - to: '[email protected]'
      - name: 'email_pager'
        email_configs:
          - to: '[email protected]'
      - name: 'email_devops'
        email_configs:
          - to: '[email protected]'
      

   2. Save and exit.

   3. Restart Alertmanager:

      sudo systemctl restart alertmanager