Using Grafana with Prometheus for Alerting and Monitoring

Open port 9323 in the firewall.

firewall-cmd --zone=public --add-port=9323/tcp

Open /etc/docker/daemon.json and add the following.

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Edit the prometheus.yml file in the /root directory.

vi ~/prometheus.yml

Change the contents to reflect the following:

scrape_configs:
  - job_name: prometheus
    scrape_interval: 5s
    static_configs:
    - targets:
      - prometheus:9090
      - node-exporter:9100
      - pushgateway:9091
      - cadvisor:8080

  - job_name: docker
    scrape_interval: 5s
    static_configs:
    - targets:
      - PRIVATE_IP_ADDRESS:9323

Edit docker-compose.yml in the /root directory.

vi ~/docker-compose.yml

Change the contents to reflect the following:

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  pushgateway:
    image: prom/pushgateway
    container_name: pushgateway
    ports:
      - 9091:9091
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    expose:
      - 9100
  grafana:
    image: grafana/grafana:5.3.0
    container_name: grafana
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=password
    depends_on:
      - prometheus
      - cadvisor

Click the plus sign (+) on the left side of the Grafana interface. Click Import. Copy the contents of the JSON file included in the lab instructions.

Paste the contents of the JSON file into the Import screen of the Grafana interface, and click Load. In the upper right-hand corner, click on Refresh every 5m and select Last 5 minutes.