Configuring Prometheus to Use Service Discovery

Libraries: If you want this lab, consider one of these libraries.
Cloud

Configuring Prometheus to Use Service Discovery

Recently, your team has deployed Prometheus to the companies Kubernetes cluster. Now it is time to use service discovery to find targets for cAdvisor and the Kubernetes API. You have been tasked with modifying the Prometheus Config Map that is used to create the `prometheus.yml` file. Create the scrape config and add the jobs for `kubernetes-apiservers` and `kubernetes-cadvisor`. Then, propagate the changes to the Prometheus pod.

Get started Contact sales

Lab Info

Level

Intermediate

Last updated

Sep 23, 2025

Duration

1h 0m

Challenge

Configure the Service Discovery Targets

Prepare the environment and create the monitoring namespace as below:

sudo su -
cd /root/prometheus
./bootstrap.sh
kubectl get pods -n monitoring

Edit prometheus-config-map.yml and add in the two service discovery targets:

vi prometheus-config-map.yml

When we're done, the whole file should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

Challenge

Apply the Changes to the Prometheus Configuration Map
Now, apply the changes that were made to prometheus-config-map.yml:
```
kubectl apply -f prometheus-config-map.yml
```
Challenge

Delete the Prometheus Pod
1. List the pods to find the name of the Prometheus pod:
```
kubectl get pods -n monitoring
```
1. Delete the Prometheus pod:
```
kubectl delete pods <POD_NAME> -n monitoring
```
1. Open up a new web browser tab, and navigate to the Expression browser. This will be at the public IP of the lab server, on port 30000:
```
http://<IP>:30000
```
1. Click on Status, and select Target from the dropdown. We should see two targets in there.

About the author

Pluralsight Skills

Pluralsight Skills gives leaders confidence they have the skills needed to execute technology strategy. Technology teams can benchmark expertise across roles, speed up release cycles and build reliable, secure products. By leveraging our expert content, skill assessments and one-of-a-kind analytics, keep up with the pace of change, put the right people on the right projects and boost productivity. It's the most effective path to developing tech skills at scale.

Real skill practice before real-world application

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Learn by doing

Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.

Follow your guide

All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.

Turn time into mastery

On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.