Using Python to Extract Prometheus Metrics

To use SSH to access the master node, enter:

ssh cloud_user@[Master's Public IP Address]

You will be prompted for the cloud_user password that is available on the lab startup page.

To review the promql.py program, navigate to the following GitHub address:

https://github.com/linuxacademy/content-aiops-essentials/blob/master/promql.py

From your terminal session on the master node, enter the following command to start the Python program:

python3 promql.py > promql.out 2> promql.err &

Note: Be sure you use the ampersand after the command so it will run in background on your server.

To stress the cluster, you may deploy the stress-test deployment with the following command:

kubectl create -f stress-test.yaml

Navigate in your browser to the Prometheus dashboard:

http://[Master Node IP]:9090

While the Python program is gathering metrics, use the following command to vary the load by changing the number of replicas:

kubectl scale deployment.v1.apps/stress-test --replicas=10

You may then increase the number of replicas to 20, 30, 40, and so on.

kubectl scale deployment.v1.apps/stress-test --replicas=[number here]

If you want to use other PromQL in the Prometheus dashboard, here are the two examples we use in the Python promql.py program:

100 - avg(irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) by (instance) * 100

And:

(node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Cached_bytes + node_memory_Buffers_bytes)) / node_memory_MemTotal_bytes * 100

You may look at the promql.py output with any of the following commands:

tail promql.out

Or:

more promql.out

Or:

tail -f promql.out