Exploring Basic HAProxy Monitoring Techniques

1. Look at and monitor the status of our HAProxy installation by directly interacting with the stats socket.

2. Look at the socket, using file /var/lib/haproxy/stats.

3. Let's try some non-interactive commands. Pull raw data from the stats socket, using nc, by sending the show stat command to the stats socket.

   It returns a lot of information in a format that's not really human-friendly, but it can be reformatted.

4. Add some formatting using cut and column, and a looping mechanism using watch. Pick as many or as few fields as you like. Play around with the formatting and see what you can come up with.

   You should see that we're getting statistics for the fields we specified in the cut command, then using column to make a nicely formatted table. The watch command refreshes every second until we interrupt it with CTRL-C.

5. Try an interactive connection using nc and the /var/lib/haproxy/stats socket.

   This will connect us to the socket and allow us to enter one command.

6. Use the prompt command to get an interactive interface. If you'd like help, use ?. Sending an empty line or using the quit command followed by ENTER will get you out of the prompt.

So, this is the most direct, hands-on way to do it, but it's a lot of overhead for daily use, unless we put a framework around it. Is there a better way?

The /var/lib/haproxy/stats socket is extremely useful, but manual interaction is a bit cumbersome. What if we had a better way to leverage it?

1. Add a code block to the end of the /etc/haproxy/haproxy.cfg file that starts an HAProxy stats web server on port 8050 for all addresses at the URI of /.

2. Hide the version of HAProxy in the statistics report for better security.

3. Restart the haproxy service.

4. Connect to the stats web page on port 8050 on the public IP (or DNS) of the lab server and check things out.

   We've got a nice, clean, web-based interface that presents our statistics in an easy-to-read format. If we want to update our statistics, we're just a refresh away.

Another handy utility we can use to get a handle on what's happening in our HAProxy environment is the inlcuded halog command-line utility.

Before we dig deeper into the logs, we need to put more data in.

1. Launch 2 ApacheBench sessions:

   ab -n 100 -c 10 https://www.site1.com/ > ~/ab_site1.log > /dev/null 2>&1 &
   

   ab -n 100 -c 10 https://www.site2.com/ > ~/ab_site2.log > /dev/null 2>&1 &
   

2. Pull HTTP traffic using curl:

   for conn in `seq 1 100` ; do curl -k https://www.site1.com/ ; done > /dev/null 2>&1 &
   

   for conn in `seq 1 100` ; do curl -k https://www.site2.com/ ; done > /dev/null 2>&1 &
   

3. Pull HTTP traffic using wget:

   for conn in `seq 1 100` ; do wget --no-check-certificate -O - https://www.site1.com/test.txt ; done > /dev/null 2>&1 &
   

   for conn in `seq 1 100` ; do wget --no-check-certificate -O - https://www.site2.com/test.txt ; done > /dev/null 2>&1 &
   

4. Pull scp traffic:

   for conn in `seq 1 100` ; do bash -c 'scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -P 2222 [email protected]:/sshfiles/ssh-test.txt . &' ; done > /dev/null 2>&1 &
   

   Now that our logs are filled with data, we can proceed!

5. Extract data from the /var/log/haproxy-combined-traffic.log file using halog:

   • Pull data on per-server HTTP statistics
   • List URLs by the number of HTTP requests
   • List URLs with 429 errors
   • List URLs by the number of errors generated
   • Try some of your own!

So, halog is a handy tool for parsing HAProxy logs to present the information contained within in an easy-to-read format that tells us exactly what we're looking for.

Congratulations, Cloud Guru! You mastered 3 helpful monitoring methods that come with HAProxy!