Table of Contents

1. Challenge

   Create the http Module within the servercheck Package and Install requests

   To begin, we need to change to the servercheck directory and create and activate our virtualenv.

   $ cd servercheck
   $ pipenv install --python python3.7
   ...
   $ pipenv shell
   (servercheck) $
   

   Next, we need to create the module to work in. We'll call it http.py. To be a module within a package, we'll need to place this file within the servercheck directory that contains an __init__.py.

   (servercheck) $ touch servercheck/http.py
   

   Now that we have a module, we need to install the requests package as a dependency of our tool:

   (servercheck) $ pipenv install requests
   

   We'll also want to add this to the REQUIRED list in our setup.py:

   setup.py (partial)

   # What packages are required for this module to be executed?
   REQUIRED = ["click", "requests"]
   

2. Challenge

   Make Concurrent Requests and Return the Results

   From within our new http module, we will create one function to act as the public interface that we'll eventually call from our CLI. Let's call this function ping_servers. It will receive the argument servers:

   servercheck/http.py

   def ping_servers(servers):
       results = {'success': [], 'failure': []}
       asyncio.run(make_requests(servers, results))
       return results
   

   The main thing we need to do here is create our results dictionary and trigger our unwritten make_requests coroutine that will handle creating and running the HTTP request tasks.

   Next, we'll create the make_requests, ping, and get functions:

   servercheck/http.py

   import asyncio
   import requests
   import os
   
   
   def get(server):
       debug = os.getenv("DEBUG")
       try:
           if debug:
               print(f"Making request to {server}")
           response = requests.get(f"http://{server}")
           if debug:
               print(f"Received response from {server}")
           return {"status_code": response.status_code, "server": server}
       except:
           if debug:
               print(f"Failed to connect to {server}")
           return {"status_code": -1, "server": server}
   
   
   async def ping(server, results):
       loop = asyncio.get_event_loop()
       future_result = loop.run_in_executor(None, get, server)
       result = await future_result
       if result["status_code"] in range(200, 299):
           results["success"].append(server)
       else:
           results["failure"].append(server)
   
   
   async def make_requests(servers, results):
       tasks = []
   
       for server in servers:
           task = asyncio.create_task(ping(server, results))
           tasks.append(task)
   
       await asyncio.gather(*tasks)
   
   
   def ping_servers(servers):
       results = {"success": [], "failure": []}
       asyncio.run(make_requests(servers, results))
       return results
   
   

   Since we want all of the HTTP requests to be made as soon as possible, we'll schedule each in its own task and utilize asyncio.gather to run all of the tasks and wait until they are all finished.

   The requests library isn't designed to work with asyncio, so we need to manually run our HTTP requests on the event loop to make sure that we can avoid waiting on the network IO.

3. Challenge

   Test Against Additional Servers Using REPL

   Now that we have all of our functions, let's test them out. Our hands-on lab environment has two other servers that have a few web servers listening on various ports. We can test our code by loading our package into the REPL and interacting with these servers:

   (servercheck) $ DEBUG=true PYTHONPATH=. python
   >>> from servercheck.http import ping_servers
   >>> servers = ('web-node1:80', 'web-node2:80', 'web-node1:3000', 'web-node2:3000', 'web-node1:8080')
   >>> ping_servers(servers)
   Making request to web-node1:80
   Making request to web-node2:80
   Making request to web-node1:3000
   Making request to web-node2:3000
   Making request to web-node1:8080
   Failed to connect to web-node1:3000
   Received response from web-node1:80
   Received response from web-node2:80
   Received response from web-node2:3000
   Received response from web-node1:8080
   {'success': ['web-node1:80', 'web-node2:80', 'web-node2:3000', 'web-node1:8080'], 'failure': ['web-node1:3000']}
   >>>
   

   We can see from our debug output that all of the requests are started rapidly and the responses come back in the order that they finish. Without the DEBUG environment variable set, we'd only see the results dictionary.

4. Challenge

   Utilize servercheck.http.ping_servers in the CLI Function

   Now that we have the ping_servers function, we can use it in the cli function to make our tool work the way that we expect it to. The cli function already collects the server information that the user passes in; we'll pass that information into the ping_servers function.

   servercheck/cli.py

   import click
   import json
   import sys
   from .http import ping_servers
   
   
   @click.command()
   @click.option("--filename", "-f", default=None)
   @click.option("--server", "-s", default=None, multiple=True)
   def cli(filename, server):
       if not filename and not server:
           raise click.UsageError("must provide a JSON file or servers")
   
       # Create a set to prevent duplicate server/port combinations
       servers = set()
   
       # If --filename or -f option is used then attempt to read
       # the file and add all values to the `servers` set.
       if filename:
           try:
               with open(filename) as f:
                   json_servers = json.load(f)
                   for s in json_servers:
                       servers.add(s)
           except:
               print("Error: Unable to open or read JSON file")
               sys.exit(1)
   
       # If --server or -s option are used then add those values
       # to the set.
       if server:
           for s in server:
               servers.add(s)
   
       # Make requests and collect results
       results = ping_servers(servers)
   
   

   Now we can print the results to match our design goal:

   servercheck/cli.py

   import click
   import json
   import sys
   from .http import ping_servers
   
   
   @click.command()
   @click.option("--filename", "-f", default=None)
   @click.option("--server", "-s", default=None, multiple=True)
   def cli(filename, server):
       if not filename and not server:
           raise click.UsageError("must provide a JSON file or servers")
   
       # Create a set to prevent duplicate server/port combinations
       servers = set()
   
       # If --filename or -f option is used then attempt to read
       # the file and add all values to the `servers` set.
       if filename:
           try:
               with open(filename) as f:
                   json_servers = json.load(f)
                   for s in json_servers:
                       servers.add(s)
           except:
               print("Error: Unable to open or read JSON file")
               sys.exit(1)
   
       # If --server or -s option are used then add those values
       # to the set.
       if server:
           for s in server:
               servers.add(s)
   
       # Make requests and collect results
       results = ping_servers(servers)
   
       print("Successful Connections")
       print("---------------------")
       for server in results['success']:
           print(server)
   
       print("\n Failed Connections")
       print("------------------")
       for server in results['failure']:
           print(server)
   

   Let's make sure that our package is installed into our virtualenv in an editable way before testing the final product:

   (servercheck) $ pip install -e .
   ...
   

   Next we'll create a test JSON file called example.json within our project's directory:

   [
       "web-node1:80",
       "web-node1:8000",
       "web-node1:3000",
       "web-node2:80",
       "web-node2:3000"
   ]
   

   To test our tool, we can use the example.json and also pass some more information -s options to the servercheck executable.

   (servercheck) $ servercheck -f example.json -s 'web-node1:80' -s 'web-node1:9000'
   Successful Connections
   ----------------------
   web-node1:80
   web-node2:80
   web-node2:3000
   web-node1:8000
   
   Failed Connections
   ------------------
   web-node1:3000
   web-node1:9000