Working with AWS VPC Flow Logs for Network Monitoring

Create S3 Bucket for VPC Flow Logs

1. Navigate to S3.
2. Note the account ID shown after the login name of cloud_user in the menu bar at the top right of the screen.
3. Copy the {account_id} without dashes to your favorite text editor.
4. Click Create bucket.
5. Enter a unique bucket name (e.g., vpc-flow-logs-{account_id}-{your-initials}). If the name you entered exists, try a different name.
6. Click Next.
7. Check the box for Default encryption.
8. Click Next.
9. Click Next.
10. Click Create bucket.
11. Select the checkbox next to the name of bucket you just created.
12. Click Copy Bucket ARN from the popup window. Paste the ARN in your favorite text editor. We will use the S3 ARN in an upcoming task.

Create VPC Flow Log to S3

1. In a new browser tab, navigate to VPC.
2. Click Your VPCs in the left-hand menu.
3. Select the LinuxAcademy VPC.
4. Select the Flow Logs tab for the VPC.
5. Click Create flow log.
6. In the Filter dropdown, select All.
7. Select Send to an S3 bucket for the Destination.
8. Paste the S3 bucket ARN you copied earlier.
9. Click Create.
10. Click Close.
11. Verify the flow log shows as Active.
12. Select the hyperlink in the Destination name field.
13. Select the Permissions tab.
14. Click Bucket Policy.
15. AWS automatically created a bucket policy when the VPC flow log was created. Notice the bucket path in the policy includes AWSLogs.
16. Please note, it can take 5-15 minutes before logs start to show up. You can proceed to the next step while waiting.

Create CloudWatch Log Group

1. Navigate to CloudWatch.
2. Click Logs in the left-hand menu.
3. Click Actions > Create log group.
4. Enter "VPCFlowLogs" for the name.
5. Click Create log group.

Create VPC Flow Log to CloudWatch

1. Navigate to VPC.
2. Click Your VPCs in the left-hand menu.
3. Select the LinuxAcademy VPC.
4. Select the Flow Logs tab for the VPC.
5. Click Create flow log.
6. In the Filter dropdown, select All.
7. Select Send to CloudWatch Logs for the Destination.
8. Select VPCFlowLogs for the Destination log group.
9. Select the IAM role with DeliverVPCFlowLogsRole in the name.
10. Click Create.
11. Click Close.
12. Verify the flow log shows as Active.
13. Select the hyperlink in the Destination name field.
14. Make sure the N. Virginia region (us-east-1) is selected.
15. Please note, it can take 5-15 minutes before logs start to show up. You can proceed to the next step while waiting.

1. Log in to the EC2 instance via SSH using the credentials provided.
2. Exit the terminal.
3. Navigate to EC2 in a new browser tab.
4. Select the instance with the name Web Server.
5. Select Security Groups view inbound rules link.
6. Observe that SSH is enabled for all source addresses.
7. Click Actions > Networking > Change Security Groups.
8. Uncheck the HTTP and SSH Access security group.
9. Select the HTTP Access security group, and verify no other security groups are selected.
10. Click Assign Security Groups.
11. Select Security Groups view inbound rules link.
12. Observe that SSH is not enabled.
13. Attempt to log in to the EC2 instance via SSH using the credentials provided — we expect this to timeout since we just selected a security group with no SSH access.
14. Cancel the SSH command after 15 seconds.
15. Return to EC2 in the AWS console.
16. Select the instance with the name Web Server.
17. Click Actions > Networking > Change Security Groups.
18. Uncheck the HTTP Access security group.
19. Select the HTTP and SSH Access security group, and verify no other security groups are selected.
20. Click Assign Security Groups.

Create CloudWatch Log Metric Filter

1. Navigate to CloudWatch.

2. Click Logs in the left-hand menu.

3. In the VPCFlowLogs row, click on the 0 filters link.

4. Click Add Metric Filter.

5. Enter the following Filter Pattern to track failed SSH attempts on port 22:

   [version, account, eni, source, destination, srcport, destport="22", protocol="6", packets, bytes, windowstart, windowend, action="REJECT", flowlogstatus]
   

6. In the Select Log Data to Test dropdown, select Custom Log Data.

7. Enter the following in the text box:

   2 086112738802 eni-0d5d75b41f9befe9e 61.177.172.128 172.31.83.158 39611 22 6 1 40 1563108188 1563108227 REJECT OK
   2 086112738802 eni-0d5d75b41f9befe9e 182.68.238.8 172.31.83.158 42227 22 6 1 44 1563109030 1563109067 REJECT OK
   2 086112738802 eni-0d5d75b41f9befe9e 42.171.23.181 172.31.83.158 52417 22 6 24 4065 1563191069 1563191121 ACCEPT OK
   2 086112738802 eni-0d5d75b41f9befe9e 61.177.172.128 172.31.83.158 39611 80 6 1 40 1563108188 1563108227 REJECT OK
   

8. Click Test Pattern.

9. Click on the Show test results link. You should see 2 of the 4 records matching.

10. Click Assign Metric.

11. Enter destination-port-22-rejects as the Filter Name.

12. Enter SSH failures as the Metric Name.

13. Click Create Filter.

Create Alarm Based on the Metric Filter

1. Click on the Create Alarm link in the new destination-port-22-rejects metric filter box.
2. Select the Greater/Equal radio button.
3. Enter 1 in the Define the threshold value edit box.
4. Click Next.
5. Select the Create new topic button.
6. Enter PROD-ALERT-{your-initials} in the Create a new topic... edit box.
7. Enter your email address in the Email endpoints that will receive the notification edit box.
8. Click Create topic.
9. Click Next.
10. Enter SSH REJECT as the Alarm name.
11. Click Next.
12. Click Create alarm.
13. You will receive an email notification asking you to confirm your subscription. Click the Confirm Subscription link in the email.

1. Log in to the EC2 instance via SSH using the credentials provided.
2. Exit the terminal.
3. Navigate to EC2 in a new browser tab.
4. Select the instance with the name Web Server.
5. Select Security Groups view inbound rules link.
6. Observe that SSH is enabled for all source addresses.
7. Click Actions > Networking > Change Security Groups.
8. Uncheck the HTTP and SSH Access security group.
9. Select the HTTP Access security group, and verify no other security groups are selected.
10. Click Assign Security Groups.
11. Select Security Groups view inbound rules link.
12. Observe that SSH is not enabled.
13. Attempt to log in to the EC2 instance via SSH using the credentials provided — we expect this to timeout since we just selected a security group with no SSH access.
14. Cancel the SSH command after 15 seconds.
15. Return to EC2 in the AWS console.
16. Select the instance with the name Web Server.
17. Click Actions > Networking > Change Security Groups.
18. Uncheck the HTTP Access security group.
19. Select the HTTP and SSH Access security group, and verify no other security groups are selected.
20. Click Assign Security Groups.
21. Observe you receive an email notification once there is a failed SSH login approximately 5-15 min. You can move to the next task while waiting.

1. Navigate to CloudWatch.
2. Click Insights in the left-hand menu.
3. Select VPCFlowLogs in the Select a log group search window.
4. Click Sample queries > VPC flow log queries > Top 20 source IP addresses with highest number of rejected requests.
5. Observe the query has changed.
6. Click Run query.

Record Reference Information to Be Used in Athena Queries

1. Navigate to S3 in a new browser tab.
2. Open {your_log_bucket} created earlier in this hands-on lab.
3. Copy {your_log_bucket} name to a text editor for later use.
4. Open the AWSLogs folder.
5. Open the {account_id} folder.
6. Copy the {account_id} to a text editor for later use if you didn’t save it earlier.
7. Open the vpcflowlogs folder.
8. Open the us-east-1 folder.
9. Open the {Year} folder.
10. Open the {Month} folder.
11. Write down the {Year}/{Month}/{Day} using the latest {Day} shown.

Create the Athena Table

1. Navigate to Athena.

2. If you come to a page with a Get Started button, click on it. Otherwise skip this step.

3. If a Tutorial popup window shows up, then click on the X in the upper right of the screen to close it. You can return to the tutorial later by selecting it in the upper right-hand menu.

4. Paste the following DDL code in the new query window:

   CREATE EXTERNAL TABLE IF NOT EXISTS default.vpc_flow_logs (
     version int,
     account string,
     interfaceid string,
     sourceaddress string,
     destinationaddress string,
     sourceport int,
     destinationport int,
     protocol int,
     numpackets int,
     numbytes bigint,
     starttime int,
     endtime int,
     action string,
     logstatus string
   )  
   PARTITIONED BY (dt string)
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ' '
   LOCATION 's3://{your_log_bucket}/AWSLogs/{account_id}/vpcflowlogs/us-east-1/'
   TBLPROPERTIES ("skip.header.line.count"="1");
   

5. Update {your_log_bucket} and {account_id} in the query window with the values from this hands-on lab.

6. Click Run query.

7. You should see a Query successful. message once this has finished executing.

Create Partitions to Be Able to Read the Data

1. Paste the following code in a new query window:

   ALTER TABLE default.vpc_flow_logs
   ADD PARTITION (dt='{Year}-{Month}-{Day}')
   location 's3://{your_log_bucket}/AWSLogs/{account_id}/vpcflowlogs/us-east-1/{Year}/{Month}/{Day}';
   

2. Update the following elements in the query window with the values from this hands-on lab:

   • {your_log_bucket}
   • {account_id}
   • {Year},{Month}
   • {Day}

3. Click Run query.

4. You should receive a Query successful. message.

1. Run the following query in a new query window:

   SELECT day_of_week(from_iso8601_timestamp(dt)) AS
     day,
     dt,
     interfaceid,
     sourceaddress,
     destinationport,
     action,
     protocol
   FROM vpc_flow_logs
   WHERE action = 'REJECT' AND protocol = 6
   order by sourceaddress
   LIMIT 100;