Back up Messages to an S3 Bucket in Kafka

Note: If you have trouble connecting via ssh, please give the lab an extra few minutes to finish setting up.

1. In the bastion host, start a container and open a shell to that container.
2. Change to the /tmp directory.
3. Start up the Kafka cluster.

1. Update the system.
2. Install the awscli tool.
3. Configure access to AWS by creating a key (note that our cloud_user access and secret access keys are on the hands-on lab page)
4. Create a new bucket in the us-east-1 region (making sure the name is globally unique, without any uppercase letters or underscores).
5. Install Vim.
6. Open the properties file.
7. Change the region to us-east-1.
8. Add the new bucket name.

1. Open an Avro console producer to the topic, and include a schema.

2. Type the nine messages following the defined schema:

   {"f1": "value1"}
   {"f1": "value2"}
   {"f1": "value3"}
   {"f1": "value4"}
   {"f1": "value5"}
   {"f1": "value6"}
   {"f1": "value7"}
   {"f1": "value8"}
   {"f1": "value9"}
   

3. Press Ctrl + C to exit the session.

1. Start the connector and load the configuration.
   • We'll then see some JSON output, including our bucket name.
2. Copy the bucket name
3. List the bucket's objects, using the bucket name you copied in the command.