Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.
  • Labs icon Lab
  • A Cloud Guru
Google Cloud Platform icon

Back up Messages to an S3 Bucket in Kafka

Kafka is known for replicating data amongst brokers to account for broker failure. Sometimes this isn't enough — or perhaps you are sharing this data with a third-party application. Backing up your messages to an alternate source can save a lot of time and energy when trying to configure access to a cluster or deliver that data quickly across the world. In this hands-on lab, we will send topic messages to Amazon S3 using Kafka Connect. By the end of the lab, you will know how to use Kafka commands for sending data to an outside data repository. (**Note:** No previous AWS knowledge is required.)

Google Cloud Platform icon

Path Info

Clock icon Intermediate
Clock icon 1h 45m
Clock icon Jul 12, 2019

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Table of Contents

  1. Challenge

    Connect and Start the Kafka Cluster

    Note: If you have trouble connecting via ssh, please give the lab an extra few minutes to finish setting up.

    1. In the bastion host, start a container and open a shell to that container.
    2. Change to the /tmp directory.
    3. Start up the Kafka cluster.
  2. Challenge

    Create a New S3 Bucket

    1. Update the system.
    2. Install the awscli tool.
    3. Configure access to AWS by creating a key (note that our cloud_user access and secret access keys are on the hands-on lab page)
    4. Create a new bucket in the us-east-1 region (making sure the name is globally unique, without any uppercase letters or underscores).
    5. Install Vim.
    6. Open the properties file.
    7. Change the region to us-east-1.
    8. Add the new bucket name.
  3. Challenge

    Start a Producer to a New Topic Named `s3_topic` and Write at Least Nine Messages

    1. Open an Avro console producer to the topic, and include a schema.

    2. Type the nine messages following the defined schema:

      {"f1": "value1"}
      {"f1": "value2"}
      {"f1": "value3"}
      {"f1": "value4"}
      {"f1": "value5"}
      {"f1": "value6"}
      {"f1": "value7"}
      {"f1": "value8"}
      {"f1": "value9"}
    3. Press Ctrl + C to exit the session.

  4. Challenge

    Start the Connector and Verify the Messages Are in the S3 Bucket

    1. Start the connector and load the configuration.
      • We'll then see some JSON output, including our bucket name.
    2. Copy the bucket name
    3. List the bucket's objects, using the bucket name you copied in the command.

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Provided environment for hands-on practice

We will provide the credentials and environment necessary for you to practice right within your browser.

Guided walkthrough

Follow along with the author’s guided walkthrough and build something new in your provided environment!

Did you know?

On average, you retain 75% more of your learning if you get time for practice.

Start learning by doing today

View Plans