Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.
  • Labs icon Lab
  • A Cloud Guru
Google Cloud Platform icon

Using Kafka Connect to Capture Data from a Relational Database

Kafka Connect can be used for a variety of databases, given it has a multitude of built-in connectors. In this hands-on lab, we set up Kafka Connect in a way allowing us to continuously stream data from a SQLite database directly into Kafka. This lab demonstrates how easy it is to use Kafka Connect to get data into Kafka for further analysis.

Google Cloud Platform icon

Path Info

Clock icon Intermediate
Clock icon 30m
Clock icon Jul 05, 2019

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Table of Contents

  1. Challenge

    Start the Docker Container and Open a New Shell Inside

    Run the following command to start the container and connect to it.

    sudo docker run -ti --rm --name sqlite-demo --network host confluentinc/docker-demo-base:3.3.0
  2. Challenge

    Start Kafka

    Start Kafka with the following command.

    cd /tmp
    confluent start
  3. Challenge

    Install SQLite3 and Create a New Database Table

    Install SQLite3 with the following commands.

    apt-get update
    apt-get install sqlite3

    Create a new database and table with the following command.

    sqlite3 test.db
    CREATE TABLE accounts (
    name VARCHAR(255));
  4. Challenge

    Insert at Least Two New Values into Your New Database Table

    Insert the values into your database table using the following command.

    INSERT INTO accounts(name) VALUES('chad');
    INSERT INTO accounts(name) VALUES('terry');
  5. Challenge

    Start Kafka Connect in Standalone Mode

    Use the following command to stop Kafka Connect from running in distributed mode.

    confluent stop connect

    Use the following commnand to start Kafka Connect in standalone mode.

    connect-standalone -daemon /etc/schema-registry/ /etc/kafka-connect-jdbc/
  6. Challenge

    Verify the Connectors and Topic Have Been Created

    Check the logs to see that the connector is finished creating.

    cat /logs/connectStandalone.out | grep -i "finished"

    List the connectors.

    curl -s localhost:8083/connectors

    Describe the topic created by the connector.

    kafka-topics --list --zookeeper localhost:2181 | grep test-sqlite-jdbc
  7. Challenge

    Start a Kafka Consumer and Write New Data to the Database

    Start a new Console Consumer:

    kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9092 --topic test-sqlite-jdbc-accounts --from-beginning

    In a new terminal, write some new values to the database table.

    sudo docker exec -it sqlite-demo //bin//bash
    cd /tmp
    sqlite3 test.db
    INSERT INTO accounts(name) VALUES('william');

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Provided environment for hands-on practice

We will provide the credentials and environment necessary for you to practice right within your browser.

Guided walkthrough

Follow along with the author’s guided walkthrough and build something new in your provided environment!

Did you know?

On average, you retain 75% more of your learning if you get time for practice.

Start learning by doing today

View Plans