- A Cloud Guru
Using Kafka Connect to Capture Data from a Relational Database
Kafka Connect can be used for a variety of databases, given it has a multitude of built-in connectors. In this hands-on lab, we set up Kafka Connect in a way allowing us to continuously stream data from a SQLite database directly into Kafka. This lab demonstrates how easy it is to use Kafka Connect to get data into Kafka for further analysis.
Table of Contents
Start the Docker Container and Open a New Shell Inside
Run the following command to start the container and connect to it.
sudo docker run -ti --rm --name sqlite-demo --network host confluentinc/docker-demo-base:3.3.0
Start Kafka with the following command.
cd /tmp confluent start
Install SQLite3 and Create a New Database Table
Install SQLite3 with the following commands.
apt-get install sqlite3
Create a new database and table with the following command.
sqlite3 test.db CREATE TABLE accounts ( id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, name VARCHAR(255));
Insert at Least Two New Values into Your New Database Table
Insert the values into your database table using the following command.
INSERT INTO accounts(name) VALUES('chad'); INSERT INTO accounts(name) VALUES('terry'); .quit
Start Kafka Connect in Standalone Mode
Use the following command to stop Kafka Connect from running in distributed mode.
confluent stop connect
Use the following commnand to start Kafka Connect in standalone mode.
connect-standalone -daemon /etc/schema-registry/connect-avro-standalone.properties /etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
Verify the Connectors and Topic Have Been Created
Check the logs to see that the connector is finished creating.
cat /logs/connectStandalone.out | grep -i "finished"
List the connectors.
curl -s localhost:8083/connectors
Describe the topic created by the connector.
kafka-topics --list --zookeeper localhost:2181 | grep test-sqlite-jdbc
Start a Kafka Consumer and Write New Data to the Database
Start a new Console Consumer:
kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9092 --topic test-sqlite-jdbc-accounts --from-beginning
In a new terminal, write some new values to the database table.
sudo docker exec -it sqlite-demo //bin//bash cd /tmp sqlite3 test.db INSERT INTO accounts(name) VALUES('william'); .quit
What's a lab?
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.