Using Kafka Connect to Capture Data from a Relational Database

Run the following command to start the container and connect to it.

sudo docker run -ti --rm --name sqlite-demo --network host confluentinc/docker-demo-base:3.3.0

Start Kafka with the following command.

cd /tmp
confluent start

Install SQLite3 with the following commands.

apt-get update

apt-get install sqlite3

Create a new database and table with the following command.

sqlite3 test.db

CREATE TABLE accounts (
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
name VARCHAR(255));

Insert the values into your database table using the following command.

INSERT INTO accounts(name) VALUES('chad');
INSERT INTO accounts(name) VALUES('terry');
.quit

Use the following command to stop Kafka Connect from running in distributed mode.

confluent stop connect

Use the following commnand to start Kafka Connect in standalone mode.

connect-standalone -daemon /etc/schema-registry/connect-avro-standalone.properties /etc/kafka-connect-jdbc/source-quickstart-sqlite.properties

Check the logs to see that the connector is finished creating.

cat /logs/connectStandalone.out | grep -i "finished"

List the connectors.

curl -s localhost:8083/connectors

Describe the topic created by the connector.

kafka-topics --list --zookeeper localhost:2181 | grep test-sqlite-jdbc

Start a new Console Consumer:

kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9092 --topic test-sqlite-jdbc-accounts --from-beginning

In a new terminal, write some new values to the database table.

sudo docker exec -it sqlite-demo //bin//bash

cd /tmp
sqlite3 test.db
INSERT INTO accounts(name) VALUES('william');
.quit