Experiments

_Kafka_command_line

2 min read

Setup Zookeeper and Kafka

Install MacOS

brew install kafka

Setup Configuration

We can create permanent data directory and point the config file to it.

  • Inside zookeeper.properties (for Zookeeper) file edit as follows:
    • dataDir : ~/data/zookeeper
  • Inside server.properties (for Kafka) file edit as follows:
    • log.dirs : ~/data/kafka

Start Zookeeper

zookeeper-server-start <path to zookeeper.properties config file>
zookeeper-server-start config/zookeeper.properties

Start Kafka

kafka-server-start <path to server.properties config file>
kafka-server-start config/server.properties

Topics

# List Topics
kafka-topics --zookeeper 127.0.0.1:2181 --list
 
# Create Topic
kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1
 
# Create Topic
kafka-topics --zookeeper 127.0.0.1:2181 --topic second_topic --create --partitions 6 --replication-factor 1
 
# Describe Topic
kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --describe

Modern Command line without zookeeper

Topics

# List Topics
kafka-topics --bootstrap-server localhost:9092 --list
 
# Create Topic
kafka-topics --bootstrap-server localhost:9092 --describe --topic first_topic
 
# Delete topic data
kafka-topics --bootstrap-server localhost:9092 --delete --topic first_topic

Producer

# Use ctrl+C to complete sending data
kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic
 
## Produce JSON
jq -rc . data.json | kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic

Experiments!!!

jq -rc . order-release.json | kafka-console-producer --bootstrap-server localhost:9092 --topic dev.us.by-realm-2.wms.orderRelease.event --property parse.key=true --property key.separator=":"
 
 
(cat header.properties; jq -rc . order-release.json) | kafka-console-producer --bootstrap-server localhost:9092 --topic dev.us.by-realm-2.wms.orderRelease.event --property parse.headers=true
 
headers.delimiter=\t
headers.separator=,
headers.key.separator=:
 
 
(cat header.properties | tr '\n' ','; jq -rc . order-release.json)
tr '\n' ','
 
{
    cat header.properties | tr '\n' ','
    echo -e "\t"
    jq -rc . order-release.json
} | kafka-console-producer --bootstrap-server localhost:9092 --topic dev.us.by-realm-2.wms.orderRelease.event --property parse.headers=true