Kafka

kafka

Deploy kafka in docker

https://hellokube.dev/posts/three-ways-zookeepeerless-kafka/

docker run -it --name kafka-zkless -p 9092:9092 -e LOG_DIR=/tmp/logs quay.io/strimzi/kafka:latest-kafka-3.1.0-amd64 /bin/sh -c 'export CLUSTER_ID=$(bin/kafka-storage.sh random-uuid) && bin/kafka-storage.sh format -t $CLUSTER_ID -c config/kraft/server.properties && bin/kafka-server-start.sh config/kraft/server.properties'

https://github.com/lensesio/fast-data-dev

Cheat sheet

https://github.com/lensesio/kafka-cheat-sheet

Install

https://www.youtube.com/watch?v=mEHTngy8-bY

https://snourian.com/kafka-kubernetes-strimzi-part-1-creating-deploying-strimzi-kafka/

1. Deploying the Cluster Operator to manage our Kafka cluster

2. Deploying the Kafka cluster with ZooKeeper using the Cluster Operator. Topic and User Operators can be deployed in this step with the same deploy file or you can deploy them later.

wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.28.0/strimzi-0.28.0.zip
/cd strimzi
sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml

kubectl create ns kafka
kubectl apply -f strimzi/install/cluster-operator/ -n kafka
nano nano strimzi/examples/kafka/kafka-persistent.yaml    # change 100G to 10G
kubectl apply -f strimzi/examples/kafka/kafka-persistent.yaml -n kafka

create and customize the Kafka deployment file based on the examples from Strimzi, and deploy it

kubectl apply -f strimzi/kafka-deployment.yaml -n kafka

Get the DNS name of the Kafka Cluster service in order to connect to it:

kubectl get svc -n kafka

You can access a service by its DNS name:

<service.name>.<namespace.name>.svc.cluster.local

The “svc.cluster.local” can be omitted in our case. If you are accessing the service from its own namespace (kafka namespace as we used in this post), you can access the Kafka cluster by its service name only:

my-cluster-kafka-bootstrap:9092.

But if you are accessing it from outside of its namespace, you need to include the namespace to the DNS name too:

my-cluster-kafka-bootstrap.kafka:9092

Run a Kafka producer and write some texts on the console after connecting to send to the topic:

https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/

kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.28.0-kafka-3.1.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --broker-list my-cluster-kafka-bootstrap:9092 --topic my-topic

Note: if we include -n kafka right after the kubectl command, you can omit the namespace from the service address: –broker-list my-cluster-kafka-bootstrap:9092.

Type a message into the console where the producer is running and press Enter. The message should be in my-topic now. Now, we need a consumer to consume the message.

run a Kafka consumer:

kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.28.0-kafka-3.1.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning

If you have added an external listener (like nodeport) to your Kafka Cluster and want to test it from outside of the Kubernetes, follow these steps:

1. Download the latest binary release of Kafka from here.

2. Get the IP of the minikube by running minikube ip* command* (mine is 192.168.99.105).

   1. get node port

   kubectl get service my-cluster-kafka-external-bootstrap -n kafka -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}'
   

   1. get ip address

   kubectl get nodes --output=jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}'
   

3. Run kubectl get svc -n kafka and find the exposed port of the kafka-external-bootstrap (highlighted with blue in the picture above)

4. The kafka-console-producer and kafka-console-consumer are in the /bin directory of the downloaded package (for Windows, navigate to /bin/windows)

5. Like above, fire up your console producer and consumer with the below commands (Windows commands are the same) and test your cluster from outside of the Kubernetes:

kafka-console-producer.sh --broker-list 192.168.99.105:30825 --topic my-topic
kafka-console-consumer.sh --bootstrap-server 192.168.99.105:30825 --topic my-topic --from-beginning