Deploying Apache Kafka on Kubernetes Local

Kafka is a powerful framework of message queue and more, so let’s install it on our local environment to test!

Our ingredients:

1 – Helm;

2 – Kubernetes;

3 – Helm Apache Kafka.

Let’s init our Helm support:

helm init

Let’s download a copy of the Helm Kafka

git clone https://github.com/confluentinc/cp-helm-charts.gitcp cp-helm-charts/values.yaml .

If you desire, you can expose kafka externally editing the values.yaml file and adding there to the cp-kafka part.

  nodeport:    enabled: true    servicePort: 19092    firstListenerPort: 31090  configurationOverrides:    "offsets.topic.replication.factor": "3"    "advertised.listeners": |-     EXTERNAL://localhost:$((31090 + ${KAFKA_BROKER_ID}))    "listener.security.protocol.map": |-     PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

Let’s install kafka inside a namespace called kafka

helm install my-confluent -n kafka cp-helm-charts -f ./values.yaml

If everything is ok you can check:

kubectl get all -n kafka

We can use the kafka-client to manipulate the kafka, to install it:

cd cp-helm-chart/exampleskubectl apply -f kafka-client.yaml

To iniciate the client:

kubectl exec -it kafka-client -- /bin/bashcd /usr/bin

To create a topic example on Kafka:

kafka-topics --zookeeper my-confluent-cp-zookeeper-headless:2181 --topic my-confluent-topic --create --partitions 1 --replication-factor 1 --if-not-exists

To put a message on kafka topic:

echo "hello world" | kafka-console-producer --broker-list my-confluent-cp-kafka-headless:9092 --topic my-confluent-topic

To consume a message of our topic:

kafka-console-consumer --bootstrap-server my-confluent-cp-kafka-headless:9092 --topic my-confluent-topic --from-beginning

Voila! our kafka is running perfectly!

Reference: https://tsuyoshiushio.medium.com/local-kafka-cluster-on-kubernetes-on-your-pc-in-5-minutes-651a2ff4dcde