Installing Kafka with Strimzi
This guide shows how to install Strimzi and create a Kafka cluster suitable for use with Cloudflow.
Strimzi is a Kubernetes operator that simplifies the process of running Apache Kafka.
Installing Strimzi
In this guide, we will use Helm to install Strimzi.
We are going to create Strimzi in the cloudflow namespace in this guide. Make sure that the namespace cloudflow exists before continuing. To create the cloudflow namespace, execute
kubectl create ns cloudflow
Add the Strimzi Helm repository and update the local index.
helm repo add strimzi https://strimzi.io/charts/ helm repo update
Install the latest version of Strimzi.
helm install strimzi strimzi/strimzi-kafka-operator --namespace cloudflow
After the install complete, the Strimzi Kafka operator should be running in the cloudflow namespace.
$ kubectl get pods -n cloudflow NAME READY STATUS RESTARTS AGE strimzi-cluster-operator-9968fd8c9-fhqmj 1/1 Running 0 17s
Creating a Kafka cluster using Strimzi
To create a Kafka cluster using Strimzi, we have to create a CustomResource of the kind Kafka in the namespace cloudflow.
Create the YAML file containing the custom resource using the following snippet:
cat > kafka-cluster.yaml << EOF
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: cloudflow-strimzi
namespace: cloudflow
spec:
kafka:
config:
auto.create.topics.enable: false
log.message.format.version: "2.3"
log.retention.bytes: 112197632
log.retention.hours: -1
offsets.topic.replication.factor: 3
transaction.state.log.min.isr: 2
transaction.state.log.replication.factor: 3
listeners:
plain: {}
tls: {}
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
replicas: 3
resources: {}
storage:
deleteClaim: false
size: 1Gi
type: persistent-claim
version: 2.4.0
zookeeper:
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
replicas: 3
resources: {}
storage:
deleteClaim: false
size: 1Gi
type: persistent-claim
EOFApply the YAML file with the Kafka custom resource to the cluster:
kubectl apply -f kafka-cluster.yaml
When the Strimzi Kafka operator has processed the custom resource, there should be six new pods in the cloudflow namespace. Three Kafka broker pods and three Zookeeper pods.
$ kubectl get pods -n cloudflow NAME READY STATUS RESTARTS AGE cloudflow-strimzi-kafka-0 2/2 Running 0 1m cloudflow-strimzi-kafka-1 2/2 Running 0 1m cloudflow-strimzi-kafka-2 2/2 Running 0 1m cloudflow-strimzi-zookeeper-0 1/1 Running 0 2m cloudflow-strimzi-zookeeper-1 1/1 Running 0 2m cloudflow-strimzi-zookeeper-2 1/1 Running 0 2m strimzi-cluster-operator-9968fd8c9-fhqmj 1/1 Running 0 5m
If you want to change any parameters of the Kafka cluster, edit the kafka-cluster.yaml file and apply it again to the cluster, as shown above. The Strimzi Kafka cluster operator will make the necessary changes when detecting an update of the custom resource.
|