Installation Prerequisites
This guide shows how to prepare your environment to install Cloudflow, using the Helm
and kubectl
command-line tools.
It also shows how to install Kafka, Spark, and Flink operators that integrate with Cloudflow.
Prerequisites
CLI Tools
Make sure you have the following prerequisites installed before continuing:
-
Helm, version 3 or later (check with
helm version
) -
Kubectl
Before proceeding, make sure that kubectl
is correctly installed and access the Kubernetes cluster to install Cloudflow. This can be done with the following command:
kubectl version
Kafka
Kafka is used by Cloudflow to connect streamlets together in a blueprint.
If you intend to connect streamlets in this way then at least one Kafka cluster should be available before installation.
Cloudflow may be used without Kafka (for example, when your application contains a single streamlet, or an Akka cluster), but if your team intends to connect streamlets together and not include Kafka connection information in each topic they define then it’s recommended to define a default
Kafka cluster at install time.
The Kafka broker bootstrap configuration string is a comma-separated list of host/port pairs used by Cloudflow to establish the connection to a Kafka cluster. The configuration string should have the following format:
broker-1-address:broker-1-port, broker-2-address;broker-2-port
If you want to test Cloudflow and need a Kafka cluster, we recommend using Strimzi, a third-party Kafka operator that can create and manage Kafka clusters.
See Installing Kafka with Strimzi as a guide on how to configure and install a Kafka cluster using Strimzi.
Storage requirements (for use with Spark or Flink)
If you plan to write Cloudflow applications using Spark or Flink, the Kubernetes cluster will need to have a storage class of the ReadWriteMany
type installed.
For testing purposes, we suggest using the NFS Server Provisioner, which can be found here: NFS Server Provisioner Helm chart
We’ll install the nfs chart in the cloudflow
namespace, if it does not exist yet, create the cloudflow
namespace:
kubectl create ns cloudflow
Add the Stable
Helm repository and update the local index:
helm repo add stable https://charts.helm.sh/stable helm repo update
Install the NFS Server Provisioner using the following command:
Depending on your Kubernetes configuration, you may want to adjust the values used during the install. Please see NFS Server Provisioner configuration options. |
helm install nfs-server-provisioner stable/nfs-server-provisioner \ --namespace cloudflow
The result of the installation is shown below, the NFS Server provisioner pod is running and the new storage class exists.
$ kubectl get pods -n cloudflow NAME READY STATUS RESTARTS AGE nfs-server-provisioner-0 1/1 Running 0 25s $ kubectl get sc NAME PROVISIONER AGE nfs cloudflow-nfs 29s standard (default) kubernetes.io/gce-pd 2m57s
- NOTE
-
The documented NFS storage class is very portable and has been verified to work on GKE, EKS, AKS and Openshift.
More Storage Options
Continue with More Storage Options
Next: Installing Cloudflow
Continue with Installing Cloudflow