Installation Prerequisites

To prepare your environment to install Cloudflow, you will need:

  • Helm version 3 or higher (check with helm version)

  • Kubectl

  • Kafka

  • If using Spark or Flink, storage configuration

Helm

Follow the documentation to install Helm version 3 or higher.

kubectl

Before proceeding, make sure that kubectl for your Kubernetes distribution is correctly installed and access the Kubernetes cluster to install Cloudflow. Cloudflow supports a variety of distributions, see versions.

Check the version with the following command:

kubectl version

Kafka

Kafka is used by Cloudflow to connect streamlets together in a blueprint. If you intend to connect streamlets in this way, at least one Kafka cluster should be available before installation. Cloudflow may be used without Kafka (for example, when your application contains a single streamlet, or an Akka cluster), but if your team intends to connect streamlets together and not include Kafka connection information in each topic they define then it’s recommended to define a default Kafka cluster at install time.

The Kafka broker bootstrap configuration string is a comma-separated list of host/port pairs used by Cloudflow to establish the connection to a Kafka cluster. The configuration string should have the following format:

broker-1-address:broker-1-port, broker-2-address;broker-2-port

If you want to test Cloudflow and need a Kafka cluster, we recommend using Strimzi during development, a third-party Kafka operator that can create and manage Kafka clusters.

See Installing Kafka with Strimzi as a guide on how to configure and install a Kafka cluster using Strimzi.

Please make sure to choose a suitable Kafka service that matches your experience with running Kafka in production.

If you plan to write Cloudflow applications using Spark or Flink, the Kubernetes cluster will need to have a storage class of the ReadWriteMany type installed.

For testing purposes, we suggest using the NFS Server Provisioner, which can be found here: NFS Server Provisioner Helm chart

We’ll install the nfs chart in the cloudflow namespace, if it does not exist yet, create the cloudflow namespace:

kubectl create ns cloudflow

Add the Stable Helm repository and update the local index:

helm repo add stable https://charts.helm.sh/stable
helm repo update

Install the NFS Server Provisioner using the following command:

Depending on your Kubernetes configuration, you may want to adjust the values used during the install. Please see NFS Server Provisioner configuration options.
helm install nfs-server-provisioner stable/nfs-server-provisioner \
  --namespace cloudflow

The result of the installation is shown below, the NFS Server provisioner pod is running and the new storage class exists.

$ kubectl get pods -n cloudflow
NAME                       READY   STATUS    RESTARTS   AGE
nfs-server-provisioner-0   1/1     Running   0          25s

$ kubectl get sc
NAME                 PROVISIONER            AGE
nfs                  cloudflow-nfs          29s
standard (default)   kubernetes.io/gce-pd   2m57s
The documented NFS storage class is very portable and has been verified to work on GKE, EKS, AKS and Openshift.
The default build tool for Cloudflow applications is sbt but there is support for using Maven to build as well. If you are going to use Maven, make sure to have it installed as well.