From the creators of Cloudflow, get monitoring, observability, online education, and expert support from Lightbend. Learn More
The Lightbend Family

Setting up a GKE cluster

ON THIS PAGE

Detailed instructions on how to install a GKE cluster with Cloudflow are described in the Cloudflow installer section of the GitHub repo.

To install a new cluster, clone the repo in your local system by selecting the latest release branch or download the latest release file from here, go to the installer/ folder and follow the instructions below.

  • Run gcloud init if this is a new cluster and you are creating a GKE cluster for the first time. This command initializes the gcloud system on your local and performs necessary settings of gcloud credentials. You can have a look at the details in the GKE documentation.

    NOTE

    This can be done once you have a valid Google account and have access to a project set up for you by the administrator.

  • Create a GKE cluster with a specific name ./create-cluster-gke.sh <cluster-name>

  • Install Cloudflow in the designated cluster ./install.sh <cluster-name> gke

This will take you through the process of installing Cloudflow in the cluster. In course of this it will ask a few questions regarding the storage class to be installed. The following should be the response to these:

Select a storage class for workloads requiring persistent volumes with access mode 'ReadWriteMany'.
Examples of these are Spark and Flink checkpointing and savepointing.

   NAME                 PROVISIONER                    SUPPORTS RW?                  DEFAULT?
1. standard             kubernetes.io/gce-pd           Unknown                       -
2. nfs                  lightbend.com/nfs              Unknown                       -
> 2

Select a storage class for workloads requiring persistent volumes with access mode 'ReadWriteOnce'.
Examples of these are Kafka, Zookeeper, and Prometheus.

   NAME                 PROVISIONER                    SUPPORTS RW?                  DEFAULT?
1. standard             kubernetes.io/gce-pd           Verified                      -
2. nfs                  lightbend.com/nfs              Unknown                       -
> 1

It then takes you through the installation of the Cloudflow operator, the strimzi Kafka operator, the Spark operator and the Flink operator. If things go ok, you will see something like the following on your console:

+------------------------------------------------------------------------------------+
|                      Installation of Cloudflow has completed                       |
+------------------------------------------------------------------------------------+

NAME                                                         READY   STATUS              RESTARTS   AGE
cloudflow-flink-flink-operator-8588dbd8f4-gsjnm              1/1     Running             0          70s
cloudflow-operator-57f47676f7-svbvj                          0/1     ContainerCreating   0          3s
cloudflow-sparkoperator-sparkoperator-69669fdd54-88brw       0/1     ContainerCreating   0          35s
strimzi-cluster-operator-7ff64d4b7-c9r2n                     1/1     Running             0          51s

The complete installtion take a bit of a time and you can follow the progress using kubectl command. All operators are installed in the namespace cloudflow. When everything is installed, you will see the following:

$ kubectl get pods -n cloudflow
NAME                                                         READY   STATUS    RESTARTS   AGE
cloudflow-flink-flink-operator-8588dbd8f4-gsjnm              1/1     Running   0          3m26s
cloudflow-nfs-nfs-6f8855d44-vdhhj                            1/1     Running   0          109s
cloudflow-operator-57f47676f7-svbvj                          1/1     Running   0          2m19s
cloudflow-sparkoperator-sparkoperator-69669fdd54-88brw       1/1     Running   0          2m51s
cloudflow-strimzi-entity-operator-5bc9695975-584fb           1/2     Running   0          27s
cloudflow-strimzi-kafka-0                                    2/2     Running   0          79s
cloudflow-strimzi-kafka-1                                    2/2     Running   0          79s
cloudflow-strimzi-kafka-2                                    2/2     Running   0          79s
cloudflow-strimzi-zookeeper-0                                2/2     Running   0          2m16s
cloudflow-strimzi-zookeeper-1                                2/2     Running   0          2m15s
cloudflow-strimzi-zookeeper-2                                2/2     Running   0          2m15s
strimzi-cluster-operator-7ff64d4b7-c9r2n                     1/1     Running   0          3m7s

This completes the process of installing the GKE cluster along with all necessary operators on it.

What’s next

Now, we are ready to Deploy to a GKE cluster.