From the creators of Cloudflow, get monitoring, observability, online education, and expert support from Lightbend. Learn More
The Lightbend Family

More Storage Options

Verified storage and example configuration

The following tabs describe the storage classes that Lightbend has verified and tested on Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS) and RedHat OpenShift (3.11). They include examples of the required .yaml file that should be used to update the Kubernetes cluster. There’s no need to set the namespace when updating the cluster.

AKS

The AzureFile storage class supports ReadWriteMany access mode and is verified to work with Cloudflow on Azure Kubernetes Service (AKS).

An important thing to note here is that the uid and gid mount options should be configured to match those used by the Cloudflow application pods, e.g. they should both have the value 185. If this is misconfigured it will lead to application runtime errors when Spark and/or Flink are not permitted to write their intermediate state to disk. 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=185
    - gid=185
    - mfsymlinks
    - nobrl
    - cache=none
parameters:
  skuName: Standard_LRS

For more information about configuring AzureFile, please read its documentation.

EKS

Amazon Elastic File System (EFS) supports the ReadWriteMany access mode and is verified working with Cloudflow on Amazon Elastic Kubernetes Service (EKS).

For more information about configuring Amazon Elastic File System, please read its dcumentation.

You must configure the rules as discussed here in security groups to enable traffic between an EC2 instance of the Kubernetes Cluster and a mount target (and thus the file system). If you do not do this, Cloudflow will not be able to use EFS.
Openshift

GlusterFS provisioner supports the ReadWriteMany access mode and is verified working with Cloudflow on Redhat Openshift.

The GlusterFS installation will create the following storage class. Note that the name of the storage class has here been set to glusterfs, this may be different in your installation.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: glusterfs-storage
parameters:
  resturl: http://heketi-storage.glusterfs.svc:8080
  restuser: admin
  secretName: heketi-storage-admin-secret
  secretNamespace: glusterfs
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete
volumeBindingMode: Immediate

For more information about configuring GlusterFS on Openshift, please read its documentation.

What’s next

With this overview of storage, it’s time to check Installing Cloudflow.