Get monitoring, observability, online education,
and expert support from Lightbend.
Learn More

cloudflow deploy

Deploys a Cloudflow application to the cluster.

Prerequisites

To deploy a Flink or Spark application it is necessary that a PVC named cloudflow-flink' or `cloudflow-spark already exists in a namespace that is equal to the application name. This name is exactly the same as the json file you’ll use to deploy, excluding the extension. e.g. For a file swiss-knife.json the namespace would be swiss-knife. Cloudflow will automatically mount this PVCs in the streamlets themselves to allow any job, Flink or Spark, to store its state.

Synopsis

Deploys a Cloudflow application to the cluster.

cloudflow deploy [flags]

Configuration files in HOCON format can be passed through with the --conf flag. Configuration files are merged by concatenating the files passed with --conf flags. The last --conf [file] argument can override values specified in earlier --conf [file] arguments. In the example below, where the same configuration path is used in file1.conf and file2.conf, the configuration value in file2.conf takes precedence, overriding the value provided by file1.conf:

$ kubectl cloudflow deploy swiss-knife.json --conf file1.conf --conf file2.conf

It is also possible to pass configuration values as command line arguments, as [config-key]=value pairs separated by a space. The [config-key] must be an absolute path to the value, exactly how it would be defined in a config file. Some examples:

$ kubectl cloudflow deploy target/swiss-knife.json cloudflow.runtimes.spark.config.spark.driver.memoryOverhead=512
$ kubectl cloudflow deploy target/swiss-knife.json cloudflow.streamlets.spark-process.config-parameters.configurable-message='SPARK-OUTPUT:'

The arguments passed with [config-key]=[value] pairs take precedence over the files passed through with the --conf flag.

As pointed out in the prerequisites some existing PVC are required to deploy a Flink or a Spark application. Choosing which PVC to use can be also configured through --conf and this will override the default values mentioned above. Here’s an example:

cloudflow.runtimes.flink.kubernetes.pods.pod {
	volumes {
		foo {
			pvc {
				name = cloudflow-another-name
				read-only = false
			}
		}
	}
	containers.container {
		volume-mounts {
			foo {
				mount-path = "/mnt/flink/storage"
				read-only = false
			}
		}
	}
}

This will use then cloudflow-another-name PVC instead of the default cloudflow-flink. You may override the default PVC configuration as long as read-only remains false and the mount-path is equal to /mnt/[runtime]/storage, where [runtime] can be flink or spark. This configuration can only be applied to runtimes, not individual streamlets.

The command supports a flag --scale to specify the scale of each streamlet on deploy in the form of key/value pairs ('streamlet-name=scale') separated by comma.

$ kubectl-cloudflow deploy call-record-aggregator.json --scale cdr-aggregator=3,cdr-generator1=3

Streamlet volume mounts can be configured using the --volume-mount flag. The flag accepts one or more key/value pair where the key is the name of the volume mount, specified as [streamlet-name].[volume-mount-name], and the value is the name of a Kubernetes Persistent Volume Claim, which needs to be located in the same namespace as the Cloudflow application, e.g. the namespace with the same name as the application.

$ kubectl cloudflow deploy call-record-aggregator.json --volume-mount my-streamlet.mount=pvc-name

It is also possible to specify more than one "volume-mount" parameter.

$ kubectl cloudflow deploy call-record-aggregator.json --volume-mount my-streamlet.mount=pvc-name --volume-mount my-other-streamlet.mount=pvc-name

You can optionally provide credentials for the docker registry that hosts the images of the application by using the --username flag in combination with either the --password-stdin or the --password flag.

If no credentials are needed, for example, if the cluster already has credentials configured or if the registry does not require authentication to pull an image, use the '--no-registry-credentials' flag.

The --password-stdin flag is preferred because it is read from stdin, which means that the password does not end up in the history of your shell. One way to provide the password via stdin is to pipe it from a file:

$ cat key.json | kubectl cloudflow deploy call-record-aggregator.json --username _json_key --password-stdin

You can also use --password, which is less secure:

$ kubectl cloudflow deploy call-record-aggregator.json --username _json_key -password "$(cat key.json)"

If you do not provide a username and password, you will be prompted for them the first time you deploy an image from a certain docker registry. The credentials will be stored in a Kubernetes "image pull secret" and linked to the Cloudflow service account. Subsequent usage of the deploy command will use the stored credentials.

You can update the credentials with the "update-docker-credentials" command.

Examples

$ kubectl cloudflow deploy call-record-aggregator.json

Options

  --conf stringArray               Accepts one or more files in HOCON format.
  -h, --help                       help for deploy
  --no-registry-credentials        Use this flag if the Kubernetes cluster already has credentials configured for the Docker registry where the Cloudflow application images are located, or if the registry is public and requires no authentication.
  -p, --password string            docker registry password.
      --password-stdin             Take the password from stdin
      --scale stringToInt          Accepts key/value pairs for replicas per streamlet (default [])
  -u, --username string            docker registry username.
      --volume-mount stringArray   Accepts a key/value pair separated by an equal sign. The key should be the name of the volume mount, specified as '[streamlet-name].[volume-mount-name]'. The value should be the name of an existing persistent volume claim.

SEE ALSO