Migration Guide 2.0.x to 2.1.x
The most notable new feature in version 2.1.x is the native Flink and Spark integration support.
When migrating from 2.0.x to 2.1.x, it is required to update the Cloudflow operator, update the
kubectl-cloudflow plugin, update the
sbt-cloudflow plugin and rebuild your application.
To take advantage of the native integration feature, you will need to take the following steps:
Undeploy the existing application from your cluster with
kubectl cloudflow undeploy
Update the cloudflow operator to 2.1.x using the helm chart, as described in Upgrading Cloudflow
Update kubectl-cloudflow plugin to 2.1.x, as described in xref:get-started:prepare-development-environment.adoc#_download_and_install_the_cloudflow_cli
Rebuild your project with
sbt buildApp, use version 2.1.x of the sbt-cloudflow plugin, use the native plugin(s) from the cloudflow-contrib project, as documented in Cloudflow contrib
The cloudflow-contrib project describes the steps you need to take next, in summary:
- Deploy your application with
kubectl cloudflow deploy
- Setup RBAC for a service account that will be used for native integration, this has to be done once, per application.
- Deploy Flink and Spark streamlets using the example
redeploy-application.sh scripts in
cloudflow-contrib/example-scripts/spark-cli directories. Please note that these scripts provide a good starting point for using native Flink integration, you can customize these to your specific environment and requirements.
Uninstall the Flink and Spark operators with the commands below:
helm uninstall --namespace cloudflow flink-operator helm uninstall --namespace cloudflow spark-operator
From version 2.0.20 and onward, Docker images will be built on top of the standard