Streaming Data Pipelines on Kubernetes

Cloudflow enables you to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes. With Cloudflow, streaming applications are comprised of small composable components wired together with schema-based contracts. Cloudflow can dramatically accelerate streaming application development—​reducing the time required to create, package, and deploy—​from weeks to hours.

With its powerful abstractions, Cloudflow allows you to easily define the most complex streaming applications:

  • Develop: Focus only on business logic, leave the boilerplate to us.

  • Build: We provide all the tooling for going from business logic to a deployable Docker image.

  • Deploy: We provide Kubernetes tooling to deploy your distributed system with a single command, and manage durable connections between processing stages.

  • Operate: With a Lightbend subscription, you get all the tools you need to provide insights, observability, and lifecycle management for evolving your distributed streaming application.

Cloudflow integrates with popular streaming engines like Akka, Spark and Flink. It also comes with a powerful CLI tool to easily manage, scale and configure streaming applications at runtime. In a nutshell, Cloudflow is an application development toolkit composed of:

  • An API definition for Streamlet, the core abstraction in Cloudflow.

  • An extensible set of runtime implementations for Streamlet(s). Cloudflow provides support for popular streaming runtimes, like Spark’s Structured Streaming, Flink, and Akka.

  • A Streamlet composition model driven by a blueprint definition.

  • A sandbox execution mode that accelerates the development and testing of your applications.

  • A set of sbt plugins that are able to package your application into a deployable container.

  • The Cloudflow operator, a Kubernetes operator that manages the application lifecycle on Kubernetes.

  • A CLI, in the form of a kubectl plugin, that facilitates manual and scripted management of the application.