Project Structure

Cloudflow uses and extends sbt, the Scala Build Tool, to support the building, packaging, and local execution of applications. The project structure of a Cloudflow application follows the general conventions and folder structure of an sbt project with a few additional considerations.

The Essential Structure

To create any Cloudflow application, we require the following folder structure and files that we describe next:

The structure of a Cloudflow project
.
├── build.sbt                 (1)
├── project                   (2)
│   ├── build.properties      (3)
│   ├── cloudflow-plugins.sbt (4)
│   └── plugins.sbt           (5)
├── src                       (6)
│   └── main
└── target-env.sbt            (7)
1 The build.sbt file specifies the build configuration, project dependencies, and eventual sub-projects.
2 The project\ folder contains additional configuration for the project build, like plugins and sbt versions. This folder is mandatory.
3 The build.properties file contains the sbt version
4 The cloudflow-plugins.sbt file contains the dependency for the Cloudflow plugin for sbt
5 In the plugins.sbt file
6 The src folder contains the code for the application.
7 The target-env.sbt file contains the configuration for the Docker registry to use when publishing the application Docker containers.

You might recognize this structure as the typical sbt project with a few additional components. Two particular files that require your attention:

  • The cloudflow-plugins.sbt adds the repository and plugin version for the Cloudflow sbt plugins and,

  • the target-env.sbt contains the configuration of the Docker registry to use for publishing the resulting Docker images.

In the following sections, we explore the Cloudflow plugin in more detail. We also review the project setup required for single and multi-runtime projects.

The Cloudflow Plugin

The Cloudflow plugin is the primary enabler of any Cloudflow application development. Internally, it contains several sbt plugin definitions that support the building of applications and enable additional functions such as the local sandbox, that lets you run the applications on the local machine.

We add the Cloudflow plugin to our application by creating a cloudflow-plugins.sbt file in the project/ folder.

val latestVersion = {
  sys.env.get("CLOUDFLOW_VERSION").filter(_.nonEmpty).fold(
    sbtdynver.DynVer(None, "-", "v")
      .getGitDescribeOutput(new java.util.Date())
      .fold(throw new Exception("Failed to retrieve version"))(_.version("-"))
  )(identity)
}

addSbtPlugin("com.lightbend.cloudflow" % "sbt-cloudflow" % latestVersion) (1)
1 Adds the sbt-cloudflow plugin with a given version to the build definition.
The version of the sbt-cloudflow plugin defines which version of Cloudflow you are using. It must match with the version installed in your Kubernetes cluster.

Once the sbt-cloudflow plugin is added to the project, a number of plugins become available to your application. These plugins are used to enable different facets of an application.

Application-level Plugins

The application-level plugins provided by sbt-cloudflow are:

The CloudflowApplicationPlugin

The CloudflowApplicationPlugin is a SBT plugin for building and publishing the application. It provides a target buildApp that validates the application against its blueprint, creates and publishes all the Docker images required, and generates the application CR, a JSON document that describes the application and is used by the Cloudflow CLI to initiate the application deployment on a Kubernetes cluster.

There must be at-most-one CloudflowApplicationPlugin enabled in a Cloudflow application.

The CloudflowLibraryPlugin

The CloudflowLibraryPlugin is used when we want to create generic libraries that use Cloudflow concepts, such as data definitions (e.g. Codec), AVRO support, Protobuf support, etc.

The CloudflowLocalRunnerPlugin

The CloudflowLocalRunnerPlugin adds the runLocal task to sbt that allows you to run complete Cloudflow application in your local development environment.

The CloudflowLocalRunnerPlugin does not require an explicit declaration in the build file. It is an autoplugin that becomes enabled when the CloudflowApplicationPlugin is present.

Runtime-specific Plugins

Cloudflow provides out-of-the-box support for Akka with Docker base images and runners that are able to launch each specific technology stack on Kubernetes.

To develop Streamlets for a specific runtime, we add the corresponding plugin to the project. To ensure proper isolation of the different runtimes, we require a different structural approach when using only one runtime or when mixing Streamlets implementations that use two or more runtimes. We explore these differences in the next section. The Akka plugins is:

CloudflowAkkaPlugin

SBT plugin for creating Akka-based streamlets.

Each runtime plugin uses its own default base image. It’s lightbend/akka-base for Akka. It could be the case that you’re working against your own container registry and therefore prefer to point to your copy or customized version of our base images. To use your base images, set cloudflowAkkaBaseImage in settings of the CloudflowAkkaPlugin in your application build.sbt.

In an any streamlet the default base image value is adoptopenjdk/openjdk8:alpine. To override this base image in your cloudflow application go to its build.sbt and set up a value in settings such as:

lazy val sampleApp = (project in file("."))
    .enablePlugins(CloudflowApplicationPlugin)
    .settings(
      cloudflowDockerBaseImage := "myRepositoryUrl/myRepositoryPath:adoptopenjdk/openjdk11:alpine",

Single Runtime Applications

Applications that use a single runtime are the simplest form of a Cloudflow application project structure and build definition. As we see in the next section, we can also combine Streamlets that use different runtimes in a single application, but that requires a slightly more complex build structure.

Let’s explore how to declare a Cloudflow application in a build definition with an example:

Single Runtime Setup
import sbt._
import sbt.Keys._

lazy val sensorData =  (project in file("."))
    .enablePlugins(CloudflowApplicationPlugin, CloudflowAkkaPlugin) (1)
    .settings(
1 use enablePlugins to activate the Cloudflow plugins for this project.
See the complete example in the examples folder of the Cloudflow project

We provide templates to get started with a Cloudflow project that uses a single runtime. See templates folder of the Cloudflow project

Multiple Runtime Applications

Applications that combine multiple streamlet runtimes must be organized using sbt multi-projects builds. Streamlets for each runtime must added be into their own separate sub-project.

You can find how-to on Cloudflow contrib project.

Where to go next?

You can learn more about the Schema-First approach used in Cloudflow