Configuration Scopes

The Cloudflow configuration model allows to set the scope of the settings for a runtime or for a streamlet instance.

Configuring Streamlets Using the streamlet Scope

The configuration for a specific streamlet instance must be specified in a cloudflow.streamlets.[streamlet-name] scope. This scope can optionally contain three sections to configure the following aspects of a streamlet instance:

Table 1. Configurations available for the streamlet

category

key

steamlet config params

config-parameters.[config-parameter-name]

runtime-specific settings

config.[runtime].[runtime-specific-setting]

kubernetes settings

kubernetes.[kubernetes-object]

You can learn more about it in Configurations Available

The example below shows this structure in action:

cloudflow {
  streamlets {
    my-streamlet {
      config-parameters {                   (1)
        // config parameter values go here
        my-config-parameter = "some-value"
        another-config-parameter = "default-value"
        another-config-parameter = ${?MY_VAR}
      }
      config {                              (2)
        // runtime settings go here
        akka.loglevel = "DEBUG"
      }
      kubernetes {                          (3)
        pods.pod.containers.container {
          // kubernetes container settings go here
          resources {
            requests {
              memory = "512M"
            }
            limits {
              memory = "1024M"
            }
          }
        }
      }
    }
  }
}
1 streamlet-parameters section
2 runtime-specific section
3 kubernetes section

You need to specify at least one of the sections shown above.

Configuring a Runtime using the runtime Scope

Configuration for all streamlets of a runtime can be specified in a cloudflow.runtimes.[runtime].config section. The configuration specified in cloudflow.runtimes.[runtime].config is merged as fallback, streamlet specific configuration takes precedence. An example is shown below:

cloudflow {
  runtimes {
    akka {
      config {
        akka.loglevel = "DEBUG"
      }
      kubernetes {
        pods.pod.containers.container {
          // kubernetes container settings go here
          env = [
            {
              name = "JAVA_OPTS"
              value = "-XX:MaxRAMPercentage=40.0 -Djdk.nio.maxCachedBufferSize=1048576"
            }
          ]

          resources {
            requests {
              cpu = 2
              memory = "512M"
            }
            limits {
              memory = "1024M"
            }
          }
        }
      }
    }
  }
}

Another example shows that you can provide configuration for all the runtimes your application uses. In this case, will cover Akka and Spark:

cloudflow.runtimes {
  spark.config {
    spark.driver.memoryOverhead=384
  }
  akka.config {
    akka {
      log-level = "INFO"
    }
  }
}