p

cloudflow

streamlets

package streamlets

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. sealed trait AccessMode extends AnyRef
  2. final case class BooleanConfigParameter(key: String, description: String = "", defaultValue: Option[Boolean] = None) extends ConfigParameter with Product with Serializable
  3. case class BootstrapServersForTopicNotFound(topic: Topic) extends Exception with Product with Serializable
  4. trait Codec[T] extends Serializable
    Annotations
    @implicitNotFound( ... )
  5. trait CodecInlet[T] extends Inlet

    A handle to read and deserialize data into elements of type T.

  6. trait CodecOutlet[T] extends Outlet

    A handle to serialize elements of type T into a partitioned stream.

  7. trait ConfigParameter extends AnyRef

    Describes arguments that have to be supplied when deploying a Cloudflow application.

    Describes arguments that have to be supplied when deploying a Cloudflow application. Each streamlet in an application can require zero or more arguments. The configuration parameter contains information for the end user entering the values and logic to validate those values so they are entered correctly.

    Streamlet developers can create their own configuration parameters by deriving from ConfigParameter, the parameter can use validation logic from a fixed set of types derived from ValidationType.

    The developer can also re-use any of the following predefined configuration parameter types:

    Each of these pre-defined parameters uses one of the following validation types:

    Example of how to create a custom configuration parameter:

    final case class MilitaryTimeParameter(key: String, defaultValue: Option[String] = None) extends ConfigParameter[String] {
     val description: String = "This parameter validates that the users enter the time in 24h format."
     val validation = RegexpValidation("^(0[0-9]|1[0-9]|2[0-3]|[0-9]):[0-5][0-9]$")
     def toDescriptor = ConfigParameterDescriptor(key,description,validation,defaultValue)
    }

    Example on how to use a configuration parameter in a streamlet:

    class RecordSumFlow extends AkkaStreamlet {
      val recordsInWindowParameter = IntegerConfigParameter("records-in-window","This value describes how many records of data should be processed together, default 65 KB", Some(64 * 1024))
      override def configParameters = Set(recordsInWindowParameter)
    
      val inlet = AvroInlet[Metric]("metric")
      val outlet = AvroOutlet[SummedMetric]("summed-metric")
      val shape = StreamletShape.withInlets(inlet).withOutlets(outlet)
    
      def createLogic = new RunnableGraphStreamletLogic() {
        val recordsInWindow = streamletConfig.getInt(recordsInWindowParameter.key)
        def sumRecords(records: Seq[Metric]) : SummedMetric {....}
        def flow = FlowWithOffsetContext[Metric].grouped(recordsInWindow).map(sumRecords)
    
        def runnableGraph() = atLeastOnceSource(inlet).via(flow).to(atLeastOnceSink(outlet))
      }
    }
  8. final case class DoubleConfigParameter(key: String, description: String = "", defaultValue: Option[Double] = None) extends ConfigParameter with Product with Serializable
  9. sealed abstract class Dun extends Serializable

    Typically used together with Future to signal completion but there is no actual value completed.

    Typically used together with Future to signal completion but there is no actual value completed. More clearly signals intent than Unit and is available both from Scala and Java (which Unit is not).

    Adopted from akka to avoid streamlets depending on akka.

  10. case class DurationConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
  11. final case class ExceptionAcc(exceptions: Vector[Throwable]) extends Exception with Product with Serializable

    An exception to return when the runner returns an accumulated list of distinct exceptions.

  12. trait Inlet extends StreamletPort

    A handle to read data according to a schema.

  13. final case class IntegerConfigParameter(key: String, description: String = "", defaultValue: Option[Int] = None) extends ConfigParameter with Product with Serializable
  14. case class LoadedStreamlet(streamlet: Streamlet[StreamletContext], config: StreamletDefinition) extends Product with Serializable
  15. final case class MemorySizeConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
  16. trait Outlet extends StreamletPort

    A handle to write data according to a schema.

  17. case class PortMapping(port: String, topic: Topic) extends Product with Serializable

    Mapping between the port name and the topic

  18. final case class RegExpConfigParameter(key: String, description: String, pattern: String, defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
  19. final case class RegexpValidationType(regExpPattern: String) extends ValidationType with Product with Serializable
  20. final case class SchemaDefinition(name: String, schema: String, fingerprint: String, format: String) extends Product with Serializable

    Describes the schema.

    Describes the schema. name should be the unique name of the schema. schema is a string representation of the schema itself. (In the case of the avro format this is a json document) The fingerprint is a consistent hash of the schema. The format specifies the format of the schema. Unique names should be used for different formats. (In the case of Avro, format is "avro")

  21. abstract class Streamlet[Context <: StreamletContext] extends AnyRef
  22. trait StreamletAttribute extends AnyRef

    Describes that a streamlet requires specific configuration by the cloudflow platform.

    Describes that a streamlet requires specific configuration by the cloudflow platform.

    The configuration value will be added by the platform under the configPath path in the Streamlet configuration.

  23. trait StreamletContext extends AnyRef
  24. case class StreamletContextData(appId: String, appVersion: String, portMappings: Map[String, Topic], volumeMounts: Option[List[VolumeMount]] = None, config: Config) extends Product with Serializable
  25. class StreamletContextException extends Exception
  26. case class StreamletDefinition(appId: String, appVersion: String, streamletRef: String, streamletClass: String, portMappings: List[PortMapping], volumeMounts: List[VolumeMount], config: Config) extends Product with Serializable
  27. trait StreamletExecution extends AnyRef

    A handle to the running Streamlet.

  28. trait StreamletLoader extends AnyRef

    Functions to load a streamlet from its configuration through reflection.

  29. trait StreamletLogic[Context <: StreamletContext] extends Serializable
  30. trait StreamletPort extends AnyRef

    A named port handle handle to read or write data according to a schema.

  31. trait StreamletRuntime extends AnyRef

    A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g.

    A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g. "akka", "spark", etc.

    Implementations will usually be provided by a runtime support library such as cloudflow-akka, cloudflow-spark and cloudflow-flink.

  32. trait StreamletShape extends AnyRef
  33. final case class StringConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
  34. final case class Topic(id: String, config: Config) extends Product with Serializable
  35. case class TopicForPortNotFoundException(port: StreamletPort, streamletDefinition: StreamletDefinition) extends Exception with Product with Serializable
  36. sealed trait ValidationType extends AnyRef
  37. case class VolumeMount(name: String, path: String, accessMode: AccessMode) extends Product with Serializable

Value Members

  1. object AkkaClusterAttribute extends StreamletAttribute with Product with Serializable
  2. object BooleanConfigParameter extends Serializable

    Describes a boolean configuration parameter.

  3. object BooleanValidationType extends ValidationType with Product with Serializable
  4. object BootstrapInfo
  5. object ClassOps
  6. object CodecInlet
  7. object DoubleConfigParameter extends Serializable

    Describes a 64 bit floating point number.

  8. object DoubleValidationType extends ValidationType with Product with Serializable
  9. object Dun extends Dun with Product with Serializable
  10. object DurationConfigParameter extends Serializable

    Describes a time duration as defined by https://github.com/lightbend/config/blob/main/HOCON.md#duration-format

    Describes a time duration as defined by https://github.com/lightbend/config/blob/main/HOCON.md#duration-format

    The following units can be used with DurationConfigParameter

    ns, nano, nanos, nanosecond, nanoseconds us, micro, micros, microsecond, microseconds ms, milli, millis, millisecond, milliseconds s, second, seconds m, minute, minutes h, hour, hours d, day, days

    Example on how to extract a duration value using config:

    val parameter = DurationDurationConfigParameter("TTL","Time to live of packets sent out","2 m")
    ...
    val duration = config.getDuration(parameter.key)
  11. object DurationValidationType extends ValidationType with Product with Serializable
  12. object IntegerConfigParameter extends Serializable

    Describes an integer configuration parameter.

  13. object IntegerValidationType extends ValidationType with Product with Serializable
  14. object MemorySizeConfigParameter extends Serializable

    Describes a memory quantity as defined by https://github.com/lightbend/config/blob/main/HOCON.md#size-in-bytes-format

    Describes a memory quantity as defined by https://github.com/lightbend/config/blob/main/HOCON.md#size-in-bytes-format

    The following units can be used with MemorySizeConfigParameter

    "B", "b", "byte", "bytes", "kB", "kilobyte", "kilobytes", "MB", "megabyte", "megabytes", "GB", "gigabyte", "gigabytes", "TB", "terabyte", "terabytes", "PB", "petabyte", "petabytes", "EB", "exabyte", "exabytes", "ZB", "zettabyte", "zettabytes", "YB", "yottabyte", "yottabytes", "K", "k", "Ki", "KiB", "kibibyte", "kibibytes", "M", "m", "Mi", "MiB", "mebibyte", "mebibytes", "G", "g", "Gi", "GiB", "gibibyte", "gibibytes", "T", "t", "Ti", "TiB", "tebibyte", "tebibytes", "P", "p", "Pi", "PiB", "pebibyte", "pebibytes", "E", "e", "Ei", "EiB", "exbibyte", "exbibytes", "Z", "z", "Zi", "ZiB", "zebibyte", "zebibytes", "Y", "y", "Yi", "YiB", "yobibyte", "yobibytes",

    Example on how to extract a duration value using config:

    val parameter = MemorySizeMemorySizeConfigParameter("max-size-of-log","Maximum size of log before wrapping","8M")
    ...
    val memorySize = config.getMemorySize(parameter.key)
  15. object MemorySizeValidationType extends ValidationType with Product with Serializable
  16. object ReadOnlyMany extends AccessMode with Product with Serializable
  17. object ReadWriteMany extends AccessMode with Product with Serializable
  18. object RegExpConfigParameter extends Serializable

    Describes a parameter that can be used for custom validation.

    Describes a parameter that can be used for custom validation.

    Note: - This is a limited set of regular expressions compared to regular expressions supported by Java/Scala. - These regular expressions are only used for validation so capture groups are not supported.

    The type of regular expressions that can be used is described in: https://github.com/google/re2/wiki/Syntax

    Example:

    val durationParameter = RegExpConfigParameter("window-duration","The duration of a sampling window", """((\d{1,2}h\s?)?(\d{1,2}m\s?)?(\d{1,2}s\s?)?)|\d{1,2}""",Some("1m 20s"))
  19. object RoundRobinPartitioner extends (Any) ⇒ String with Serializable

    A round-robin partitioning function.

    A round-robin partitioning function. Elements written to a CodecOutlet that uses this partitioner will be distributed in round-robin fashion across the topic partitions.

  20. object ServerAttribute extends StreamletAttribute with Product with Serializable

    Describes that a Streamlet is a server, which will want to listen on a container port.

  21. object StreamletContextDataJsonSupport extends DefaultJsonProtocol

    Helper object for creating an instance of StreamletContextData from JSON.

  22. object StreamletDefinition extends Serializable
  23. object StreamletShape
  24. object StringConfigParameter extends Serializable

    Describes a UTF8 string

  25. object Topic extends Serializable
  26. object VolumeMount extends Serializable

Ungrouped