Packages

t

cloudflow.spark

SparkStreamlet

trait SparkStreamlet extends Streamlet[SparkStreamletContext] with Serializable

The base class for defining Spark streamlets. Derived classes need to override createLogic to provide the custom implementation for the behavior of the streamlet.

Here's an example:

// new custom `SparkStreamlet`
object MySparkProcessor extends SparkStreamlet {
  // Step 1: Define inlets and outlets. Note for the outlet you need to specify
  //         the partitioner function explicitly
  val in = AvroInlet[Data]("in")
  val out = AvroOutlet[Simple]("out", _.name)

  // Step 2: Define the shape of the streamlet. In this example the streamlet
  //         has 1 inlet and 1 outlet
  val shape = StreamletShape(in, out)

  // Step 3: Provide custom implementation of `SparkStreamletLogic` that defines
  //         the behavior of the streamlet
  override def createLogic() = new SparkStreamletLogic {
    override def buildStreamingQueries = {
      val dataset = readStream(in)
      val outStream = dataset.select($"name").as[Simple]
      val query = writeStream(outStream, out, OutputMode.Append)
      Seq(query)
    }
  }
}
Annotations
@deprecated
Deprecated

(Since version 2.2.0) Use contrib-sbt-spark library instead, see https://github.com/lightbend/cloudflow-contrib

Linear Supertypes
Serializable, Serializable, Streamlet[SparkStreamletContext], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkStreamlet
  2. Serializable
  3. Serializable
  4. Streamlet
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def createLogic(): SparkStreamletLogic
    Attributes
    protected
  2. abstract def shape(): StreamletShape
    Definition Classes
    Streamlet

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val StopTimeout: FiniteDuration
  5. val applicationName: String
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def attributes: Set[StreamletAttribute]

    A set of custom attributes that a streamlet can use to activate features in the Cloudflow runtime.

    A set of custom attributes that a streamlet can use to activate features in the Cloudflow runtime.

    Attributes
    protected[cloudflow]
    Definition Classes
    Streamlet
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def configParameters: IndexedSeq[ConfigParameter]

    Defines a set of configuration parameters that will be used in this streamlet to lookup environment-specific configuration to be provided at deployment time.

    Defines a set of configuration parameters that will be used in this streamlet to lookup environment-specific configuration to be provided at deployment time.

    Definition Classes
    Streamlet
  10. final def configuredValue(context: SparkStreamletContext, configKey: String): String
  11. implicit final def context: SparkStreamletContext

    Returns the StreamletContext in which this streamlet is run.

    Returns the StreamletContext in which this streamlet is run. It can only be accessed when the streamlet is run.

    Attributes
    protected
    Definition Classes
    Streamlet
  12. final def createContext(config: Config): SparkStreamletContext

    Creates a StreamletContext for the appropriate runtime

    Creates a StreamletContext for the appropriate runtime

    Attributes
    protected
    Definition Classes
    SparkStreamlet → Streamlet
  13. def customAttributes: Set[StreamletAttribute]

    A set of custom attributes that a streamlet can use to activate features in the Cloudflow runtime.

    A set of custom attributes that a streamlet can use to activate features in the Cloudflow runtime.

    Definition Classes
    Streamlet
  14. def defineConfigParameters(): Array[ConfigParameter]

    Java API

    Java API

    Defines configuration parameters that will be used in this streamlet to lookup environment-specific configuration to be provided at deployment time.

    Definition Classes
    Streamlet
  15. def defineCustomAttributes(): Array[StreamletAttribute]

    Java API

    Java API

    Defines a set of custom attributes that a streamlet can use to activate features in the Cloudflow runtime.

    Definition Classes
    Streamlet
  16. def defineVolumeMounts(): Array[VolumeMount]

    Java API Defines volume mounts that can be used by the streamlet to mount a volume in a local path.

    Java API Defines volume mounts that can be used by the streamlet to mount a volume in a local path.

    Definition Classes
    Streamlet
  17. def description: String
    Definition Classes
    Streamlet
  18. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  20. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. final def getContext(): SparkStreamletContext

    Java API

    Java API

    Returns the StreamletContext in which this streamlet is run. It can only be accessed when the streamlet is run.

    Attributes
    protected
    Definition Classes
    Streamlet
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. final def inlets: IndexedSeq[Inlet]
    Definition Classes
    Streamlet
  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. final def jsonDescriptor: String

    JSON-Encoded String representing the descriptor of this streamlet.

    JSON-Encoded String representing the descriptor of this streamlet.

    Definition Classes
    Streamlet
  27. def labels: IndexedSeq[String]
    Definition Classes
    Streamlet
  28. lazy val log: Logger
    Attributes
    protected
    Definition Classes
    Streamlet
    Annotations
    @transient()
  29. def logStartRunnerMessage(buildInfo: String): Unit
    Definition Classes
    SparkStreamlet → Streamlet
  30. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  31. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  32. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def outlets: IndexedSeq[Outlet]
    Definition Classes
    Streamlet
  34. final def run(context: SparkStreamletContext): StreamletExecution

    Runs the streamlet.

    Runs the streamlet.

    Definition Classes
    SparkStreamlet → Streamlet
  35. final val runtime: SparkStreamletRuntime.type
    Definition Classes
    SparkStreamlet → Streamlet
  36. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  37. def toString(): String
    Definition Classes
    AnyRef → Any
  38. def updateSparkSession(session: SparkSession): SparkSession

    Override this method to modify the org.apache.spark.SparkSession used in this SparkStreamlet.

    Override this method to modify the org.apache.spark.SparkSession used in this SparkStreamlet. By default this method does not modify the session.

  39. def volumeMounts: IndexedSeq[VolumeMount]

    Defines volume mounts that can be used by the streamlet to mount a volume in a local path.

    Defines volume mounts that can be used by the streamlet to mount a volume in a local path.

    Definition Classes
    Streamlet
  40. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  41. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  42. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Streamlet[SparkStreamletContext]

Inherited from AnyRef

Inherited from Any

Ungrouped