Packages

c

cloudflow.spark

SparkStreamletLogic

abstract class SparkStreamletLogic extends StreamletLogic[SparkStreamletContext]

Provides an entry-point for defining the behavior of a SparkStreamlet. Overide the method buildStreamingQueries to build the collection of StreamingQuery that needs to run as part of the business logic for the SparkStreamlet.

Here's an example of how to provide a specialized implementation of SparkStreamletLogic as part of implementing a custom SparkStreamlet:

// new custom `SparkStreamlet`
object MySparkProcessor extends SparkStreamlet {
  // define inlets, outlets and shape

  // provide custom implementation of `SparkStreamletLogic`
  override def createLogic() = new SparkStreamletLogic {
    override def buildStreamingQueries = {
      val dataset = readStream(in)
      val outStream = dataset.select($"name").as[Simple]
      val query = writeStream(outStream, out, OutputMode.Append)
      Seq(query)
    }
  }
}
Annotations
@deprecated
Deprecated

(Since version 2.2.0) Use contrib-sbt-spark library instead, see https://github.com/lightbend/cloudflow-contrib

Linear Supertypes
StreamletLogic[SparkStreamletContext], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkStreamletLogic
  2. StreamletLogic
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkStreamletLogic()(implicit context: SparkStreamletContext)

Type Members

  1. implicit class StreamingQueryExtensions extends AnyRef

Abstract Value Members

  1. abstract def buildStreamingQueries: StreamletQueryExecution

    Derived classes need to override this method to provide a custom implementation of the logic to build a StreamletQueryExecution object containing one or more StreamingQuerys that need to be executed.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def config: Config
  7. implicit val context: SparkStreamletContext

    Returns the StreamletContext in which this StreamletLogic is run.

    Returns the StreamletContext in which this StreamletLogic is run. It can only be accessed when the streamlet is run.

    Definition Classes
    SparkStreamletLogic → StreamletLogic
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getContext(): SparkStreamletContext

    Java API

    Java API

    Returns the StreamletContext in which this StreamletLogic is run. It can only be accessed when the streamlet is run.

    Definition Classes
    SparkStreamletLogic → StreamletLogic
  13. final def getMountedPath(volumeMount: VolumeMount): Path

    The path mounted for a VolumeMount request from a streamlet.

    The path mounted for a VolumeMount request from a streamlet. In a clustered deployment, the mounted path will correspond to the requested mount path in the VolumeMount definition. In a local environment, this path will be replaced by a local folder.

    volumeMount

    the volumeMount declaration for which we want to obtain the mounted path.

    returns

    the path where the volume is mounted.

    Exceptions thrown

    [[cloudflow.streamlets.MountedPathUnavailableException MountedPathUnavailableException ]] in the case the path is not available.

  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def readStream[In](inPort: CodecInlet[In])(implicit encoder: Encoder[In], typeTag: scala.reflect.api.JavaUniverse.TypeTag[In]): Dataset[In]

    Read from inlet to generate a Dataset.

  20. final def session: SparkSession
  21. final def streamletRef: String
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    AnyRef → Any
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  27. final def writeStream[Out](stream: Dataset[Out], outPort: CodecOutlet[Out], outputMode: OutputMode, optionalTrigger: Option[Trigger] = None)(implicit encoder: Encoder[Out], typeTag: scala.reflect.api.JavaUniverse.TypeTag[Out]): StreamingQuery

    Write a StreamingQuery into outlet using the specified OutputMode

Inherited from StreamletLogic[SparkStreamletContext]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped