Running the Application in the Sandbox

Now that we have the application code implemented, let’s build the application and exercise its functionality locally without requiring deploying it in a cluster.

Run the Streamlets Locally

The sbt runLocal command allows you to run your application on your local machine without a Kubernetes cluster.

If you are trying to run the examples contained in the upstream cloudflow repository remember to run export CLOUDFLOW_VERSION=2.3.0 before invoking sbt.
  1. From the sbt shell, invoke runLocal:

    You should see output similar to the following:

sbt:sensor-data> runLocal
[info] Streamlet 'sensordata.SensorDataToMetrics' found
[info] Streamlet 'sensordata.MetricsValidation' found
[info] Streamlet 'sensordata.SensorDataHttpIngress' found
[info] Streamlet 'sensordata.ValidMetricLogger' found
[info] Streamlet 'sensordata.InvalidMetricLogger' found
[success] /path/to/sensor-data/src/main/blueprint/blueprint.conf verified.

            ┌────────────┐
            │http-ingress│
            └─────┬──────┘
                  │
                  v
           ┌─────────────┐
           │[sensor-data]│
           └──────┬──────┘
                  │
                  v
              ┌───────┐
              │metrics│
              └───┬───┘
                  │
                  v
             ┌─────────┐
             │[metrics]│
             └─────┬───┘
                   │
                   v
             ┌──────────┐
             │validation│
             └───┬───┬──┘
                 │   │
                 │   └───────┐
                 │           │
                 v           v
 ┌─────────────────┐ ┌───────────────┐
 │[invalid-metrics]│ │[valid-metrics]│
 └────────┬────────┘ └─────┬─────────┘
          │                │
          v                v
  ┌──────────────┐  ┌────────────┐
  │invalid-logger│  │valid-logger│
  └──────────────┘  └────────────┘
---------------------------- Streamlets per project ----------------------------
 sensor-data-scala - output file: file:/tmp/cloudflow-local-run3239069643364862589/sensor-data-scala-local.log

	http-ingress [sensordata.SensorDataHttpIngress]
	- HTTP port [3000]
	invalid-logger [sensordata.InvalidMetricLogger]
	metrics [sensordata.SensorDataToMetrics]
	valid-logger [sensordata.ValidMetricLogger]
	validation [sensordata.MetricsValidation]

--------------------------------------------------------------------------------

------------------------------------ Topics ------------------------------------
[invalid-metrics]
[metrics]
[rotor-speeds]
[sensor-data]
[valid-metrics]
--------------------------------------------------------------------------------

----------------------------- Local Configuration -----------------------------
Using Sandbox local configuration file: src/main/resources/local.conf
--------------------------------------------------------------------------------

------------------------------------ Output ------------------------------------
Pipeline log output available in folder: /tmp/cloudflow-local-run3239069643364862589
--------------------------------------------------------------------------------

Running sensor-data-scala
To terminate, press [ENTER]

While the application is running in this local mode, it’s possible to exercise all its interfaces and observe the output written by the streamlets logging or standard output.

The line sensor-data-scala - output file: <path-to-temp-file> indicates the location of the captured output. You can follow the output using command line utilities (tail) or a text editor that support live updates (e.g. sublime text)

We can also appreciate that the ingress streamlet is reachable on the TCP port 3000.

http-ingress [sensordata.SensorDataHttpIngress]
	- HTTP port [3000]

Sending sample data to this port will exercise the pipeline:

$ curl -i -X POST localhost:3000 -H "Content-Type: application/json" --data '{"deviceId":"c75cb448-df0e-4692-8e06-0321b7703992","timestamp":1495545646279,"measurements":{"power":1.7,"rotorSpeed":3.9,"windSpeed":105.9}}'

The output should be similar to:

HTTP/1.1 202 Accepted
Server: akka-http/10.1.11
Date: Wed, 10 Jun 2020 18:11:18 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 88

The request has been accepted for processing, but the processing has not been completed

You can then observe the results by inspecting the output file of the application, as explained above. In the <path-to-temp-file, you should see output similar to this:

2020-10-02 12:04:07 INFO  ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "power", "value": 1.7}
2020-10-02 12:04:07 INFO  ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "rotorSpeed", "value": 3.9}
2020-10-02 12:04:07 INFO  ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "windSpeed", "value": 105.9}

Trying to send invalid data such as:

$ curl -i -X POST localhost:3000 -H "Content-Type: application/json" --data '{"deviceId":"c75cb448-df0e-4692-8e06-0321b7703992","timestamp":1495545646279,"measurements":{"power":1.7,"rotorSpeed":3.9,"windSpeed":-105.9}}'

Will produce, as expected, output from the invalid logger:

2020-10-02 12:19:23 WARN  ActorSystemImpl:119 - Invalid metric detected! {"metric": {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "windSpeed", "value": -105.9}, "error": "All measurements must be positive numbers!"}

To make this easy, we include in the folder of the example a script send-local-data.sh that takes a sample dataset from the test-data/ directory and sends it record-by-record to the HTTP ingress port for processing.

When you are done experimenting with the running application, you can stop it by pressing ENTER

Running sensor-data-scala
To terminate, press [ENTER]


[info] Attempting to terminate local application

What’s next

Now, we are ready to Deploy to a Kubernetes Cluster.