Running the Application in the Sandbox
Now that we have the application code implemented, let’s build the application and exercise its functionality locally without requiring deploying it in a cluster.
Run the Streamlets Locally
The sbt runLocal
command allows you to run your application on your local machine without a Kubernetes cluster.
If you are trying to run the examples contained in the upstream cloudflow repository remember to run
export CLOUDFLOW_VERSION=2.2.0 before invoking sbt .
|
-
From the
sbt
shell, invokerunLocal
:You should see output similar to the following:
sbt:sensor-data> runLocal
[info] Streamlet 'sensordata.SensorDataToMetrics' found
[info] Streamlet 'sensordata.MetricsValidation' found
[info] Streamlet 'sensordata.SensorDataHttpIngress' found
[info] Streamlet 'sensordata.ValidMetricLogger' found
[info] Streamlet 'sensordata.InvalidMetricLogger' found
[success] /path/to/sensor-data/src/main/blueprint/blueprint.conf verified.
┌────────────┐
│http-ingress│
└─────┬──────┘
│
v
┌─────────────┐
│[sensor-data]│
└──────┬──────┘
│
v
┌───────┐
│metrics│
└───┬───┘
│
v
┌─────────┐
│[metrics]│
└─────┬───┘
│
v
┌──────────┐
│validation│
└───┬───┬──┘
│ │
│ └───────┐
│ │
v v
┌─────────────────┐ ┌───────────────┐
│[invalid-metrics]│ │[valid-metrics]│
└────────┬────────┘ └─────┬─────────┘
│ │
v v
┌──────────────┐ ┌────────────┐
│invalid-logger│ │valid-logger│
└──────────────┘ └────────────┘
---------------------------- Streamlets per project ----------------------------
sensor-data-scala - output file: file:/tmp/cloudflow-local-run3239069643364862589/sensor-data-scala-local.log
http-ingress [sensordata.SensorDataHttpIngress]
- HTTP port [3000]
invalid-logger [sensordata.InvalidMetricLogger]
metrics [sensordata.SensorDataToMetrics]
valid-logger [sensordata.ValidMetricLogger]
validation [sensordata.MetricsValidation]
--------------------------------------------------------------------------------
------------------------------------ Topics ------------------------------------
[invalid-metrics]
[metrics]
[rotor-speeds]
[sensor-data]
[valid-metrics]
--------------------------------------------------------------------------------
----------------------------- Local Configuration -----------------------------
Using Sandbox local configuration file: src/main/resources/local.conf
--------------------------------------------------------------------------------
------------------------------------ Output ------------------------------------
Pipeline log output available in folder: /tmp/cloudflow-local-run3239069643364862589
--------------------------------------------------------------------------------
Running sensor-data-scala
To terminate, press [ENTER]
While the application is running in this local mode, it’s possible to exercise all its interfaces and observe the output written by the streamlets logging or standard output.
The line sensor-data-scala - output file: <path-to-temp-file>
indicates the location of the captured output.
You can follow the output using command line utilities (tail) or a text editor that support live updates (e.g. sublime text)
We can also appreciate that the ingress
streamlet is reachable on the TCP port 3000.
http-ingress [sensordata.SensorDataHttpIngress]
- HTTP port [3000]
Sending sample data to this port will exercise the pipeline:
$ curl -i -X POST localhost:3000 -H "Content-Type: application/json" --data '{"deviceId":"c75cb448-df0e-4692-8e06-0321b7703992","timestamp":1495545646279,"measurements":{"power":1.7,"rotorSpeed":3.9,"windSpeed":105.9}}'
The output should be similar to:
HTTP/1.1 202 Accepted
Server: akka-http/10.1.11
Date: Wed, 10 Jun 2020 18:11:18 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 88
The request has been accepted for processing, but the processing has not been completed
You can then observe the results by inspecting the output file of the application, as explained above.
In the <path-to-temp-file
, you should see output similar to this:
2020-10-02 12:04:07 INFO ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "power", "value": 1.7}
2020-10-02 12:04:07 INFO ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "rotorSpeed", "value": 3.9}
2020-10-02 12:04:07 INFO ActorSystemImpl:99 - valid-logger {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "windSpeed", "value": 105.9}
Trying to send invalid data such as:
$ curl -i -X POST localhost:3000 -H "Content-Type: application/json" --data '{"deviceId":"c75cb448-df0e-4692-8e06-0321b7703992","timestamp":1495545646279,"measurements":{"power":1.7,"rotorSpeed":3.9,"windSpeed":-105.9}}'
Will produce, as expected, output from the invalid logger:
2020-10-02 12:19:23 WARN ActorSystemImpl:119 - Invalid metric detected! {"metric": {"deviceId": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1495545646279, "name": "windSpeed", "value": -105.9}, "error": "All measurements must be positive numbers!"}
To make this easy, we include in the folder of the example a script send-local-data.sh
that takes a sample dataset from the test-data/
directory and sends it record-by-record to the HTTP ingress port for processing.
When you are done experimenting with the running application, you can stop it by pressing ENTER
Running sensor-data-scala
To terminate, press [ENTER]
[info] Attempting to terminate local application
What’s next
Now, we are ready to Deploy to a Kubernetes Cluster.