December 2019

Written by Odysseas Lamtizidis, EdgeX Foundry Technical Community Member

In this post we will be looking into the integration of two fundamental Open Source platforms for the Internet of Things.

The first being Linux Foundation’s EdgeX Foundry platform as the backbone of our system and Node-RED which will be used to simulate a Device Service.

But first things, first. What is Node-RED?

As we read from their website:

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Node-RED thus is an event-driven flow-based programming platform that is built on Node.js, leveraging it’s lightweight nature and extensive scalability to create applications mainly for the Internet of Things.

A flow-based programming language is a language where the application functionality is expressed as a series of black-boxes that are connected, creating a chain where the output of a box is the input of the next one. Each box receives a message, performs a certain functionality on it and then forwards it to the output and the next box.

Node-RED has various ready nodes (boxes), from creating http resources to converting the message to JSON or running an arbitrary user-defined javascript functions. Below we see an example of a flow as seen from the web editor.

Image X shows a flow in the node-RED web editor, where multiple nodes are defined and connected to create a certain application logic. The flows start with an MQTT listen node which will be triggered whenever an MQTT message is broadcasted in a certain topic.

The project was started from IBM’s Emerging Technology Services group and has a vivid community. You can find more information on the site and in the forums.

EdgeX Integration

As mentioned above, we will be using Node-RED as a custom device service, replacing the standard EdgeX Foundry device-sdk-go library which is used to develop custom device services.

A device-service is a “service” that defines and controls an array of IoT devices, but following the EdgeX micro-services architecture, a device-service as seen from the EdgeX system, is in essence a RESTful API.

Thus, by following the device-service specifications document, one can easily create the appropriate flows in Node-RED so that:

1. The device-service properly registers itself to EdgeX
2. The device-service has the necessary API resources according to the device-service spec
3. The device-service performs arbitrary functionality to manage it’s devices

In our use-case implementation, we will be using Node-RED to introduce a proof-of-concept integration of a Lora-device, using the ChirpstStack (aka Loraserver) network stack and their application-level server. In essence, since the application server can not be directly integrated into EdgeX (different API structure), we will be using Node-RED service as a middleware, translating the EdgeX commands/API calls to Chirpstack’s application server and vice-versa.

Image XX illustrates the high-level architecture of the device service system we described, showcasing the use of Node-RED as a middleware for the 2 distinct RESTful APIs. The image is taken from an original work, please see #More Information.

Node-RED introduction

The development of an application in Node-RED is fairly simple, the user installs Node-RED using a deployment of choice (npm, docker, etc.), as detailed in the docs, and then access the web-editor at http://localhost:1880. The user then proceeds to select nodes and connect one to another. There are a handful of tutorials on the node-RED docs for the user to grab the gist of the platform, the learning curve is truly small.

After creating the flow, the user simply clicks “deploy” and the flow is activated by Node-RED. Node-RED is fairly customisable, enabling the change various aspects of the application, using the settings.js file. We invite the user to explore the documentation and the node-RED forum so as to gain better understanding of the capabilities of the platform.

An important aspect that is worth mentioning is the variable (context) storage mechanism (Context Store). The variables (flow, global) are normally stored in memory but in order to achieve persistency even at the event of a reboot, the user should enable the file-storage, as indicated in the docs. This will save the variables every X seconds in a JSON file, performing checkpoints of persistency. The interval can be reduced, but it can have an impact on performance, especially if we use a flash-storage (such as an sd card) which in the long-term will greatly stress the card.

EdgeX Foundry Registration

In order for the device-service to register with EdgeX, it simply has to perform a series of API calls to the EdgeX core-services, namely to core-data and to core-metadata, according to the EdgeX API walkthrough that is available at the EdgeX docs.

Disclaimer: Please note that the docs are somewhat outdated and the device addressable is no longer needed, the device resource is defined at the provisioning phase.

X2

The reference flow is shown in image X2. The main building blocks are the HTTP REQUEST node and FUNCTION node. For ease of use, for each http request, we define every attribute of the request (body, url) at the function node that proceeds each HTTP REQUEST node. This enables complete customisation and easy debugging for each node. For increased ease-of-use, we defined the majority of the bodies in a function at the start of the flow so we can easily modify common parts (like device name).

The bodies can either be saves as different attributes of the msg object or as flow variables that are loaded before each HTTP REQUEST node. The first is more memory efficient.

After a successful registration, a special flag is saved that is checked at the start of the flow so in the case of a service restart, the device-service will know not to repeat the registration. Moreover, the use-case flow lacks advance flow control mechanisms that will pause the registration in case of an error or controls that enable the device-service to restart registration from the last successful step.

At the moment, the PoC flow we simply restart from the start in case of restart. This is acceptable as EdgeX will simply ignore duplicate objects.

Regarding chirpstack, the registration subflow starts at the end of the EdgeX Registration flow. At some point in the registration, an endpoint is defined so that the Chirpstack application server makes an HTTP request at the Node-RED API endpoint, each time a value is received from the LoRa device. Node-RED receives that value and transforms the request to an EdgeX appropriate format in order to forward it to EdgeX-core-data service.

Device-service API resources

The PoC chooses to implement a subset of the available API specification and doesn’t support the addition of more than one device under the device service. The API conforms to the device specification where the reader can read in-detail the description of each resource.

The API structure is shown in image X1. Note that according to the EdgeX specs, the commands are issued to the device service using a this url structure: device/<device_id>/<command_name>, where device_id is generated at the device provisioning phase, while the command_name is defined at the device profile.

In order to support this structure, the API endpoint is set to listen for any url with the structure device/:variable_1/:variable2, and then proceeds to check the variables at the FUNCTION node that supersedes the HTTP endpoint node. Thus, the user can create a single endpoint to receive all the requests with the above structure and then differentiate depending on the variables.

Finally, the /health endpoint is used by the consul service registry. The user could implement a flow logic where errors are caught and saved into a flow or global variable so they can be used to generated the appropriate error code and message when the /health endpoint is used by consul. The PoC does not implements such a logic and the device-service is set to always as healthy in Consul.

Moreover, a command is defined, that simply returns the last read value from the lora-device.

Arbitrary Functionality

We could enable any kind of functionality, from running arbitrary shell commands on the system or entire program logic in FUNCTION nodes. In this PoC the device-service simply works as a middleware.

More Information:

The source-code for the above blog-post can be found on Github, it is part of a larger project where all the platforms that are described are presented in depth (such as ChirpStack aka Loraserver). The project’s document can be found on ResearchGate.

To learn more about EdgeX Foundry, visit the website or wiki.  Or, you can join the conversation and ask questions on our Slack channel.

This blog originally ran on Odysseas’ blog on Notion. You can view the original here.