By Jim White, EdgeX Foundry Technical Steering Committee Chair
When you think of Ireland, most people envision bright green, fresh landscapes. An island cleaned regularly by plentiful rain and always made new. The Ireland name is appropriate to our latest release – which is EdgeX made fresh, a major overhaul.
EdgeX Foundry 2.0 is a release over a year in the making!
This is the EdgeX community’s second major release and our 8th release overall. Indeed, the Ireland release contains both new features and non-backward compatible (with any EdgeX 1.x release) changes. In general, this release took a huge step to eliminate, or at least significantly reduce, about four years of technical debt accumulation, while also adding new capabilities.
Why Move to EdgeX 2.0?
Whether you are an existing EdgeX user or new to the platform and considering adoption, there are several reasons to explore the new EdgeX 2.0 Ireland release:
- First, we’ve completely rewritten and improved the microservice APIs. I’ve provided some more details on the APIs below, but in general, the new APIs allow for protocol independence in communications, better message security, and better tracking of data through the services. They are also more structured and standardized with regard to API request / response messages.
- EdgeX 2.0 is more efficient and lighter (depending on your use case). For example, Core Data is now entirely optional.
- EdgeX 2.0 is more reliable and supports better quality of service (depending on your setup and configuration). For example, you can use a message bus to shuffle data from the sensor collecting Device Services to exporting Application Services. This will reduce your reliance on the REST protocol and allow you to use a message broker with QoS levels elevated when needed.
- EdgeX 2.0 has eliminated a lot of technical debt and includes many bug fixes. As an example, EdgeX micro service ports are now within the Internet Assigned Numbers Authority (IANA) dynamic/private port range so as not to conflict with well-known system or registered ports.
Introducing the V2 APIs
During one of our semi-annual planning meetings back in 2019, the community agreed that we needed to address some challenges in the way that EdgeX microservices communicated – both with each other and with third- party systems. We identified some technical debt and knew that it had to be addressed in order to provide the platform the base from which new features, better security, and additional means of communications could be supported in the future. This meant changing the entire API set for all EdgeX microservices.
What was “wrong” with the old V1 APIs? Here were some of the issues:
- Intertwined data model/object information with poor URI paths that exposed internal implementation details and made the APIs difficult to use. Here is an example comparing the v1 and v2 calls to update a device.
V1: PUT /device/name/{name}/lastconnected/{time}
V2: PATCH /device [with JSON body to specify the device info]
- In many cases, requests often lacked any formal response object. The HTTP status code was all that was returned.
- The APIs and model objects were REST specific and didn’t support other communication protocols (such as message bus communications via MQTT or the like).
- Used a model that was very heavy – often containing full embedded model objects instead of model object references. For example, in v1, when requesting device information, the device profile and Device Service were also returned.
- Used inconsistent naming and included unnecessary elements. This made it hard for users to figure out how to call on the APIs without checking the documentation.
- Lacked a defined header that could be used to provide security credentials/tokens in the future.
Changing the entire API set of a platform is a monumental task. One that required the collective community to get behind and support. Given the size of the task, the community decided to spread the work over a couple of release cycles. We began the work on the new APIs (which we called the V2 APIs) last spring while working on the Hanoi release and completed the work with this release.
The new APIs come with common sense request and response objects. These objects share new features such as a correlation ID which allows requests and responses among services to be associated, and it also allows the flow of data through the system to be tracked more easily. When a piece of sensor data is collected by a Device Service, the correlation ID on that data is the same all the way through the export of that data to the north side systems. Below is a simple example of the new request and responses – in this case to add a new Device Service.
Importantly, the V2 APIs set us up for new features in the future while also making EdgeX easier to interact with.
EdgeX 2.0 Feature Highlights
Technical debt removal and the new V2 APIs aside, what else was added in EdgeX 2.0? Plenty! Here is a sample of some of the more significant features.
Message Bus and Optional Core Data
In EdgeX 1.0, all communications from the Device Services (the services that communicate with and collect data from “things”) and Core Data was by REST. In EdgeX 2.0, this communication can be performed via message bus (using either Redis Pub/Sub or MQTT implementations). In addition to more “fire and forget” communications, the underlying message bus brokers used in message bus communications offer guarantees of delivery of the message.
Moreover, EdgeX 2.0 takes the message bus implementation a step further by allowing Device Services to send the data directly to Application Services via message bus. Core data becomes an optional secondary subscriber in this instance. For organizations that do not need to persist sensor data at the edge, this option allows the entire Core Data service to be removed from deployment helping to lighten EdgeX’s footprint and resource needs.
REST communications from Device Services to Core Data can still be done, but it is not the default implementation in EdgeX 2.0. The diagrams below depict the old versus new message bus service communications.
Improved security
In the 2020 IoT developer survey, security remained one of the top 3 concerns of people building and fielding IoT/edge solutions. It also remains a prime concern of the members of our community. The EdgeX project, with each release, has worked to improve the security of the platform. In this release, there were several additions.
- We created a module and a pattern that provides a common means for all the services to retrieve secrets from the secret store (EdgeX uses Vault as the secret store by default). We call this feature “secret provider for all.”
- EdgeX uses Consul for service registry and configuration. In this release, the API Gateway is used to allow access to Consul’s APIs. Access to the Consul APIs in EdgeX 1.0 was denied making changes in Consul difficult.
- Consul is now bootstrapped and started with its access control list system enabled – offering better authentication/authorization of the service and better protection of the key/value configuration it stores.
- Fewer processes and services are required to run as root in Docker containers.
- The API Gateway (Kong) setup has been improved and simplified.
- EdgeX now prevents Redis, the persistent store for EdgeX, from running in insecure mode.
Simplified Device Profiles
Device profiles are the way that users define the characteristics about sensors/devices connected to EdgeX – that is the data they provide, and how to command them. For example, a device profile for BACnet thermostats provides information on data a BACnet thermostat sends, such as current temperature and humidity level. It also defines which types of commands or actions EdgeX can send to the BACnet thermostat such as the ability to set the cooling or heating point.
Device profiles are specified in YAML or JSON and uploaded to EdgeX. As such, they are the critical descriptions that makes EdgeX work. Device profiles allow users to describe their sensors and devices to EdgeX without having to write lots of programming code.
Writing device profiles in previous releases could be long and tedious depending on the device and what type of data it provided. We greatly simplified and shortened device profiles in EdgeX 2.0. As an example, here is the same essential profile in EdgeX 1 and EdgeX 2.
New Device Services
The growth of an open-source community can often be measured by how well it is attracting new contributors into the project. During this release, thanks to some new contributors, we added several new Device Services. Here is the new Device Services you can now find in the project:
- Constrained Application Protocol (CoAP) is a web transfer protocol used in resource constrained nodes and networks.
- GPIO (or general pin input/output) is a standard interface used to connect microcontrollers to other electronic devices. It is also very popular with Raspberry Pi adopters given its availability on the devices.
- Low Level Reader Protocol (LLRP) is a standardized network interface to many RFID readers.
- Universal Asynchronous Receiver/Transmitter (UART) is serial data communications and is used in modems and can be used with USB in a USB to UART bridge.
These Device Services were contributed to EdgeX 1, but are being updated this summer to EdgeX 2.0.
New Graphical User Interface
In this release, a new and complete interface using Angular.JS (with some Go code for backend needs) has been added. This GUI, complete with EdgeX branding, aligns with industry standards and technologies for user interfaces and should provide a platform for future GUI needs.
In addition to providing the ability to work with and call on EdgeX services, it provides a GUI “wizard” to help provision a new device (with associated Device Service and device profile). What would typically take a user several complex REST API calls can now be done with a wizard interface that carefully walks the user through the creation process providing simple fill-in forms (with context help) to create the device.
The GUI also provides a visual status display that allows users to track the memory and CPU usage of the EdgeX services. In the future, additional visualization will be added to be able to display sensor data.
Application Services
Application Services are used to export EdgeX data off the edge to enterprise and cloud systems and applications. Application Services filter, prepare and transform the sensor data for easy consumption. Several additions were made to the Application Services and the SDK that helps create them (called the application functions SDK). The following is a list of some of the new features and functions:
- New functions to filter sensor readings by device profile name and device resource name before exporting
- Allowing multiple HTTP endpoints to be specified for export by one Application Service
- Subscription to multiple message bus (enabling multiple filters by subscriptions)
- Provided a new template for easier/faster creation of custom Application Services
Additionally, a new LLRP inventory Application Service was contributed to help correlate data coming from RFID readers to backend inventory systems.
EdgeX Ready
EdgeX Ready is a program designed to highlight member organizations that demonstrate the ability to work with EdgeX. The EdgeX Ready program was launched with the EdgeX 2.0 release in order to promote awareness of users and their organizations that have EdgeX expertise. This is the first step or a precursor to potentially exploring an EdgeX certification program.
Today, the EdgeX Ready self-assessment process requires an EdgeX adopter to:
- Get and run an instance of EdgeX (and associated services)
- Create and validate a device profile.
- Use the profile with an EdgeX Device Service to provision a device and send sensor data into EdgeX.
When an organization completes the EdgeX Ready process, they signal to other community members that they have successfully demonstrated they have EdgeX knowledge and experience. In return, the EdgeX community highlights EdgeX Ready members on the EdgeX Foundry website with an EdgeX Ready badge (shown below) next to their logo.
Additionally, the program hopes to promote sharing of EdgeX device connectivity elements and sample data sets, which are central to the current submission process and helpful to others learning EdgeX. Future EdgeX Ready programs may highlight additional levels of ability and experience.
Under the Hood and Behind the Scenes
Effort by DevOps, Testing, Quality Control and Outreach teams often go unnoticed in any software effort. These teams are composed of unsung heroes of the EdgeX 2.0 as they contributed substantially to this massive undertaking. In addition to their normal roles of building, testing and marketing EdgeX, they managed to further the project in many ways.
Our DevOps team cleaned up the EdgeX Docker image names (removing unnecessary image prefix/suffix names) and provided descriptions and appropriate tags on all of our images in Docker Hub – helping adopters more quickly and easily find the EdgeX Docker images they need. They also added informative badges to our GitHub repositories to help community developers and adopters understand the disposition and quality of the code in the repositories.
Our Test/QA team added new integration tests, performance tests and “smoke” tests to give the project an early indicator when there is an issue with a new version of a 3rd party library or component such as Consul – allowing the community to address incompatibilities or issues more quickly.
Finally, our marketing team revamped and upgraded our www.edgexfoundry.org website and create a Chinese version of our website in support of our ever growing Chinese community of adopters. The welcome mat of our project now has a clean new look, better organization, and a lot more information to provide existing and potential adopters. The website should help EdgeX adopters the information and artifacts they need more quickly and easily, while also highlighting the accomplishements of the community and its membership.
If all this wasn’t enough, here is a list of some of the other accomplishments the project achieved during this release cycle:
- Improved the Device Service contribution review process.
- Incorporated use of conventional commits in the code contribution process.
- Started a program to vet 3rd party dependencies; insuring 3rd party dependencies are of sufficient license, quality and development activity to support our project.
- Help launch the Open Retail Reference Architecture project to foster the development of an LF Edge reference architecture for retail adopters.
- Entered into liaison agreements with the Digital Twin Consortium and AgStack (a new LF project) to figure out how EdgeX can be better integrated into digital twin systems and help facilitate solutions in the agricultural ecosystem.
More on Ireland
Find more information about the release at these locations.
On to Jakarta
As always, I want to take this opportunity to thank the wonderful EdgeX team and community. EdgeX exists because of the incredibly smart and dedicated group of people in the community. I also want to thank all the fine people at the Linux Foundation. This release was especially long and difficult, but this team was up to the challenge and they are providing you another great release. I am proud of their work and privileged to be the chairman of this group.
But our work is not done. The EdgeX community has already held its planning session for our next release – codenamed Jakarta – that will release around November of 2021. Look for more details on our planning meeting and the Jakarta release in another blog post I’ll have out soon.
Enjoy the summer, stay safe and healthy (let’s get this pandemic behind us), and please give EdgeX 2.0 a try.
