Skip to main content
Category

EdgeX Foundry

Edgex Foundry Goes Full Message Bus With Its Latest EdgeX 2.3 “Levski” Release

By Blog, EdgeX Foundry, Project Release
EdgeX Foundry now fully embraces decoupled message bus communication within the platform

I am delighted to announce today the availability of the EdgeX Foundry 2.3 release which is codenamed Levski. This is the project’s 11th official release since its inception in 2017 and includes some significant new features and benefits to users that I’ll explain in this blog.

Having been elected as the chair of the EdgeX Technical Steering Committee (TSC) earlier this summer, it’s great to be able to announce the news of this release but it really is the technical expertise and dedication of the EdgeX development team that helps bring these releases to the wider community. Thanks again for everyone’s effort here. Great to be part of a group of many different companies and individuals working together to make great open source edge software.

What is Levski?

First let me give mention to the Levski codename. As you may know, each EdgeX release is named after a certain place or location in the world, with the specific place chosen each time significant contributors to the project. This version was named by two long term EdgeX contributors, Diana Atanasova and Malini B

handaru, both from VMware. For version 2.3 and the letter “L”, we come back over to Europe because Levski is a large mountain in the beautiful European country of Bulgaria. Bulgaria is also Diana’s homeland so it’s nice to be able to recognize that with the naming of this release.

New in EdgeX 2.3 – More about the Message Bus

Perhaps the biggest single new feature is the enhancement to support the delivery of commands via the EdgeX message bus.

We’ve previously made great strides in EdgeX V2 by delivering data from the southside (from devices and sensors) to the northside (to the core and application layers) via an internal message bus. This release also adds the support in the other direction, i.e., from the northside (the application layers and core layers) down to the southside (the devices and sensors) on the same message bus. Previously the southbound communication was exclusively via REST. Moving to support message bus-based communication in both directions is a big advancement and brings key benefits in terms of reduced latency and increased scalability. The asynchronous communication and the QoS-based control that you get from the message bus implementations that you can choose adds delivery guarantees and retransmissions of messages as needed.

Message Bus

More Run-Time Data

Another key area of development for Levski has been the focus on providing EdgeX users with more live information about how the system is running. EdgeX 2.3 adds System or “Control Plane” Events that can provide live updates as to what is happening. For example, users can receive notifications that new devices are added or that there has been a network disconnection from a specific device.

Somewhat related to System Events are the Telemetry Metrics. In contrast to events though, the metrics provide numeric information relating to how the platform is operating. Examples include the number of data readings that are persisted by core data or the number of secrets or tokens that are stored. Building of what was delivered previously, Levski adds more metrics across all of the services.

All of this information can be collected and reported however the user sees fit. In addition, the System Events and Telemetry Metrics mechanisms are available for users to add their own metrics as needed.

Many other updates and enhancements

There are many other new additions in this last release cycle including:

  • Availability of NATS as alternative implementation of the internal EdgeX Message Bus. NATS is a popular and lightweight protocol with native delivery of EdgeX messages and potential advantages in high availability (HA) use cases
  • Authenticated access to the MQTT message bus
  • Securing of the Consul registry service with access tokens
  • Passing status at the Open Source Security Foundation (OpenSSF)
  • Initial construction of the EdgeX STRIDE threat model
  • Improving the EdgeX development process with a new Use Case Requirements (UCRs) phase in the design procedure

See the release notes here for full details of what the Levski release provides.

New adopters and use cases for EdgeX

Running as a mature, stage 3 Linux Foundation project, there are now many users and adopters of EdgeX technology around the world. That includes users who download and deploy the open source EdgeX code, but also users of commercial products (including ours at IOTech) that are based on EdgeX. Take a look at some of the companies who are users of EdgeX. There are also a good set of presentation videos from different adopters of the technology.

One of my personal aims as TSC chair is to help encourage the adoption and wider use of EdgeX. Thanks to Building System Integrators (BSI) who recorded a talk at a recent TSC meeting. It would be great to hear from more companies who would like to do something similar. Please do reach out to me to arrange it.

What’s coming next?

Each release marks a busy period where of course we finalize the current version, but already we are looking forward at what we can achieve in the next release. Codenamed Minnesota and expected to be version 3.0, we are looking to add new features to help with the scalable configuration of EdgeX. Tune into the Minnesota technical planning conference that we are running next week where we will scope out what we can do. As always, attendance and contributions are very welcome.

Thanks again to everyone who had a hand in this release, and we look forward to more successful releases in the future.

Your Guide to LF Edge (+ Related) Sessions at ONE Summit

By Akraino, Blog, EdgeX Foundry, Event, Home Edge, LF Edge, Open Horizon, Project EVE

In case you missed it, the ONE Summit agenda is now live! With 70+ sessions delivered by speakers from over 50 organizations, at ONE Summit, you can meet industry experts who will share their edge computing knowledge across 5G, factory floor, Smart Home, Robotics, government, Metaverse, and VR use cases, using LF Edge projects including Akraino, EdgeX Foundry, EVE and more.

Save your seat for the ONE Summit today and add these edge sessions to your schedule. We hope to see you in Seattle, WA November 15-16!

Tuesday, November 15:

9:00am – 9:15am

11:30am – 12:00pm

12:10pm – 12:40pm

12:10pm – 12:40pm

2:00pm – 2:30pm

2:40pm – 3:10pm

  • Proliferation of Edge Computing in Smart Home
    • Speakers:
      • Suresh Lalapet Chakravarthy, Staff Engineer, Samsung R&D Institute India – Bangalore
      • Nitu Sajjanlal Gupta, Lead Engineer, Samsung R&D Institute India – Bangalore
    • Featured LF project: Home Edge

3:40pm – 4:10pm

3:54pm – 4:01pm

4:20pm – 4:50pm

  • 4:20pm – 5:30pm
    • Featured LF project: Project EVE
Wednesday, November 16

11:30am – 12:00pm

12:10pm – 12:40pm

2:00pm – 2:30pm

  • Deploying and Automating at the Edge
    • Speakers:
      • William Brooke Frischemeier, SR. Director Head Of Product Management Unified Cloud BU, Rakuten Symphony
      • Mehran Hadipour, VP- BD & tech Alliances, Rakuten Symphony

3:40pm – 4:10pm

4:20pm – 4:50pm

4:20pm – 5:30pm

Hurry! Early Bird (Corporate) registration closes September 9! Bookmark the ONE Summit website to easily find updates as more event news is announced, and follow LF Edge on Twitter to hear more about the event. We hope to see you in Seattle soon!

 

Preventive Healthcare using Intelligent Edge IoT Computing – ‘Round the Clock EHR’

By Blog, EdgeX Foundry

By:  Utpal Mangla – Industry EDGE Cloud – IBM Cloud Platform;  Atul Gupta – Lead Data Architect – IBMLuca Marchi – Industry EDGE and Distribute Cloud – IBM Cloud Platform; and Saurabh Agrawal – Practice Lead, Network Cloud, 5G & EDGE 

Preventive healthcare has always been taught and talked about in most healthcare institutions, but seldom practiced. The collective benefits are immense, and the realization of a fraction of those benefits can help optimize the cost-delivery aspects of healthcare services. It can also act as a catalyst for scaling healthcare services for growing populations in most parts of the world. 

The common issues in implementing Preventive Healthcare include providing guidance, educating the vulnerable population masses, and interacting with healthcare professionals. Most of the challenges in regular screenings or monitoring of diabetes, cholesterol, cancer, and mental health are known cost and labor related issues.

Technology has helped to achieve some primary objectives, but the ultimate goal is still far away. IoT and Robotics solutions have been deployed in healthcare facilities and continue to advance quickly.  But with new tools, techniques, and cloud infrastructure,  it’s time for innovative new solutions that come with  emerging IoT devices, like increased connectivity, speed, and lower costs. 

 

Intelligent Edge IoT can provide intense data gathering and processing before sending the prepared data into the Health Ecosystem. Early on, IoT’s were mostly sensors used to gather data and were often cumbersome when it came to software code updates, and they didn’t have enough compute power to run any data processing. The new generation of intelligent IoT’s, which can be deployed at the Edge, can compute and process data ahead of sending it to Cloud computing functions for more sophisticated analysis, including machine learning and deep learning algorithms.

These intelligent IoT’s once deployed as human wearables, facility sensors, etc. can transform EHR (Electronic Health Record) to next generation 24x7x365 EHR (Round the Clock EHR)

This innovation of ‘Round the Clock EHR’ can transform the healthcare industry by bringing healthcare professionals much closer to patients in understanding and diagnosing medical issues early on. It can continuously stream and feed data to ML (machine learning), DL (deep learning) algorithms to help and improve AI solutions. The risk factors can be constantly monitored, thresholds can be adjusted and re-calibrated in real-time with sophisticated software. The Preventive healthcare can not only detect the early signs but also feed it to research facilities for new research programs and eliminate unnecessary analysis. 

The Preventive care strategies which also encompass social and environmental conditions can now be monitored with the ‘Round the Clock EHR’, which contains the patient or potential-patient data along with their detailed data on living conditions, geographical conditions, social setup which may trigger onset of a disease or unfavorable conditions. On a larger scale, the Healthcare ecosystem data and ‘Round the Clock EHR’ can provide feed into wider studies to generate health patterns as people move and Preventive Healthcare can lead to new advancements.

As referenced in https://www.ncbi.nlm.nih.gov/books/NBK537222/ [Prevention Strategies Lisa A. Kisling; Joe M Das.] – Primary prevention consists of measures aimed at a susceptible population or individual. Secondary prevention emphasizes early disease detection, and its target is healthy-appearing individuals with subclinical forms of the disease. Tertiary prevention targets both the clinical and outcome stages of a disease.

Analyzing these 3 preventive healthcare areas from the ‘Data and Intelligent Edge IoT Lens’, it is apparent that the commonality is gathering, processing, and providing good quality round-the-clock granular data to most sophisticated ML, DL algorithms can definitely transform this area of healthcare transformation.

Our advancing society needs blend of medical and technology research implemented in a way to foster growth, scalability, improvements at lower cost and implementation cycles. These de-coupled solutions are easy to implement and promote wider Healthcare ecosystem in our connected social network.

References: 

https://www.ncbi.nlm.nih.gov/books/NBK537222/ [Prevention Strategies Lisa A. Kisling; Joe M Das.]

Congratulations to the Recipients of the 2022 EdgeX Awards!

By Blog, EdgeX Foundry

By Jim White, outgoing EdgeX Foundary TSC Chair

Each year since its founding, EdgeX Foundry awards at least four members of its contributing community with Contribution and Innovation awards.  On this fifth anniversary of the project’s founding, we honored seven distinguished engineers for their efforts on the project.

Innovation Award

Presented to individuals who have provided the most innovative solution to the open-source project over the past year

Byron Nevis and Jim Wang (both from Intel) designed and implemented the new “Delayed Start Services” security capability which is an innovative way to handle providing security tokens to services that start after the EdgeX security framework has started and already handed out tokens to all the known services.  Bryon and Jim have led most EdgeX security efforts over the past two years.  But this year, they provided some real innovation around securely storing secrets, distributing secrets, and making secrets available to other services.  Their innovation will allow adopters to add new device services and application services to EdgeX over time and as use cases demand without requiring a restart of the system.

Anthony Casagrande and Marc Fuller (both from Intel) have been instrumental in Intel’s use of EdgeX to show how it can be a platform in support of retail edge use cases.  Anthony/Marc’s work has provided both a device service and an application service to ingest and use RFID information via the LLRP (Low Level Reader Protocol) which is used in bar code reading equipment found in many retail store locations.  In addition to their inventions of these ingesting and using services, Anthony and Marc have found (and in some cases fixed) a number of bugs and have identified many needed features (submitting more than 25 Github issues) that real world adopters of the platform need.

Emilo Reyes (again from Intel – there seems to be a theme here!) has been a contributor to the EdgeX DevOps team for the last few years and has been a vital part of making the EdgeX community a better, more streamlined community. Emilio has passion for quality and with his experience around unit testing, has contributed hundreds of tests for our Jenkins global pipeline libraries providing us continuous confidence in our ability to deliver quality code. Emilio also has a passion for automation and has been a driving force behind much of our GitHub automation for EdgeX.   For instance, he created automated management of GitHub labels and milestones from a central repository, greatly reducing the number of repetitive tasks needed to manage labels/milestones across 20+ repositories. 

Contribution Award

Recognizes individuals who have contributed leadership and helped EdgeX Foundry advance and continue momentum in the past year.

 

Iain Anderson (from IOTech Systems) – has been a solid contributor, working group leader, and architect/thought leader since the founding of EdgeX in 2017.  As Device Service chairman, he has overseen several releases (major and minor) and is currently responsible for 11 active EdgeX device services and helping to usher in 4 more that are in active development.  He is the project’s first and foremost authority in C development and personally handles most of the C device service and SDK development.  Iain has over 530 commits to the project (almost 50K lines of code) which puts him #6 all time on the project.  He is also #7 in all time Pull Request submissions for the project.  He is a quiet, steadfast, never-a-complaint contributor that has championed many of the project’s advances such as device service and device profile simplifications, message bus communications from device services, device filtering, and future device service record and replay features.

 

Siggi Skulason (from Canonical) – updated and significantly improved the EdgeX CLI tool over the course of the last two release cycles.  Importantly, Siggi upgraded the tool from the V1 APIs to the V2 APIs, added support for all the service endpoints (versus a small subset selection of APIs), made the CLI available as a snap, removed a significant amount of technical debt, and improved the products help and documentation.  

 

I am happy to honor and call out these gentlemen for their efforts.  I don’t know of too many open-source projects that go to honor and thank its contributors with awards like this.  To be nominated and then selected by your peers – especially of the caliber of the EdgeX engineering community – is such a great recognition of their work.  Congratulations Bryon, Jim, Anthony, Marc, Emilo, Iain and Siggi.  Jobs well done.

You can view the Awards presentation here:

 

Thank You EdgeX

It’s been a productive few months – one release out, another planned and being worked on, and a time to shell out some well earned “kudos”.  Before I go, I want to let the community know  I am stepping aside from my role as TSC chair, and a new set of leaders are stepping up to take EdgeX into its future – and a bright future it has.

About a month after I joined Dell Technologies in 2015, I was handed the task of finding an IoT software platform to put on our new brand of gateways.  EdgeX began life on my kitchen island with an idea, an architecture and a small bit of code.  With the support of the company, some great leaders, and a collection of some of the brightest engineers I have ever worked with, my work was expanded on, productized and launched as the edge software you know today.  For the past seven years (almost to the day), EdgeX has been at the center of my professional working life (and my wife would probably add that it included much of my personal life).  Creating it, taking it into open source, working with the Linux Foundation to make it available and known, working at IOTech to commercialize the idea, and leading this wonderful community has been the highlight of my career.  It has allowed me to travel the world, meet so many amazing people, be a part of an incredible creative process, and watch something I started get used to create solutions that help people all over the globe.  EdgeX has exceeded even my wildest dreams.  It’s just hard to wrap my brain around it even today.  I am no Einstein.  But if you have an idea (even a moderately good idea), find some amazing people around you that can help turn the idea into reality, and you can catch lightning in a bottle.

I would need a separate blog post to thank everyone I owe for this experience and the success of EdgeX.  That is not an exaggeration.  “Thank you” is not enough but to the EdgeX community, people I worked with at Dell and those in my current home of IOTech Systems, I hope you take it as a small down payment for all I owe you.

I’ll close by suggesting that if anyone ever offers you the chance to work on, let alone start, an open-source project – jump at the opportunity.  You will be better for the experience.  I’ll paraphrase Winston Churchill – many forms of software development have been tried and will be tried in this world.  No one pretends that open-source development is perfect or all-wise.  Indeed, it has been said that open-source development is the worst form of software development except for all other forms that have been tried.  

EdgeX – small at the edge, but forever big in my heart.

EdgeX Foundry Issues Bug Fix Release (v2.1.1) of its EdgeX Jakarta Release

By Blog, EdgeX Foundry
The EdgeX Foundry project, a highly-scalable and flexible open source framework that facilitates interoperability between devices and applications at the IoT edge, issued a bug fix release (version 2.1.1) of the EdgeX Jakarta release.  Importantly, as Jakarta is under long-term-support (LTS), this new bug fix release addresses a critical Common Vulnerabilities and Exposures (CVE) issue.  Adopters using the Jakarta LTS release should upgrade immediately to this bug fix release.  The CVE is documented on the project’s GitHub site.  In addition, this bug fix release also addresses some less critical bugs that were identified and fixed through the Kamakura release cycle.  Details on what is in version 2.1.1 can be found on the project’s Wiki site.
All of the fixes in this Jakarta dot release are already available in the Kamakura release (the latest, but non-LTS, release from EdgeX).  Adopters of Kamakura do not need to make any changes as the CVE and other bug fixes are already part of the latest EdgeX release.

LF Edge Releases Industry-Defining Edge Computing White Paper to Accelerate Edge/ IoT Deployments

By Akraino, Announcement, Baetyl, EdgeX Foundry, eKuiper, Fledge, Home Edge, LF Edge, Open Horizon, Project EVE, Secure Device Onboard, State of the Edge

Collaborative community white paper refines the definitions and nuances of open source edge computing across telecom, industrial, cloud, enterprise and consumer markets

 SAN FRANCISCO – June 24, 2022 –  LF Edge, an umbrella organization under the Linux Foundation that aims to establish an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system, today announced continued ecosystem collaboration via a new collaborative white paper, “Sharpening the Edge II: Diving Deeper into the LF Edge Taxonomy & Projects.” 

A follow-up to the LF Edge community’s original, collaborative 2020 paper which provides an overview of the organization and details the LF Edge taxonomy, high level considerations for developing edge solutions and key use cases,the new publication dives deeper into key areas of edge manageability, security, connectivity and analytics, and highlights how each project addresses these areas. The paper demonstrates maturation of the edge ecosystem and how the rapidly growing LF Edge community has made great progress over the past two years towards building an open, modular framework for edge computing. As with the first publication, the paper addresses  a balance of interests spanning the cloud, telco, IT, OT, IoT, mobile, and consumer markets.  

“With the growing edge computing infrastructure market set to be worth up to $800B by 2028, our LF Edge project communities are evolving,” said Jason Shepherd, VP Ecosystem, ZEDEDA  and former LF Edge Governing Board Chair. “This paper outlines industry direction through an LF Edge community lens. With such a diverse set of knowledgeable stakeholders, the report is an accurate reflection of a unified approach to defining open edge computing.” 

“I’m eager to continue to champion and spearhead the great work of the LF Edge community as the new board chair,” said Tina Tsou, new Governing Board chair, LF Edge.  “The Taxonomy white paper that demonstrates the accelerated community momentum seen by open source edge communities is really exciting and speaks to the power of open source.” 

The white paper, which is now available for download,  was put together as the result of broad community collaboration, spanning insights and expertise from subject matter experts across LF Edge project communities: Akraino, EdgeX Foundry, EVE, Fledge, Open Horizon, State of the Edge, Alvarium, Baetyl, eKuiper, and FIDO Device Onboard. 

ONE Summit North America 2022

Join the broader open source ecosystem spanning Networking, Edge, Access, Cloud and Core at ONE Summit North America, November 15-16 in Seattle, Wash. ONE Summit is the one industry event focused on best practices, technical challenges, and business opportunities facing decision makers across integrated verticals such as 5G, Cloud, Telco, and Enterprise Networking, as well as Edge, Access, IoT, and Core. The Call for Proposals is now open through July 8, 2022. Sponsorship opportunities are also available. 

 

About The Linux Foundation 

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation’s projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, and more.  The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

 

###

Calling all Edge App Developers! Join the ETSI/LF Edge Hackathon

By Akraino, Blog, EdgeX Foundry, LF Edge

We’re partnering with ETSI ISG MEC to host a Hackathon to encourage open edge developers to build solutions that leverage ESTI MEC Services APIs and LF Edge (Akraino) blueprints. Vertical use cases could include Automotive, Mixed and Augmented Reality, Edge Computing and 5G, and more. Teams are highly encouraged to be creative and propose to develop solutions in additional application verticals. 

Here’s how it works:

  • Participants will be asked to develop an innovative Edge Application or Solution, utilizing ETSI MEC Service APIs and LF Edge Akraino Blueprints.
  • Solutions  may include any combination of client apps, edge apps or services, and cloud components.
  • The Edge Hackathon will run remotely from June to September with a short-list of best teams invited to complete with demonstrations and a
  • Hackathon “pitch-off” at the Edge Computing World Global event in Santa Clara, California Silicon Valley on October 10th-12th

Submissions are due 29th June 2022.  More details, including how to register, are available here.

Additional background information:

Edge Computing provides developers with localized, low-latency resources that can be utilized to create new and innovative solutions, which are essential to many application vertical markets in the 5G era.

  • ETSI’s ISG MEC is standardizing an open environment that enables the integration of applications from infrastructure and edge service providers across MEC (Multi-access Edge Computing) platforms and systems, which offers a set of open standardized Edge Service APIs to enable the development and deployment of edge applications at scale.
    • Teams will be provided with a dedicated instance of the ETSI MEC Sandbox for their work over the duration of the Hackathon. The MEC Sandbox is an online environment where developers can access and interact with live ETSI MEC Service APIs in an emulated edge network set in Monaco, which includes 4G, 5G, & WiFi networks configurations, single & dual MEC platform deployments, etc.
  • LF Edge’s Akraino project offers a set of open infrastructure and application Blueprints for Edge, spanning a broad variety of application vertical use-cases. Each blueprint offers a high-quality full-stack solution that is tested and validated by the community.

However, teams are not required to use a specific edge hosting platform or MEC lifecycle management APIs. Developers may use any virtualization environment of their choosing. 

Submissions are due 29th June 2022.  More details, including how to register, are available here.

What’s Next for EdgeX: Onto Levski

By Blog, EdgeX Foundry

By Jim White, EdgeX Foundry TSC chair

May was a busy month at EdgeX Foundry.  EdgeX released version 2.2, Kamakura in the middle of May (details here) and went straight into planning our fall release – code named Levski.  We also selected our 2022 EdgeX Award winners as well, and I’ll be posting a follow-up  to congratulate them and speak to some of their efforts.

Levski

First, what is the next release of EdgeX going to be about?  It is slated to be released in November of 2022.  It will, in all likelihood be another minor dot release (version 2.3 to be precise) that is backward compatible with all EdgeX 2.x releases.  The Levski release will also not be an LTS release.  Jakarta remains the first and only LTS for EdgeX for now (more on that below).  By the way, in case you are wondering where the name Levski (or the name of any EdgeX release) comes from, a top contributor to our project is selected in each release cycle to name an upcoming release.  Levski is named after one of our CLI developers’ favorite home-country (she is from Bulgaria) mountain peak.

You will see the words “likely” and “anticipate” show up a lot in this post because a lot can happen in the span of six months when building open-source software.  These caveats are there to remind the reader that what is planned is not set in stone – so if you are an adopter/user of EdgeX, design accordingly.

Major Features

Two of the most important features that we are looking to land in the Levski release are:

  • North-to-south message bus communications
  • Control plane (aka system management) events

North-to-south messaging will set up the means for a 3rd party application or cloud system to use MQTT to communicate with EdgeX’s command service and trigger an actuation or data fetch command all the way through to the devices/sensors.  

Today, communications going from north to south in EdgeX are accomplished via REST.  Therefore, things are not really asynchronous and there isn’t the opportunity to set the quality of service (QoS) of the communications in these instances.  Levski will allow adopters of EdgeX to use messaging to trigger actions and communicate data needs.  This compliments what is already available in EdgeX which is the ability to use messaging from south to north.  The design for this feature was finalized in this past Kamakura release cycle.  You can see the design document for more details.

The second major feature to highlight in Levski is control plane events (CPE) or also known as system management events.  This feature will allow EdgeX to send important events to would-be listeners (could be other EdgeX services, could be 3rd party subscribers) that something (an event) has happened in EdgeX.  Examples of control plane events are that a new device/sensor was provisioned or a new profile was uploaded.  Control plane events could also report on micro service issues – such as not having access to a database it needs.  Each EdgeX micro service will define its own control plane events, however in this release, it is anticipated that control plane events will first be tried with core metadata.

Other Features and Fixes

More Metrics

During the Kamakura release cycle, we implemented a new beta metrics/telemetry feature.  This feature allows any service to report (via message bus) on telemetry from the service that allows for better monitoring and management of EdgeX.  Telemetry such as how many events have been persisted, or how long does it take for a message to be processed in an app service can now be collected and published by EdgeX Metrics.  In this release cycle, we plan to proliferate metrics across all services (it was only in a few services for Kamakura) and take the feature out of beta status.

Experiment with MQTT authentication in service-to-service communications

Today, EdgeX has an API gateway to protect the REST APIs.  We also secure external message bus communications, but there is nothing that secures message bus communications internally.  In Levski, the community is going to be exploring possible solutions for securing MQTT message traffic.  We plan to explore and prototype with OpenZiti – a zero trust open-source project that provides an overlay network for secure communications between services and may even allow us to provide an alternative to our API gateway (Kong today).

Unit of Measures

In this past release cycle, we spent some time designing a solution that would allow units of measure to be associated to all of the edge sensor data collected.  There are many “standard” units of measure in the world.  We did not pick one, but allowed the adopter to pick / use the units of measure they want associated to their data.  In the Levski release, we hope to implement the design we formalized in the last release.  See our design documentation for more details.

Miscellaneous

The community is seeing more efforts by users/adopters to deploy EdgeX in non-containerized environments and wanting to build EdgeX for other environments (ARM32, RISC-V, etc.).  There is also a growing desire on the part of adopters to be able to have versions of our micro services that don’t include dependencies or modules that aren’t being used (for example, not including 0MQ when not using 0MQ for communications or not including security modules in a deployment that is not using EdgeX security features).  Therefore, the project is looking to provide more “build” from scratch options that remove unnecessary libraries, modules, etc. in environments where these features are not needed.  We call them “lite builds” and they should be more prevalent in Levski.

A growing demand for NATs as a message bus implementation in EdgeX will have the community doing some research and experimentation with NATS in this release cycle.  It is unlikely that full-fledged NATS message bus support comes out of this release, but we should begin to understand if NATs can be used as a message but implementation for our messaging abstraction and know the benefits of NATs over Redis Pub/Sub or MQTT for our use cases.

As you hopefully can see, Levski, while considered a minor release, will likely include a number of useful new features.

EdgeX 3.0

As part of the planning cycle, the EdgeX community leadership also considered the question of the next major release (EdgeX 3.0) and next LTS release.  By project definition, major releases are reserved for significant new features and inclusion of non-backward compatible changes.  We have tentatively slated the spring of 2023 as the target time for EdgeX 3.0.  We have been collecting a number of fixes and features that will require non-backward compatible changes.  We think the time will be right to work on these in the Minnesota release (code name for the spring 2023 release).  If all goes to plan, an LTS release will follow that in the fall of 2023.  The Napa release (code name for the fall 2023 release) would be the community’s second LTS release.  The 3rd major release and 2nd LTS release would fall exactly two years from our 2nd major release (Ireland in spring 2021) and first LTS (Jakarta in fall 2021).

Adopters should take some comfort in the fact that the project is healthy, stable and is looking out to the future.  As always, we welcome input and feedback on how EdgeX is supporting your edge/IOT solutions and what we can do better.

EdgeX Foundry Turns 5 Years Old & Issues its 10th Release

By Blog, EdgeX Foundry

By Jim White, EdgeX Foundry Technical Steering Committee Chair

EdgeX Foundry, an LF Edge project, turned five years old in April and celebrated with its 10th release!

Today, the project announced the release of Kamakura – version 2.2 of the edge/IoT platform.  EdgeX has consistently released two versions a year (spring and fall) since its founding in 2017.  Looking back to the founding and announcement of the project at Hannover Messe 2017, I would be lying if I was to say I expected the project to be where it is today.

Many new startup businesses don’t last five years. For EdgeX to celebrate five years, steadily provide new releases of the platform each year, and continue to grow its community of adopters is quite an achievement.  It speaks both to the need for an open edge / IoT platform and the commitment and hard work of the developers on the project.  I am very proud of what this community has accomplished.

We’ll take a closer look at EdgeX highlights over the past five years in an upcoming blog post, so for now, let’s dive into the Kamakura release. 

 

Here’s an overview of what’s new:

Kamakura Features

While this “dot” release follows on the heels of the project’s first ever long-term support release (Jakarta in the fall of 2021), the Kamakura release still contains a number of new features while still maintaining backward compatibility with all 2.x releases.  Notably, with this release, EdgeX has added:

  • Metrics Telemetry collection: EdgeX is about collecting sensor/device information and making that data available to analytics packages and enterprise systems in a consistent way so it can be acted on.  However, EdgeX had only minimal means to report on its own health or status.  An adopter could watch EdgeX service container memory or CPU usage or use the “ping” facility to make sure the service was still responsive, but that is about it.  In the Kamakura release, new service level metrics can be collected and sent through the EdgeX message bus to be consumed and monitored.  Some initial metrics collection is provided in the core data service (ex: number of events and reading persisted) for this release, but the framework for doing this system wide is now in place to offer more in the next release or allow an adopter to instrument other services to report their metrics telemetry as they see fit.  The metrics information can be easily subscribed to by a time series database, dashboard or custom monitoring application.

 

  • Delayed Start Services:  When EdgeX is running with security enabled, an initialization service (called security-secretstore-setup) provides each EdgeX micro service with a token which is used to access the EdgeX secrets store (provided by Vault).  These tokens have a time to live (TTL) which means if they were not used or renewed, they expired.  This created problems for services that may need to start later – especially those that are going to connect to future sensors/devices.  Furthermore, not all EdgeX services are known when the platform initializes.  An adopter may choose to add new connectors (both north and south) all the time to address new needs.   These types of issues and others often meant that adopters often had to shutdown EdgeX and restart all of EdgeX in order to provide new security tokens to all services.  In Kamakura, this issue has been addressed with a new service that uses mutual authentication TLS and exchanges a SPIFFE X.509 SVID for getting secret store tokens.  This new service allows a service to get or renew a token used to access the EdgeX secrets store anytime and not just at the bootstrapping / initialization of EdgeX as a whole.

 

  • New Camera Device Services:  While EdgeX somewhat supported connectivity to ONVIF cameras through a camera device service, the service was incomplete (for example not allowing the camera’s PTZ capability to be accessed).  With Kamakura, EdgeX will now have two camera device services.  A new ONVIF camera device service is more complete and tested against server popular camera devices.  It allows for PTZ and even the discovery of ONVIF compliant cameras.  A second camera service – a USB camera service – will help manage and monitor simple webcams.  Importantly, EdgeX will, through the USB camera service, be able to get a USB camera’s video stream to other packages (such as an AI/ML engine) to incorporate visual inference results into the EdgeX data sensor fusion capability.

 

  • Dynamic Device Profiles: in prior releases, device profiles were considered mostly static.  That is, once a device profile was used and associated to a device or any other EdgeX object like an event/reading, it was not allowed to change.  One could not, for example, have an empty device profile and add device resources (the sensor points collected by a device service) later as more details of the sensor or use case emerged.  With the Kamakura release, device profiles are much more dynamic in nature.  New resources can be added all the time.  Elements of the device profile can change over time.  This allows device profiles to be more flexible and adaptive to the sensors and use case needs without having to remove and re-add devices all the time.

 

  • V2 of the CLI:  EdgeX has had a command line interface tool for several releases.  But the tool was not completely updated to EdgeX 2.0 until this latest release.  For that matter, the CLI did not offer the ability to call on all 100% of the EdgeX APIs.  With Kamakura, the CLI now provides 100% coverage of the REST APIs (any call that can be done via REST can be done through the CLI) in addition to making the CLI tool compatible with EdgeX 2.0 instances.  Several new features were added as well to include tab completion, use of the registry to provide configuration, and improved result outputs.

The EdgeX community worked on many other features/fixes and behind the scenes improvements to include:

  • Updated LLRP/RFID device service and application services
  • Added or updated GPIO and CoAP services 
  • Ability to build EdgeX services on Windows platform (providing optional inclusion of ZMQ libraries as needed)
  • CORs enablement
  • Added additional testing – especially around the optional use of services
  • Added testing to the GUI
  • Optimized the dev ops CI/CD pipelines
  • Kubernetes Helm Chart example for EdgeX
  • Added linting (the automated checking of source code for programmatic, stylistic and security issues) part of the regular build checks and tests on all service pull requests
  • Formalized release and planning meeting schedule
  • Better tracking of project decisions through new GitHub project boards

Tech Talks

When EdgeX first got started, we provided a series of webinars called Tech Talks to help educate people on what EdgeX is and how it works.  We are announcing that a new Tech Talk webinar series will begin May 24 and run weekly through June.  The topic list includes:

  •   May 24 – Getting Started with EdgeX
  •   June 7 – Build an EdgeX Application Service
  •   June 14 – Build an EdgeX Device Service
  •   June 21 – Creating a Device Profile
  •   June 28 – Getting started with EdgeX Snaps

These talks will be presented by distinguished members of our community.  We are going to livestream them through YouTube at 9am PDT on the appointed days.  These new talks will help provide instruction in the new EdgeX 2.0 APIs introduced with the Ireland release.  You can register for the talks here:

What’s Next: Fall Release – “Levski”

Planning for the fall 2022 release of EdgeX – codenamed Levski – is underway and will be reported on this site soon.  As part of the Kamakura release cycle, several architectural decision records were created which set the foundation for many new features in the fall release.  This includes a north-south messaging subsystem and standardization of units of measure in event/readings. 

 EdgeX is 5, but its future still looks long and bright.

 

EdgeX 3.0 – the Future of Edge Computing

By Blog, EdgeX Foundry

By Jim White, EdgeX Foundry TSC Chair

Recently, the EdgeX Foundry project (a Linux Foundation project and part of the LF Edge umbrella of projects), released a dot release (v2.1) on top of our second major release which came out this summer.  This release was codenamed Jakarta and it is the community’s 9th overall release.  Jakarta was a stabilization release and is our first long-term support (LTS) release.

In reading my blog post title, you may be asking, what is this joker-of-a-technical-steering-committee-chairman talking about?  The paint on EdgeX 2.0 and our first ever LTS isn’t even dry and he’s talking about the third major release?

To put everyone’s mind at ease, EdgeX 3.0 is a long way off.  EdgeX 2.0 was a very big release.  The LTS release was a big commitment of support on the part of our community (we will support Jakarta for 24 months as defined by our LTS policy).  Furthermore, the community recently held its semi-annual planning meeting for the next release (codenamed Kamakura), and we know the spring 2022 release will be another dot release (EdgeX 2.2) with some additional new features but still backward compatible with all 2.x releases.  So, there is nothing on the immediate horizon that says EdgeX 3.0 is eminent.

I am not going to put a timeline on EdgeX 3.0 availability.  As a project, I am very proud of the fact that we have regularly released twice a year.  In 4 years of existence, EdgeX has had just 2 major releases.  EdgeX 3.0 would be the third, and given our current cadence.  Do the math and you can see that we are looking at more than year before EdgeX 3.0 is even remotely possible.  EdgeX 3.0 here serves as a metaphor of what big things are on the horizon for the project.

What I am going to pose in this post is a vision and a roadmap for the future of EdgeX Foundry.  I have the privilege of owning a front row seat in the creation and use of EdgeX for more than 6 years now.  I don’t suggest that I have all the answers or perfect vision with regard to the future, but I think my position grants me enough context and understanding to make some forecasts.  History combined with current requirements sprinkled with a few strong technical indicators can present a pretty good directional.  

To provide future me and my prognostications some escape, I include a couple of caveats with regard to my “vision”:  major industry disruption and time.  What I am presenting is a vision based on technology we know today or expect tomorrow.  Significant disruption (i.e., Internet or smartphones size impacts) is not something I can predict and it would potentially make this roadmap useless.  I am also not suggesting everything I am envisioning will be or even has to be in EdgeX 3.0.  The vision is a guidepost, but the journey may take a bit longer and more than one major release to accomplish – so again EdgeX 3.0 is metaphorical.  

OK – that said, allow the crazy TSC chairman to paint the picture of EdgeX 3.0.

Take the Bus but Allow Walking

When EdgeX was initially introduced, all service communications were via REST.  Whether your application needed something from an EdgeX service or two EdgeX services needed to talk, we used REST over HTTP.  This was a conscious decision.  We felt that REST was clean, simple and well understood.  If you are looking to get adopted and you have a complex tool to solve complex problems, you at least want to make it easy to communicate with.  REST is pretty easy to work with.

Quality of service needs, throughput efficiency, pub/sub model vs point-to-point communications, and message size are reasons why architects choose a message bus approach over REST.  These reasons are valid reasons to adopt messaging in edge/IoT computing too.

Over time, EdgeX has instituted more message bus communications for our services.  As of EdgeX 2.0, All communications from sensor to enterprise/cloud (the northward travel of the sensor data), can be done via message bus to deliver edge data.

 

In EdgeX 3.0, adopters must have the opportunity to use a message bus for all communications.  Most notably, an enterprise application or cloud system should be able to send a message on a message bus (MQTT, AMQP, Redis Streams, etc.) to request actuation on a device/sensor or to get details about what is happening at the edge.  Today, communications from north to south are all via REST.  Queries into EdgeX services or requests to change metadata, schedules or any type of configuration are also via REST.  Going forward, EdgeX 3.0 needs to allow communications with all services to happen by message bus.  That means all EdgeX services will need to be outfitted with topic endpoints and subscriptions to receive messages as well as the means to publish a message to a message bus.

This does not mean REST should be removed.  EdgeX is also about flexibility and simplicity.  REST interfaces suffice for some production use cases.  Further, REST makes it easier for a student of EdgeX to get up to speed and explore the platform.  REST also offers a great deal of assistance when debugging a broken system.  Use of REST allows users to walk around the services with a simple browser (whether trying to learn it or fix it), and with no additional setup or tools.

Finally with regard to EdgeX 3.0 and messaging, the platform should also embrace AMQP as an alternative to MQTT and Redis Pub/Sub (and ZMQ in some limited cases).  AMQP offers more features such as better support for cache and proxy.  AMQP is popular among some industries (finance and business) and is used in some IoT circles (notably supported in Microsoft’s Azure IoT Hub).  Because AMQP requires more resources (see issues with resource constraints below), it should not be the default message bus implementation.  Providing an optional implementation of the EdgeX message bus via AMQP, however, allows EdgeX to compete with products at the edge using this protocol and give it one more tentacle of flexibility.

Size Matters and Time is Relative

Since the beginning of my journey in IoT/edge computing, we were always working in limited resource environments but with the hope and expectation that Moore’s law would eventually be applied at the edge just as it has on our desktops, phones, enterprises and clouds.  This is a false assumption.  

I do think that the availability of resources at the edge is trending upward – it just happens much more slowly than every two years (per the law).  Why?  Two reasons: scale and time.

If your company was to replace all of its employee laptops, how many laptops would they be replacing?  Hundreds?  Thousands?  Sizeable yes.  And not cheap, but IT organizations rotate our laptops about every 2-3 years as a matter of course.  Agile development practices have our enterprise applications updated every few weeks.

Now imagine you are a city and you want to put an edge compute platform on every traffic signal.  Back of the napkin math says there are about 200 to 6000 compute platforms you are going to need for a square mile of city (using 4-12 lights per intersection, and about 45-550 intersections per square mile in a city – thanks Google).  How many square miles in a city?  Given Philadelphia has around 130 mi2 and Seatle has around 80 mi2, let’s just say 100 mi2 as a good working number.  That means we would need to field 20K to 600K platforms!  The cost to stick a high-end server with lots of resources on each traffic light would break a city.  The sheer scale of the deployment is too big to expect the use of something other than smaller/cheaper edge platforms.  This is not an isolated case.  IoT / edge deployments tend to get very big, very quickly.

Second reason – Moore’s law generally applies to information technology (IT) areas much better than it applies to operational technology (OT) areas.  Einstein was right – time is relative.  In OT realms, the rate of change is slower – much slower.  So even if companies, governments and institutions had access to cheaper, faster and bigger, stronger technology, deploying it into the hot, smelly, dirty, wet, corrosive and generally hard to reach places that edge computing goes to doesn’t happen overnight.  Getting to these systems can be a nightmare.  Most systems in OT stay in place at least 7 years.  It is not unusual to find edge technology that is in place for 15 years or more.  This is a far cry from the 2-3 year (or less) upgrade cycles found in IT environments.

Consequently, the environments that a platform like EdgeX needs to run in are constrained and likely to remained constrained for some considerable time into the future.

EdgeX started life as a Java platform.  It was too big to meet edge resource constraints.  We’ve worked hard to get our micro services lean and to operate in limited resource environments.  We also made an increasing number of services optional (use a minimal set of EdgeX services to support bare bones use cases).

As a project, we have to fight the urge to add to the platform such that it can only be used in expensive, non-resource constrained environments.  From the project’s onset, we used a small Raspberry Pi as our guidepost.  We were not endorsing Raspberry Pi.  It was a measuring stick.  If we could run EdgeX on that platform, we were within the realm of being able to run in a smaller, resource constrained, and yes cheaper, edge environment.

That measuring stick is still relevant to EdgeX 3.0 for reasons stated above.  

  • EdgeX needs to run in 1GB of RAM or less.  
  • EdgeX can run on a single core platform
  • EdgeX needs to run within 32GB of storage space.  
  • EdgeX needs to be able to ingest sensor data, make a decision (from a rules engine or other analytics package), and actuate a device in less than one second (within a single instance of EdgeX).
  • EdgeX needs to startup and be operational is 10 seconds or less.

There are some implications of this that are mentioned in other future platform decisions below (such as use of 3rd party components).  But importantly, as we add to EdgeX, we must take care not to allow EdgeX to get so big as to grow beyond the constraints of its slow-changing OT environment.

Adopt Versus Build – Finding Edge Worthy Components

For expediency and because the areas were not our specialty, the EdgeX community choose to adopt some enterprise grade open-source products.  We chose Drools initially to give us a rules engine to drive low latent decisions at the edge.  We incorporated Kong and Vault to provide our security API gateway and secret store.  We use Consul for our configuration/registry service by default.  We initially chose MongoDB for persistence.  

These were the right decision at the time.  We took a path where we do the edge and we let others do the things that they are good at.  “You do you” we said to 3rd party components and “we’ll do edge.”  Security, in particular, is not something we felt we should be creating on our own.

Over time, we have chosen alternates to help lower the EdgeX resource footprint.  We replaced MongoDB with Redis.  We chose eKuiper to replace Drools.  Still, it was a relatively hands off approach to incorporating best of breed 3rd party components.  Again – “you do you” and “we’ll do edge.”

But many of these choices, due to their enterprise and cloud native nature, are big – too big.  They are wonderful, but offer more capability than what will ever be used at the edge. They were not built for the edge.  eKuiper being an exception, but even there we have seen a large expansion of their feature set to address more (and perhaps more than the edge needs?).  The you-do-you, and we-do-edge approach is something we need to reconsider for EdgeX 3.0.  As a customer said to me recently, “EdgeX needs a diet plan.”

Have a look at the EdgeX performance numbers in the table below.  With regard to EdgeX services, most are under 25MB or less image footprint.  Memory usage by any EdgeX service is around 11MB and CPU utilization is miniscule.  Now note the size of the non-EdgeX services (those without the EdgeX prefix).  The numbers are stark.  Over the last few releases, EdgeX services are trending within a few megabytes of their original performance numbers (some even getting smaller).  3rd party services are getting bigger.    Its not that EdgeX needs a diet plan so much as we need to put our 3rd party components on a diet plan.

It is the 3rd party components that are costing EdgeX most on its ability to fit on resource constrained platforms.  There is a chance that there are still other, smaller options for some of these components.  But are the options built for the edge and with an edge attitude toward resource constraints? 

EdgeX has come to a point where it must consider one of a few options in adoption of 3rd party components:

  • Locate 3rd party component providers that are thinking edge resource constraints and use their smaller components to replace the current enterprise sub-systems we use today.
  • Partner with the current 3rd party component providers; give them our edge and resource requirements, and see if they are willing to create smaller, lighter versions of their products for the edge.
  • Create our own smaller, lighter components to address edge needs.  Perhaps start from trying to take an existing open-source product and cut it down; more limited in functionality but fine for the edge.
  • Allow adopters to unplug the heavy bits or implement their own.  Some use cases don’t require the 3rd party components.  When they are required, and by providing abstractions for all 3rd party components, EdgeX can continue to use the same 3rd party components as implementations for those abstractions, but make it easier for adopters and commercial implementers to create or select alternate implementations as use case resource constraints dictate.

I prefer these in order.  Finding other alternates has so far proven to be a challenge.  I recognize that other projects have their own priorities and supporting edge computing may not rise to a level of their concern (although one would hope that the size of the IoT edge opportunity would attract some).  Almost certainly, a project like eKuiper (now a sister project in LF Edge) would be open to finding ways to modularize or otherwise offer resource constrained functionality for resource constrained edge platforms.

Where we cannot find or convince a third party, open-source project to help meet our component need, EdgeX may have to look at implementing their own.  A dubious sounding task, but in actuality may not be that daunting given that edge needs are a map-reduce function against the bigger enterprise versions.  Depending on the architecture, we may find we can take a selective scalpel to the 3rd party component and create an edge worthy edition reasonably quickly.

There are some use cases where the security components are not needed (in physically restricted environments for example).  Some adopters may just drop the weight of our enterprise level components if we make it easy for them.  Forcing adopters to have to create their own components based on our abstractions should be used only as temporary fix or the answer when well-known, proprietary options exist for adopters to use in production deployments.

No longer can we just do “edge parts”.  If a 3rd party component was built for the enterprise / cloud and is not meeting our edge needs, we must begin to explore alternatives – to include our own implementation – to satisfy edge resource constraints.

We are going to want to add additional capability to EdgeX (as discussed in various parts of this document).  We are going to want to improve the platform.  If we want to stay within the resource constraints of our target host platform, we are going to need to reduce resource consumption in 3rd party areas to allow EdgeX to grow in other areas.

Not Cloud Native – Aim for Edge Native

Cloud native is the current rage in software development.  It is a term used to describe building and running applications that exploit cloud computing delivery models.  That is, the ability for an application to access compute resources in more of an on-demand way with scale, resiliency and flexibility in mind.  Specifically, cloud native applications take advantage of micro service architectures, containerized deployment and orchestration, agile development process, 12-factor app patterns, and good DevOps automation to usher work from code to deployable artifact.

There is a movement to drive cloud native principles in software engineering to the edge.  While there are some good aspects of cloud native computing that can be applied directly, using a cloud native approach on edge applications forgets many of the constraints of the edge (see here for a good list of some of those differences).  Others are calling for a modified approach that considers the unique requirements of the edge.  That is, they are cherry-picking elements of cloud native but still considering the constraints of the edge.  This approach is called edge native.  

EdgeX has adopted some elements of the edge native approach (small micro services, service availability tolerance, good DevOps automation, etc.).  In fact, EdgeX adopted some of these elements even before the term edge native existed.  I believe that our EdgeX community would agree with the general philosophy in edge native computing.  That is, we would agree that edge applications should be “built from the ground up with the Edge in mind – just like Cloud-Native applications are built for the Cloud.”  

The definition of edge native is still somewhat nebulous and debated.  Depending on the source, EdgeX adheres to some elements of edge native, but does not adhere to other characteristics.  And there are some guidelines of developing edge native applications that I would not suggest EdgeX adhere to blindly. 

As an example, use of containers is considered a staple (required?) in many edge native guidelines.  EdgeX has always provided containers, but doesn’t require use of containers.  In fact, the project supports alternate delivery technologies (like snap packages – developed by Canonical) and understands the reality of today’s edge infrastructure.  

  • Some OT groups have not embraced containerized workloads and have suggested it may be several years before they support containers in production.  
  • Some resource constrained platforms make use of containers impossible.  
  • Certain devices/sensors are going to make an all-container strategy a challenge.  

While EdgeX supports container use, it does not dictate it.  Flexibility is key so long as the edge is very heterogenous.  

What principals of edge native computing should EdgeX look to embrace that it does not do today?

Distributable

Edge native applications should be able to move as resources (compute, network, storage, etc.) dictate.  Services should be able to scale out to the edge or scale back to the enterprise / cloud as resources warrant.  For example, an application service could run at the far edge and provide for low-latency decisions when resources are available.  But they may also run in the enterprise or in the cloud when resources aren’t available.

In theory, EdgeX services were designed to be distributed on different hosts (on physical systems or virtual machines with a different IP address).  However, in reality, there are a number of issues with the current architecture that might make distributing services across hosts difficult.   Chief among them is that when the services are distributed, there isn’t an easy way to secure the communications between services – a necessity for edge native applications.  Providing central configuration and secrets across the distributed services is not fully addressed in EdgeX.  EdgeX 3.0 needs to allow any EdgeX micro service to live anywhere, anytime and still operate securely.

Resiliency and Rapid Recovery

Enterprise and cloud platforms and associated resources (compute, network, storage, etc.) are relatively stable.  Edge platforms are notoriously unstable.  EdgeX 3.0 needs services to be more resilient to edge failures and outages.  When service failures do occur, the services need to be brought back up quickly.

This does not mean EdgeX services need to be “highly available.”  High availability (HA) is the ability of a system or service to operate continuously without failing for long, agreed upon lengths of time.  It often requires some orchestration capability to monitor the services and offer “backup” or redundant services in the face of failure.  The resource constrained nature of edge platforms makes offering HA at the edge a particular challenge.

Dependent services will go down.  Resources like the network will become unavailable.  The EdgeX 3.0 services must be built to be withstand these issues.  When a service does go down, it must be able to be restarted quickly and not require a lot of new configuration or setup on the part of the user (such as new security keys that cannot automatically be provisioned).

I submit that in EdgeX 3.0, a service remains up and continues to try to acquire any dependent services or resources indefinitely unless trying to do so creates other issues.  It should also be able to use the alert/notifications service (if it is up) to alert on the situation to a default HTTP REST endpoint, send an email, SMS message or otherwise alert the systems overseer.  When the dependency is available, the service should come back on line and continue normal functions without the need for other intervention on the part of the user.

When a service fails, it should be able to restart within the one-minute system startup time and without other intervention on the part of the user that had to issue the start (or restart) command.

The community should also offer example scripts or services that would check that all services are functioning and when a service (or dependent facility such as the database) are down, it attempts to restart them.  The example “restart” service or script does not have to be part of a default EdgeX 3.0 deployment, but should serve to help adopters think about how to keep an edge system alive even during partial failures.

Indeed, when something like Kubernetes is used to deploy and orchestrate EdgeX services, resiliency and rapid recovery may be offered taken care of by that environment.  But EdgeX must always be able to handle itself when platform constraints don’t allow for CNCF types of deployment management.

The Other Data – EdgeX Control Plane Telemetry and Health / Monitoring

Today, EdgeX does not report on its own health and operations.  Therefore, there is no way to automate any type of higher order management of the platform.  For example, there would be no way to know if a device service is reporting sensor data as expected – at least not without manual intervention and exploration of log files (provided the right log levels are set).

Starting with the Delhi release, EdgeX offers a system management service.  This service was created to be able to perform a limited set of monitoring and control plane functions.  This includes:

  • Starting, stopping, restarting the EdgeX services
  • Providing memory and CPU usage of the service
  • Provide a service’s configuration
  • Provide an indication of whether the service is operational based on its response to ping requests

Because some of these functions (start, stop, restart and metrics collection) are dependent on how the services are deployed and running (via container or on a native Linux OS), the architecture of the service requires an “agent” request information from a specific platform-dependent “executor” to carry out much of the system management service functionality.  The system management service, therefore, suffers from several issues:

  • It doesn’t work for all environments (Windows)
  • It has to be informed of all new or removed EdgeX services (not easy to do in dynamic situations)
  • The architecture (two components versus a single service) makes it more difficult to setup and run

Perhaps, most importantly, the service isn’t providing any capability that couldn’t be obtained from use of other tools depending on how EdgeX is deployed and where it is running.  For example, if EdgeX is deployed via Docker containers, Portainer or Linux tools could be used to do most of the system management functions except for providing configuration which is always available via Consul.

The intention was that the system management service would eventually be extended to get more information from each service (telemetry and event information that was specific to EdgeX) and make that available to monitoring services and/or other applications.  The telemetry and event information is not something an outside tool could provide since it would require knowledge and access to EdgeX service internals.  

Where that leaves the project is with a service that is redundant to better tools and technology and unable to provide much needed (and otherwise unobtainable) telemetry data that it should be providing.  For this reason, the community has decided to mark the service deprecated with the Jakarta release.  It may not be removed from the platform until something else provides for its replacement, but at least the community is signaling that the system management service in its current form is nearing end of life.

By the way, as reported earlier in this post, the system management agent (SMA) is the most expensive service in the EdgeX inventory by a factor of 10.  So, removing this service as it exists today in EdgeX paves the way for additional features going forward.

EdgeX 3.0 needs to offer much more data to adopters about its health and operations.  Telemetry or metrics from each service can be used to understand whether a sensor is reporting correctly, if the system has the proper resources to support adding additional sensors, or even if sensors are being used to flood the system with information (denial of service through sensors).

Rather than collecting telemetry by having an outside service request (pull) it, each service needs to be able to publish (push) telemetry out.  The telemetry can be pushed anywhere.  In early implementations, EdgeX services may just be configured to send telemetry to a designated message topic where it is up to the adopter to figure out how to collect, use or respond to any telemetry.  Later, the telemetry data can be treated as alternate edge data (control plane data vs sensor data) that is consumed by EdgeX application services, rules engines or other analytics services.  Telemetry data does not originate from a “thing”, but its data can help drive operations and actuation at the edge as necessary.  

For example, imagine a device service reported telemetry about the number events it produced over a given period of time.  If the telemetry – in the form of control plane Events/Readings – were picked up by application services and routed to the rules engine, the rules engine could be configured to look out for sensors reporting more events than expected.  This could trigger the shutdown of that rogue sensor until a user explores the reason for the extra reporting.

Application services could also subscribe to service telemetry messages in order to filter, aggregate and otherwise prepare and ship telemetry data out of EdgeX – just as they are used to export sensor data today.

Each service is responsible for certain functions and responsibilities within an EdgeX instance.  Each service knows (or should know) what is critical to its operations and functions.  Therefore, each service should have the means to report telemetry specific and important to that service, and make this EdgeX specific information available to EdgeX as well as external systems.

The Event/Reading structure and services that handle Event/Readings may need to be modified slightly in recognition that the Event/Reading may contain sensor data or service telemetry data.   Hopefully, Telemetry data and sensor data Event/Readings will differ only slightly in EdgeX 3.0 thereby requiring only minimal change.

EdgeX 3.0 services will need additional configuration so as to control how much and what telemetry gets reported.  The user should be able to increase or decrease the telemetry reporting based on operational circumstances and use case needs (not unlike how logging output is adjusted today).  Telemetry or metrics collection can impact the performance of EdgeX services.  There may be a need, based on resource constraints in some deployments, to complete turn off telemetry collection and publication for the entire instance.

The basic premise of initial EdgeX telemetry collection is already specified in a proposed EdgeX ADR.

More EdgeX 3.0 Features

In addition to the vision provided above, EdgeX 3.0 will need to add some additional capability that it does not have today.

Alternate Language App Functions SDK

EdgeX has two device service (DS) software development kits (SDKs); one in Go lang and the other in C.  Since most of the platform is created in Go lang, it seems fitting and natural to have a DS SDK in Go.  We also have a DS SDK in C because it is the most natural fit for most low-level protocols and “thing” communications.

On the north side, we have just a single App Functions SDK to create new/custom application services (AS) in Go Lang.  Again, given EdgeX’s foundation in Go, a Go lang SDK here also makes sense.  Going forward, I anticipate more interfaces with artificial intelligence (AI) / machine learning (ML) and other analytics packages.  One of the more popular languages for the AI/ML communities is Python.  It would seem appropriate that EdgeX 3.0 provide a north side SDK that is more familiar to those more likely to need and build AS.  Other organizations are using EdgeX to translate from one OT protocol (like Modbus) to another OT protocol (like BACnet or OPC UA).  OT people tend to use C and C++ and it stands to reason that an App Functions SDK in one of these languages may better support their needs.  In general, the SDKs at either north or south end of EdgeX need to tend toward the language tendencies of the user groups most likely to use them. 

Distributed Ledger Support

IoT platforms are anticipated to generate 80 zettabytes of data by 2025 by one recent report.  As that data begins to be shared – potentially even bought and sold on the open market – the origination, legitimacy, and ownership of that data is going to need to be monitored and managed.  This screams for distributed ledger technology (DLT).  The closer that the data can be tracked and attributed, the better.  This means that IoT/edge platforms like EgdeX will need to integrate with DLT platforms in order to put its sensor data into a digital ledger.

EdgeX 3.0 will need to offer some initial integration to popular, open source DLT.  DLT can be resource intensive.  So DLT at the edge may be an option to be used only when the use case dictates it and the edge platform can support it.

K8s Is Coming

I said earlier that EdgeX 3.0 will move closer to edge native, but that doesn’t mean it has to offer high availability.  However, without a doubt, Kubernetes (K8s) is moving to support the edge.  K3s, MicroK8s, KubeEdge and other efforts stemming from the Kubernetes are just the forerunners to what will be some type of K8s support at the edge.

Not every organization will support its use at the edge – it can be complex.  Not all edge environments will have the resources to run some form of K8s at the edge.  And using K8s in whatever form it takes to deploy and orchestrate edge native applications will have to make some allowances for edge constraints and challenges.  But make no mistake, like winter, K8s is coming.  EdgeX 3.0 will need to do more for those looking to use K8s.  Providing deployment / orchestration assistance equal to that which we do for Docker Compose and Snaps will be imperative.

We have always claimed to be deployment/orchestration technology agnostic.  We will remain so.  But at some point, we should expect some form of K8s technology (probably a far cry from what we see today which is largely inadequate to edge native) to be the predominant means of deploying edge applications.  We will also have to educate those coming from a cloud native world why K8s is not always a fit for edge native.

UoM

We have already begun a conversation in the community about how to support unit of measure standards with regard to our sensor data collection.  That is, how to attribute sensor data to a well-defined unit of measure.

IoT / edge platforms have been be specific about the sensor data collected.  For that matter, devices are pretty lack with regard to how they send data.  I own two IoT temperature probes that send an integer of “793” when trying to tell me the temperature reading is 79.3 Fahrenheit.

In order for the sensor data to be more trusted and offer better value (see DLT comments above), the data will need to be tied to an appropriate unit of measure and that unit of measure should be well defined by some body.

EdgeX 3.0 will not create or dictate the unit of measure standard, but it will allow those that need more specificity to label sensor data with a unit of measure and the standard it comes from.

This discussion and the premise for a unit of measure solution are already specified in a proposed EdgeX ADR.

Automate Thing Provisioning

The “last mile” is a term that originated in supply chain management and then adopted by telecommunications. It was used to describe the most difficult part of a journey or furthest part in a network – which usually was found at the end or literally the last mile.

In edge computing, the last mile is that between our platform and the actual device or sensor.  Connecting the sensors or devices with all the protocols, data formats, complex hardware, etc. is a real challenge and is why EdgeX is so important.  EdgeX, as we know, helps to simplify and standardize how “things” of the OT world get connected to the IT world.

While EdgeX makes the last mile shorter, so to speak, there is still a fair amount of ceremony and work required to provision a new sensor.  As an EdgeX user, you still have to provide a device profile, issue the correct Metadata calls or providing the right device configuration in order to onboard a new “thing”.  We provide some amount of device discovery and automated provisioning, but it is limited to a few protocols and doesn’t go far enough.  

Many sensors/devices today can provide more information about the resources they have to offer or the actuation commands they support.  Many sensors/devices advertise their presence and offer platforms like EdgeX the ability to discover and onboard them automatically.

EdgeX 3.0 needs to complete the journey and make the last mile of connectivity easier for adopters.  Where possible, and with appropriate safeguards, EdgeX 3.0 should be able to onboard a new senor/device as soon as it is powered up and in communication range of EdgeX and the host it runs on.

There will be challenges, especially with more legacy protocols.  However, an edge platform that enables “thing” connectivity simply by powering the sensor on will help to make IoT / edge computing truly more ubiquitous.  

Wrap Up

The opinions and vision expressed in this post are my own.  While I am currently the EdgeX Foundry TSC chairman, the vision depicted here is not yet the opinion adopted by the EdgeX community.  It is not the codified roadmap for EdgeX.  I hope it will be – or that the community will take this vision and improve upon it, which they usually do.  “A goal is not always meant to be reached; it often serves simply as something to aim at.”

We’ve spent the last two years working on EdgeX 2.0 and our first LTS release.  I am so proud of the work that was accomplished and where EdgeX is at, but I don’t want adopters or the community to think we have reached the end.  We are not at the end.  We are at the beginning of creating the best open source IoT / edge platform on the planet.