Category

Project EVE

How Do We Prevent Breaches Like Verkada, Mirai, Stuxnet, and Beyond? It Starts with Zero Trust.

By Blog, Project EVE

By Jason Shepherd, LF Edge Governing Board Chair and VP of Ecosystem at ZEDEDA

This blog originally ran on the ZEDEDA Medium blog. Click here for more content like this.

News of the Verkada hack that breached 150,000 surveillance cameras sent shockwaves through the security world last week. For those of us in the edge computing industry, it simply underscored what we already knew — securing distributed devices is hardAs intelligent systems increasingly expand into the field, we’ll see more and more attacks of this sort if we continue to leverage the same security stance and tools that we’ve used for decades within perimetered locations like data centers and secure telecommunication facilities.

In this article, I will go through a few of the widely-cited edge breaches in the industry and highlight how a zero trust security model optimized for the unique challenges of the edge, such as employed by ZEDEDA, would have helped prevent them from happening.

In a hack reminiscent of Verkada, the 2016 Mirai virus infected millions of cameras and turned them into bots that together launched a massive DDOS attack on upstream networks, briefly taking down the internet in the Northeastern US. Something on the order of under twenty combinations of username and password got into all those cameras, because the developers made it too easy to change, or not even possible to change, these security credentials. Often this was due to prioritizing usability and instant gratification for users over security.

Another commonly-cited example is the massive Target data breach in 2014 that was a result of attackers accessing millions of customers’ credit card information by way of a networked HVAC system. The hackers stole credentials from a HVAC contractor and were able to access the payment system because the operations network the HVAC was on wasn’t properly segmented from the IT network.

In a final example, the 2010 Stuxnet breach involved malware that was loaded into process control systems by using a USB flash drive to bypass the network air gap. The worm then propagated across the internal process control network, scanning for Siemens S7 software on industrial PCs. When successful, the virus would send unexpected commands to PLCs controlling industrial processes while giving the operators a view of normal operation.

Viruses like Stuxnet that focus on compromising industrial systems are especially concerning because attacks can lead to immediate loss of production, or worse life. This is compared to breaches of IT systems which typically play out over long periods of time, with compromises to privacy, financial data and IP.

With these examples in mind, what is unique about the proverbial edge that makes security such a challenge?

  • : Part of the value of IoT and edge computing comes from having devices connected across the organization, providing a holistic view of operations. Over time we will see edge device deployments grow into the trillions, and traditional data center solutions for security and management are not designed for this kind of scale.
  •  Distributed edge computing resources rarely have the defense of four physical walls or a traditional network perimeter. This requires a security approach that assumes that these resources can be physically tampered with and doesn’t depend upon an owned network for defense.
  • The edge is at the convergence of the physical and digital worlds. In addition to a highly heterogeneous landscape of technologies, we also have to account for diverse skill sets spanning IT and OT (e.g. network and security admins, DevOps, production, quality and maintenance engineers, data scientists).
  •  As OT and IT converge at the edge, each organization’s often conflicting priorities must be considered. While OT typically cares about uptime and safety, IT prioritizes data security, privacy and governance. Security solutions must balance these priorities in order to be successful.
  • Many IoT devices are too resource-constrained to host security measures such as encryption, plus the broad install base of legacy systems in the field was never intended to be connected to broader networks, let alone the internet. Because of these limitations, these devices and systems need to rely on more capable compute nodes immediately upstream to serve as the first line of defense, providing functions such as root of trust and encryption.

The Verkada hack and its predecessors make it clear that edge computing requires a zero trust architecture that addresses the unique security requirements of the edge. Zero trust begins with a basic tenant — trust no one and verify everything.

At ZEDEDA, we have built an industry-leading zero trust security model into our orchestration solution for distributed edge computing. This robust security architecture is manifested in our easy-to-use, cloud-based UI and is rooted in the open source EVE-OS which is purpose-built for secure edge computing.

ZEDEDA contributed EVE-OS to form Project EVE in 2019 as a founding member of the Linux Foundation’s LF Edge organization, with the goal of delivering an open source, vendor-agnostic and standardized foundation for hosting distributed edge computing workloads. EVE-OS is a lightweight, secure, open, universal and Linux-based distributed edge operating system with open, vendor-neutral APIs for remote lifecycle management. The solution can run on any hardware (e.g., x86, Arm, GPU) and leverages different hypervisors and container runtimes to ensure policy-based isolation between applications, host hardware, and networks. The Project EVE community is now over 60 unique developers, with more than half not being affiliated with ZEDEDA.

EVE-OS Zero Trust Components

Let’s take a look at the individual components of the EVE-OS zero trust security framework.

  • EVE-OS leverages the cryptographic identity created in the factory or supply chain in the form of a private key generated in a hardware security model (e.g., TPM chip). This identity never leaves that chip and the root of trust is also used to store additional keys (e.g., for an application stack such as Azure IoT Edge). In turn, the public key is stored in the remote console (e.g., ZEDCloud, in the case of ZEDEDA).
  • An edge compute node running EVE-OS leverages its silicon-based trust anchor (e.g., TPM) for identity and communicates directly with the remote controller to verify itself. This eliminates having a username and password for each edge device in the field, instead all access is governed through role-based access control (RBAC) in a centralized console. Hackers with physical access to an edge computing node have no way of logging into the device locally.
  • EVE-OS has granular, software-defined networking controls built in, enabling admins to govern traffic between applications, compute resources, and other network resources based on policy. The distributed firewall can be used to govern communication between applications on an edge node and on-prem and cloud systems, and detect any abnormal patterns in network traffic. As a bare metal solution, EVE-OS also provides admins with the ability to remotely block unused I/O ports on edge devices such as USB, Ethernet and serial. Combined with there being no local login credentials, this physical port blocking provides an effective measure against insider attacks leveraging USB sticks.
  • All of these tools are implemented in a curated, layered fashion to establish defense in depth with considerations for people, process, and technology.
  • All features within EVE-OS are exposed through an open, vendor-neutral API that is accessed remotely through the user’s console of choice. Edge nodes block unsolicited inbound instruction, instead reaching out to their centralized management console at scheduled intervals and establishing a secure connection before implementing any updates.

Returning to the above examples of security breaches, what would the impact of these attacks have looked like if the systems were running on top of EVE-OS? In short, there would have been multiple opportunities for the breaches to be prevented, or at least discovered and mitigated immediately.

  •  In the Verkada and Mirai examples, the entry point would have had to be the camera operating system itself, running in isolation on top of top EVE-OS. However, this would not have been possible because EVE-OS itself has no direct login capabilities. The same benefit would have applied in the Target example, and in the case of Stuxnet, admins could have locked down USB ports on local industrial PCs to prevent a physical insider attack.
  • In all of these example attacks, the distributed firewall within EVE-OS would have limited the communications of applications, intercepting any attempts of compromised devices to communicate with any systems not explicitly allowed. Further, edge computing nodes running EVE-OS deployed immediately upstream of the target devices would have provided additional segmentation and protection.
  • EVE-OS would have provided detailed logs of all of the hackers’ activities. It’s unlikely that the hackers would have realized that the operating system they breached was actually virtualized on top of EVE-OS.
  • Security policies established within the controller and enforced locally by EVE-OS would have detected unusual behavior by each of these devices at the source and immediately cordoned them off from the rest of the network, preventing hackers from inflicting further damage.
  • Centralized management from any dashboard means that updates to applications and their operating systems (e.g. a camera OS) could have been deployed to an entire fleet of edge hardware powered by EVE-OS with a single click. Meanwhile, any hacked application operating system running above EVE-OS would be preserved for subsequent forensic analysis by the developer.
  • Thezero-trust approach, comprehensive security policies, and immediate notifications would have drastically limited the scope and damage of each of these breaches, preserving the company’s brand in addition to mitigating direct effects of the attack.

The diverse technologies and expertise required to deploy and maintain edge computing solutions can make security especially daunting. The shared technology investment of developing EVE-OS through vendor-neutral open source collaboration is important because it provides a high level of transparency and creates an open anchor point around which to build an ecosystem of hardware, software and services experts. The open, vendor neutral API within EVE-OS prevents lock-in and enables anyone to build their own controller. In this regard, you can think of EVE-OS as the “Android of the Edge”.

ZEDEDA’s open edge ecosystem is unified by EVE-OS and enables users with choice of hardware and applications for data management and analytics, in addition to partner applications for additional security tools such as SD-WAN, virtual firewalls and protocol-specific threat detection. These additional tools augment the zero trust security features within EVE-OS and are easily deployed on edge nodes through our built-in app marketplace based on need.

Finally, in the process of addressing all potential threat vectors, it’s important to not make security procedures so cumbersome that users try to bypass key protections, or refuse to use the connected solution at all. Security usability is especially critical in IoT and edge computing due to the highly diverse skill sets in the field. In one example, while developers of the many smart cameras that have been hacked in the field made it easy for users to bypass the password change for instant gratification, EVE-OS provides similar zero touch usability without security compromise by automating the creation of a silicon-based digital ID during onboarding.

Our solution is architected to streamline usability throughout the lifecycle of deploying and orchestrating distributed edge computing solutions so users have the confidence to connect and can focus on their business. While the attack surface for the massive 2020 SolarWinds hack was the centralized IT data center vs. the edge, it’s a clear example of the importance of having an open, transparent foundation that enables you to understand how a complex supply chain is accessing your network.

At ZEDEDA, we believe that security at the distributed edge begins with a zero-trust foundation, a high degree of usability, and open collaboration. We are working with the industry to take the guesswork out so customers can securely orchestrate their edge computing deployments with choice of hardware, applications, and clouds, with limited IT knowledge required. Our goal is to enable users to adopt distributed edge computing to drive new experiences and improve business outcomes, without taking on unnecessary risk.

To learn more about our comprehensive, zero-trust approach, download ZEDEDA’s security white paper. And to continue the conversation, join us on LinkedIn.

Scaling Ecosystems Through an Open Edge (Part Three)

By Blog, LF Edge, Project EVE, Trend

Getting to “Advanced Class”, Including Focusing on Compounding Value with the Right Partners

By Jason Shepherd, LF Edge Governing Board Chair and VP of Ecosystem at ZEDEDA

 

Image for post

This blog originally ran on the ZEDEDA Medium blog. Click here for more content like this.

Thanks for continuing to read this series on ecosystem development and the importance of an open edge for scaling to the true potential of digital transformation — interconnected ecosystems that drive new business models and customer outcomes. In parts one and two, I talked about various ecosystem approaches spanning open and closed philosophies, the new product mindset related to thinking about the cumulative lifetime value of an offer, and the importance of the network effect and various considerations for building ecosystems. This includes the dynamics across both technology choices and between people.

In this final installment I’ll dive into how we get to “advanced class,” including the importance of domain knowledge and thinking about cumulative value add vs. solutions looking for problems that can be solved in other, albeit brute force, ways.

Getting to advanced class

I talked about the importance of compounding value in part one of this series and will touch on this concept a little more here, in addition to the criticality of domain knowledge. Speaking of the later, hundreds of IoT platform providers claim they can do a use case like Predictive Maintenance (PdM) but few actually have the domain knowledge to do it. A PdM solution not only requires analytics tools and data science but also an intimate knowledge of failure patterns based on various attributes (vibration, oil particulates, temperature, motor current, etc.). Often an operator on a factory floor that has “been there done that” sits alongside a data scientist to help program the analytics algorithms based on his/her tribal knowledge. A former colleague once worked on a PdM solution with a line operator named Brad, who was the expert that helped the data scientist understand what machine and process behaviors were non-issues, despite looking like bad news, and vice versa. They jokingly called the end result “Bradalytics”.

Further, there’s a certain naivety in the industry today in terms of pushing IoT solutions when there’s already established “good enough” precedent. In the case of PdM, manual data acquisition with a handheld vibration meter once a month or so is a common practice to accurately predict impending machine failures because these failures don’t typically happen overnight. An industrial operator can justify this manual data collection as OpEx and the only thing permanently installed on their machines are brass pads that indicate where to apply the contact-based handheld sensor, from which data is manually loaded into an analytical tool.

Similar manual procedures are all too common in other industries, such as USB data loggers used to monitor temperature in cold chain logistics operations. In another example, structural engineers have historically used the practice of attaching an expensive sensor package to monitor the structural integrity of a bridge or building for a few weeks, only to then move this equipment on to the next piece of infrastructure.

The promise of IoT is that this instrumentation is becoming so inexpensive that it can be deployed permanently everywhere for acquiring real-time data from the physical world; however, the value of deploying this infrastructure still must be greater than the cost of deploying and maintaining it through its full lifecycle in addition to the associated risk in terms of maintaining security and privacy.

Don’t get me wrong — PdM enabled by IoT is a valuable use case — despite machines typically failing over a long period of time it’s expensive to roll a truck to do repairs, especially if an industrial process experiences a loss of production. For example, downtime in a major jetliner factory can be upwards of $20k a minute! It’s just important to think about the big picture and whether you’re trying to solve a problem that has already been solved.

Looking at the bigger picture, a supply chain linked to a manufacturing process is a great example of an ecosystem that easily justifies an IoT solution for real-time monitoring and analytics. In a multi-stage manufacturing operation, the cost of discovering a flaw within a given part increases steadily with each process stage. The cost is even higher if that part gets into the supply chain and it’s higher yet if a defective product gets to a customer, not to mention impacting the manufacturer’s brand. Here the cumulative value of instrumenting throughout the product lifecycle is very high and definitely warrants a solution that can report issues the moment they occur.

Speaking of the holistic value that I touched on in part one and above, the real potential is not just the remote monitoring of a single machine, rather a whole fleet of interconnected machines. Imagine a world where you can send a tech out with work instructions to preventatively repair a fleet of machines in order to get the most out of a truck roll to a given location. This is similar to the story of “Fred and Phil” in part one, in which Phil wanted the propane tanks to be bone dry before rolling a truck. And on top of that — imagine that the algorithm could tell you that it will cost you less money in the long run to replace a specific machine altogether, rather than trying to repair it yet again.

It goes beyond IoT and AI, last I checked, systems ranging from machines to factory floors and vehicles are composed of subsystems from various suppliers. As such, open interoperability is also critical when it comes to Digital Twins. I think this is a great topic for another blog!

Sharpening the edge

In my recent Edge AI blog I highlighted the new LF Edge taxonomy white paper and how we think this taxonomy will help put an end to the current market confusion caused by various industries (cloud, telco, IT, OT/industrial, consumer) defining the edge with a strong bias towards their interests/PoV, not to mention taxonomies that use ambiguous terms such as “thin and thick” and “near and far” that mean different things to different people. The team at ZEDEDA had a heavy hand in shaping this with the LF Edge community. In fact, a lot of the core thinking stems from my “Getting a Deeper Edge on the Edge” blog from last year.

As highlighted in the paper, the IoT component of the Smart Device Edge (e.g., compared to client devices like smartphones, PCs, etc. that also live at this sub-category) spans a single node with 256MB of memory up to a small server cluster, deployed outside of a physically-secure data center or Modular Data Center (MDC). The upper end of this spectrum is increasingly blurring into the Kubernetes paradigm thanks to efforts like K3S. However, due to resource constraints and other factors these nodes will not have the exact same functionality as higher edge tiers leveraging full-blown Kubernetes.

Below the Smart Device Edge is the “Constrained Device Edge”. This sub-tier consists of a highly fragmented landscape of microcontroller-based devices that typically have their own custom OTA tools. Efforts like Microsoft’s Sphere OS are trying to address this space and it’s important to band together on efforts like this due to the immense fragmentation at this tier.

Ultimately it’s important to think of the edge to cloud as a continuum and that there isn’t an “industrial edge” vs. an “enterprise edge” and a “consumer edge” as some would contend. Rather, it’s about building an ecosystem of technology and services providers that create necessarily unique value-add on top of more consistent infrastructure while taking into account the necessary tradeoff across the continuum.

We build the guts so you can bring the glory

You can think of ZEDEDA’s SaaS-based orchestration solution as being like VMware Tanzu in principle (in the sense that we support both VMs and containers) but optimized for IoT-centric edge compute hardware and workloads at the “Smart Device Edge” as defined by the LF Edge taxonomy. We’re not trying to compete with great companies like VMware and Nutanix who shine at the On-prem Data Center Edge up through the Service Provider Edge and into centralized data centers in the cloud. In fact, we can help industry leaders like these, telcos and cloud service providers extend their offerings including managed services down into the lighter compute edges.

Our solution is based on the open source Project EVE within LF Edge which provides developers with maximum flexibility for evolving their ecosystem strategy with their choice of technologies and partners, regardless of whether they ultimately choose to take an open or more closed approach. EVE aims to do for IoT what Android did for the mobile component of the Smart Device Edge by simplifying the orchestration of IoT edge computing at scale, regardless of applications, hardware or cloud used.

The open EVE orchestration APIs also provide an insurance policy in terms of developers and end users not being locked into only our commercial cloud-based orchestrator. Meanwhile, it’s not easy to build this kind of controller for scale, and ours places special emphasis on ease of use in addition to security. It can be leveraged as-is by end users, or white-labeled by solution OEMs to orchestrate their own branded offers and related ecosystems. In all cases, the ZEDEDA cloud is not in the data path so all permutations of edge data sources flowing to any on-premises or cloud-based system are supported across a mix of providers.

As I like to say, we build the guts so our partners and customers can bring the glory!

In closing

I’ll close with a highlight of the classic Clayton Christiansen book Innovator’s Dilemma. In my role I talk with a lot of technology providers and end users, and I often see people stuck in this pattern, thinking “I’ll turn things around if I can just do what I’ve always done better”. This goes not just for large incumbents but also fairly young startups!

One interaction in particular that has stuck with me over the years was a IoT strategy discussion with a large credit card payments provider. They were clearly trapped in the Innovator’s Dilemma, wondering what to do about the mobile payment players like Square that had really disrupted their market. As they were talking about how they didn’t want another Square situation to happen again, I asked them “have you thought about when machines start making payments?”. The fact that this blew their minds is exactly why another Square is going to happen to them, if they don’t get outside of their own headspace.

When approaching digital transformation and ecosystem development it’s often best to start small, however equally important is to think holistically for the long term. This includes building on an open foundation that you can grow with and that enables you to focus your investments on business relationships and innovation rather than reinvention. Sure, you may be able to build your own orchestration solution like we have at ZEDEDA, or your own full turn-key stack, but if you do either in a proprietary silo then you’ve just locked yourself out of the biggest potential — creating a network effect across various markets through increasingly interconnected ecosystems! Again, imagine if the internet was owned by one company.

We invite you to get involved with Project EVE, and more broadly-speaking LF Edge as we jointly build out this foundation. You’re of course free to build your own controller with EVE’s open APIs, or reach out to us here at ZEDEDA and we can help you orchestrate your commercial IoT edge ecosystem. We’re here to help you start small and scale big and have a growing entourage of technology and service partners to drive outcomes for your business.

Interesting Developments In Edge Hypervisors

By Blog, Industry Article, Project EVE, State of the Edge

Written by Rex St. John, EGX Developer Relations at NVIDIA

This article originally ran on Rex’s LinkedIn page. For more content like this, connect with him on LinkedIn. 

After building Edge Computing ecosystems at Intel and Arm, I have recently made the switch to working on Edge Computing at NVIDIA. Several people have asked me to share my perspective and learnings, so I am starting this informal, personal series on the topic. All opinions shared here are my own personal opinions and not those of my employer.

Objective

In this article, I will share two reasons why some experts in the industry are investing in hypervisor technology as well as two interesting open source edge hypervisor solutions to be aware of. For edge definition nitpickers (you know who you are), I am going to be referring to the “Device Edge” here. There are many other definition for “Edge,” if you are curious, read this LF Edge white paper.

The Hovercraft Analogy

For those of you who are unfamiliar, a hypervisor is kind of like a hovercraft that your programs can sit inside. Like hovercrafts, hypervisors can provide protective cushions which allow your applications to smoothly transition from one device to another, shielding the occupants from the rugged nature of the terrain below. With hypervisors, the bumps in the terrain (differences between hardware devices), are minimized and mobility of the application is increased.

Benefits of Hypervisors

Benefits of hypervisors include security, portability and reduced need to perform cumbersome customization to run on specific hardware. Hypervisors also allow a device to concurrently run multiple, completely different, operating systems. Hypervisors also can help partition applications from one another for security and reliability purposes. You can read more about hypervisors here. They frequently are compared to, used together with or even compete with containers for similar use cases, though they historically require more processing overhead to run.

Two Reasons Why Some (Very Smart) Folks Are Choosing Hypervisors For The Edge

A core challenge in Edge Computing is the extreme diversity in hardware that applications are expected to run on. This, in turn, creates challenges in producing secure, maintainable, scalable applications capable of running across all possible targets.

Unlike their heavier datacenter-based predecessors, light-weight hypervisors offer both the benefits of traditional hypervisors while also respecting the limited resources found on the device edge. Here are two reasons why some in the industry are taking a careful look at edge hypervisors.

Reason 1: Avoiding The Complexity And Overhead of Kubernetes

One potential reason for taking a hypervisor-based approach at the edge is that there may be downsides in pursuing Kubernetes for smaller clusters. These include the difficulty in building and managing a team who can properly setup and scale a real-world Kubernetes application due to the overhead and complexity of Kubernetes itself. In some cases, such as in running a cluster of 4-5 nodes, it might be desirable to use more streamlined approaches involving a controller and light-weight hypervisors. This is the approach taken by EVE, mentioned in more detail below.

Reason 2: Ease Of Modernizing Brown-Field Industrial IT

Another pressing reason for choosing edge hypervisors is that “brown-field” installations of existing edge hardware are extremely expensive to upgrade to follow modern IT “best practices.” Hypervisors provide a path forward that does not involve rewriting old systems from scratch as the code running on older machines can frequently be shifted into a hypervisor and neatly managed and secured from there (a process referred to as “Workload Consolidation.”)

Let’s take a look at two interesting examples of edge hypervisors to understand further.

Hypervisor #1: Project ACRN

No alt text provided for this image

The first edge hypervisor we will look at is called ACRN, which is a project hosted by the Linux Foundation. ACRN has a well documented architecture and offers a wide range of capabilities and configurations depending on the situation or desired outcome.

No alt text provided for this image

ACRN seeks to support industrial use cases by offering a specific partitioning between high-reliability processes and those which do not need to receive preferential security and processing priority. ACRN accomplishes this separation by specifying a regime for sandboxing different hypervisor instances running on the device as shown above. I recommend keeping an eye on ACRN as it seems to have significant industry support. ACRN supported platforms currently tend to be strongly x86-based.

Hypervisor #2: EVE (part of LF Edge)

Also a project hosted on the Linux Foundation, EVE differs from ACRN in that it belongs to the LFEdge project cluster. Unlike ACRN, EVE also tends to be more agnostic about supported devices and architectures. Following the instructions hosted on the EVE Github page, I was able to build and run it on a Raspberry Pi 4 within the space of ten minutes, for example.

No alt text provided for this image

In terms of design philosophy, EVE is positioning itself as the “Android of edge devices.” You can learn more about EVE by watching this recent webinar featuring the Co-Founder of Zededa, Roman Shaposhnik. EVE makes use of a “Controller” structure which provisions and manages edge nodes to simplify the overhead of operating an edge cluster.

Wrapping Up

Expect to see more happening in the hypervisor space as the technology continues to evolve. Follow me to stay up to date with the latest developments in Edge Computing.

Cultivating Giants to Stand On: Extending Kubernetes to the Edge

By Blog, Project EVE

Written by Roman Shaposhnik, Project EVE lead and Co-Founder &VP of Products and Strategy at ZEDEDA

This content originally ran on the ZEDEDA Medium Blog – click here for more content like this.

Kubernetes is more than just a buzzword. With Gartner predicting that by the end of 2025, 90% of applications at the edge will be containerized, it’s clear that organizations will be looking to leverage Kubernetes across their enterprises, but this isn’t a straightforward proposition. There’s much more involved than just repurposing the architecture we use in the data center in a smaller or more rugged form factor at the edge.

The edge environment has several major distinctions from data centers that must be addressed in order to successfully leverage Kubernetes:

Computers, whether massive data center machines or small nodes on the smart device edge, are essentially three parts — hardware, operating system (OS) and runtime — running in support of some sort of application. And within that, an operating system is just a program that allows the execution of other programs. We went through a time in the 1990s where it was believed that the OS was the only part that mattered, and the goal was to find the best one, like the OS was a titan with the entire world on its shoulders. The reality though is that it’s actually turtles all the way down! By this I mean that with virtualization, computers are not limited to just three parts. We can slice each individual section in many different ways, with hardware emulation, hypervisors, etc.

Image for post

Split it or join it: either way you get something exciting

So how can we look at these building blocks in the most optimal way in 2020?

We first have to talk about where we find computers — the spectrum of computers being deployed today is vast. From giant machines in data centers doing big things all the way to specialized computers that might be a smart light bulb or sensor. In the middle is the proverbial edge, which we call the Smart Device Edge.

Image for post

Image courtesy of LF Edge

As we look at how to best run Kubernetes on the smart device edge, the answer is that it’s a triplet of K3s, some kind of operating system (or support for K3s) and some sort of hardware.

And so then, if we have K3s and we have hardware, what’s the best possible way to run K3s on hardware? The answer is a specialized operating system.

Just like the team behind Docker used a specialized operating system when they had to run Docker on a MacBook Pro — they created Docker Desktop, which is a specialized engine — like an operating system — that’s only there in support of Docker. And so for the smart device edge, we’ve created EVE, a lightweight, secure, open, and universal operating system built to address the unique security and scale requirements of edge nodes deployed outside of the data center.

Image for post
EVE is to Edge what Android is to Mobile

What makes EVE different? It’s the only OS that enables organizations to extend their cloud-like experience to edge deployments outside of the data center while also supporting legacy software investments. It provides an abstraction layer that decouples software from the diverse landscape of IoT edge hardware to make application development and deployment easier, secure and interoperable. The hosting of Project EVE under LF Edge ensures vendor-neutral governance and community-driven development.

Ready to learn more and to see EVE in action? Check out the full discussion.

You Can Now Run Windows 10 on a Raspberry Pi using Project EVE!

By Blog, Project EVE

Written by Aaron Williams, LF Edge Developer Advocate

Ever since Project EVE came under the Linux Foundation’s LF Edge umbrella, we have been asked about porting (and we wanted to port) EVE to the Raspberry Pi, so that developers and hobbyists could test out EVE’s virtualization of hardware.  Both were looking for an easy way to evaluate EVE by creating simple PoC projects, without having to buy a commercial grade IoT gateway or another device.  They wanted to just get started with something they already had on their desk.  And we are excited to announce that we have completed the first part of the work needed to run Windows on a Raspberry Pi 4!  We have posted the tutorial on our community wiki and it takes less than an hour to get it up and running.

The RPi

The Raspberry Pi was first released in 2012 with the goal of having a cheap and easy way to teach high school students how to code.  It had USB ports to attach a keyboard and mouse, HDMI to hook up to your TV, GPIO (General Purpose Input/Output) pins for IoT, and a networking cable for internet access.  Thus for $35 you had a great, cheap computer that ran Linux.  The Raspberry Pi Foundation sold a lot of these devices to schools, but the RPi really took off as developers and home hobbyists discovered them, thinking “Wow a $35 Linux computer, I wonder if I could do that home IoT project I have been planning?”  Plus, in many companies, the RPi became a great way to create “real” demos and PoCs cheaply.

Fast forward 7 years and we knew that we wanted to port EVE to the RPi, because it was such a large part of the IoT world, especially demos explaining IoT concepts.  (It is much easier to go to your manager or spouse and ask for $50 RPi vs. $500+ for some hardware.)  But, according to Erik Nordmark, TSC Chair of Project EVE, “the GIC (Global Interrupt Controller) and the proprietary RPI boot code on the RPi3 (and earlier models) prevented it from booting into a Type 1 Hypervisor like Xen without a hacking up strange emulation code.”  Thus, while it was possible, it would take a lot work and might not work well.

This changed with the release of the RP4.  Roman Shaposhnik, also of Project EVE, and Stefano Stabellini, of the Xen Project, saw that the RPi 4 had a regular GIC-400 interrupt controller that Xen supports right of out the box.  Thus, getting Xen to work on the RPi should be pretty easy, right?  And as they documented in their article from Linux.com, it wasn’t. “We were utterly oblivious that we were about to embark on an adventure deep in the belly of the Xen memory allocator and Linux address translation layers.”  But soon their hard work paid off and they were able to get Xen working and submitted a number of patches that will be part of the Linux 5.9 release.  (To learn about Roman and Stefano’s adventure, see their article Xen on Raspberry Pi 4 Adventures at Linux.com).  With that done, the EVE team turned their attention to see what work would be needed to complete the virtualization of the RPi, which went pretty smoothly.

Why Windows?

We have been saying since the start of Project EVE, that EVE is to IoT as Android is to phones.  Android allows you to write your code and push it to the device without having to worry about the hardware underneath.  EVE works much the same way.  It virtualizes the hardware, allowing you to push your code across devices.  And since EVE is open source, everyone benefits from the “plumbing” being handled by the community.  The plumbing in this case is the addition of a new device or family of devices. Thus, when a device is added, the whole community benefits.  IoT devices do have an extra security concern, namely they not usually housed in a locked building and instead are out in the open.  Therefore, the physical security of the device cannot be guaranteed and it must be assumed that there is no physical security.  Because of this, the default for EVE is turn off all external ports, such as USB ports.

This brings us to the question, why Windows?  Simply, why not?  Windows doesn’t belong on a Raspberry Pi, so we figured that it would be fun to see if it would work.  And it worked right out of the box, we just needed to find a containerized version of Windows and then we just deployed it (it is really that easy).  And it is a lot of fun using Windows knowing that it is running on a Raspberry Pi!

What work is left

We haven’t done a lot of testing and for us this is a PoC, so we won’t have a full list of limitations.  But here are a couple of things that we have found.  We will update our tutorial page as the community finds and fixes them.

Asking Petr Fedchenkov and Vladimir Suvorov, lead developers on Project EVE about issues that they have run into, Petr mentioned, “The biggest issue is that there are no drivers for GPU and Windows 10 ARM64 doesn’t have virtio-gpu support.  So, we are using ramfb (RAM Framebuffer), which is much more limited.”  In layman’s terms, this means that today if you plug a keyboard and monitor into the RPI, you will interact with EVE, not the Windows desktop.  The easy workaround is to run RDP (Windows Remote Desktop) or VNC, but in our testing RDP works much better.

The native WiFi does work , but you do need to turn it on via EVE.  Remember EVE is designed to give you control of your devices, remotely, yet securely so, the WiFi is turned off by default.  As you build EVE for your RPi, you have the option of passing in a SSID, which turns of the WiFi.

Bluetooth is currently not working.  USB ports should work, but there needs to be some configuration.  We are also not sure about the GPIO pins, we haven’t tested them yet.  (see below on how to help out to get these working and tested.

Call for Help

While it is pretty amazing what we have accomplished, but we need a lot of help.  If have any interest, in part of this, please let us know.  We need help with getting the USB’s fully working plus the items mentioned above.  Is there device that you would like to see EVE work on?  Please help us port it to that device.  We also could use tech writers, bloggers, or anyone that can help us improve our documentation and/or can help us get the word out about EVE.  If you are interested, please visit our GitHub, Wiki, or slack channel (#eve).

Scaling Ecosystems Through an Open Edge (Part Two)

By Blog, Industry Article, Project EVE, Trend

Why an Open, Trusted Edge is Key to Realizing the True Potential of Digital Transformation

By Jason Shepherd, LF Edge Governing Board member and VP of Ecosystem at ZEDEDA

This content originally ran on the ZEDEDA Medium Blog – click here for more content like this.

In  of this series, I walked through various approaches to ecosystems and highlighted how business value tends to find a natural equilibrium across stakeholders. In this second installment I’ll walk through the importance of an open edge for scaling ecosystems and realizing the true potential of digital transformation, as well as providing some tips on building ecosystems that I’ve picked up over the past years.

Enabling increasingly connected Intranets of Things

Imagine if the overall internet was built as a closed ecosystem, controlled by a small set of organizations, much less one. Of course, there are browsing restrictions placed at a company level and in some countries, but the internet simply wouldn’t have made the same massive impact on society without fundamental openness and interoperability.

All data is created at the edge, whether it be from user-centric or IoT devices. A few years ago, the number of devices on the internet surpassed the global population and the growth for IoT devices is expected to continue at an exponential rate. This will not only unlock new paths to value, but also .

However, as it turns out, the term “Internet of Things” is actually a bit of a misnomer. It’s really about a series of increasingly connected Intranets of Things. Starting with simple examples like the story of Fred and Phil from part one of this series, ecosystems will get increasingly larger and more interconnected as the value to do so exceeds the complexity and risk. So how do we carry this out at scale?

The importance of an open edge

Previously, I’ve outlined the  when deploying applications closer to devices in the physical world, outside the confines of a traditional data center. Edge and IoT solutions inherently require a diverse collection of ingredients and expertise to deploy and over the past five years the emerging market has attempted to address this fragmentation with a dizzying landscape of proprietary platforms — each with wildly different methods for data collection, security and management.

That said, having hundreds of closed, siloed platforms and associated ecosystems is definitely not a path to scale. As with the internet itself, it is important to have an open, consistent infrastructure foundation for IoT and edge computing, and from there companies can decide how open or closed they want to build their business ecosystems on top.

The diversity of the IoT edge makes it impractical to develop one catch-all standard to bind everything together. Open source software frameworks are an excellent way to bridge together various ingredients, unifying rather than reinventing standards, accommodating legacy installations, and enabling developers to focus on value creation.  is an example of a collaborative effort to build an open edge computing foundation in partnership with other open source and standards-oriented initiatives. The key to this being the base for a truly open edge ecosystem is the vendor-neutral governance offered by the Linux Foundation.

In this recent  I highlighted that the winners in the end will be the ones creating differentiated value through their domain knowledge, building necessarily unique software and hardware and offering great services — not those that are reinventing the middle over and over again. AI models for common tasks such as identifying a person’s demographic, detecting a license plate number or determining if an object is a water bottle or weapon will be commonplace, meanwhile there will always be room to differentiate with AI when specific industry context is in play.

Trust is essential

I’ve written in the past about the “” being selling or sharing data, resources and services across total strangers, all while maintaining privacy on your terms. This is the ultimate scale factor, but we need to turn to technology for help because it simply isn’t feasible to take people out to dinner fast enough one by one to build the necessary trust relationships.

Consumers are often comfortable pledging allegiance to specific brands and giving up a little privacy as long as they trust the provider and get value. However, in order to build more complex ecosystem relationships that span private and public boundaries at scale we not only need open interoperability but also ensure that no single entity owns the trust.

As such, we also need to collaborate on a technology foundation that automates the establishment of trust as data flows across heterogeneous systems. We’re seeing the industry increasingly step up here, from distributed ledger efforts like IOTA and Hyperledger that provide smart contracts to the  and the emerging  which aims to build out the concept of data confidence fabrics by layering trust insertion technologies with a system-based approach. Both the ToIP Foundation and Alvarium are focused on facilitating trust in both machine-generated data and across human relationships.

While these efforts don’t replace the need to build business relationships (and take people out to dinner!), they will provide necessary acceleration. Moreover, beyond helping us scale complex ecosystems these tools can also aid in combating the increasing issue with deepfakes and ensuring ethical AI solutions, in addition to accelerating workload consolidation at the edge and dealing with regulatory requirements like GDPR… but these are blog topics for another time!

Tips on building an ecosystem (hint, hint: domain knowledge rules)

When I started building the IoT ecosystem with the team at Dell back in 2015 it was clear that the market was going to go vertical before going horizontal. This is a typical pattern in any new market and proprietary offers often get the initial traction while solutions based on an open foundation always win in the end when it comes to sheer scale (recall the Apple — Android discussion from part one…).

So, our ecosystem strategy starting in 2015 was to go super broad before going deep, partnering widely and enabling the “cream to rise to the top”. I joked with the team back then that we were going to do “Tinder” and then “D-Harmony”. Sure enough, the partners that had the most initial traction were those that were laser-focused on one use case, meanwhile the horizontal “peanut-butter platforms” were stalling. Case in point, if you do a little bit of everything you rarely do one thing really well.

But it was really about these vertically-focused platforms’ domain knowledge that resonated with a specific customer need and desired outcome. They built their technology platforms to get to data but their real value was that domain knowledge. Over time as we get to a more consistent, open foundation for Edge and IoT solutions these providers will either need to pivot to being more like system integrators, or build differentiated software and/or hardware. So, in 2017 we purposely switched from going broad to being more deliberate in partnerships, focusing more on partners that have realized the importance of separating domain knowledge from the underlying technology foundation and helping with matchmaking across the partner landscape.. Enter “D-Harmony”!

In closing

Deploying IoT and edge computing solutions takes a village and it’s important to establish an entourage of partners that have a “go to dance move” rather than working with those that are trying to do too much and as a result not doing anything particularly well. Five years have passed and a lot of providers have felt the pain of trying to own everything and have since realized the importance of having focus and establishing meaningful partnerships. So, now I joke with the team at ZEDEDA as we build up our ecosystem of market-leading pure-play solutions and domain experts that we’re going straight to “Z-Harmony”!

Thanks for reading. In the third and final part of this series, I’ll provide more insights on how we get to “advanced class” and more on what we’re doing at ZEDEDA to help build the necessarily open foundation to facilitate ecosystem scale. In the meantime, feel free to drop me a line with any comments or questions.

Xen on Raspberry Pi 4 with Project EVE

By Blog, Project EVE

Written by Roman Shaposhnik, Project EVE lead and Co-Founder &VP of Products and Strategy at ZEDEDA, and Stefano Stabellini, Principal Engineer for Xilinx

Originally posted on Linux.com, the Xen Project is excited to share that the Xen Hypervisor now runs on Raspberry Pi. This is an exciting step for both hobbyists and industries. Read more to learn about how Xen now runs on RPi and how to get started.

Raspberry Pi (RPi) has been a key enabling device for the Arm community for years, given the low price and widespread adoption. According to the RPi Foundation, over 35 million have been sold, with 44% of these sold into industry. We have always been eager to get the Xen hypervisor running on it, but technical differences between RPi and other Arm platforms made it impractical for the longest time. Specifically, a non-standard interrupt controller without virtualization support.

Then the Raspberry Pi 4 came along, together with a regular GIC-400 interrupt controller that Xen supports out of the box. Finally, we could run Xen on an RPi device. Soon Roman Shaposhnik of Project EVE and a few other community members started asking about it on the xen-devel mailing list. “It should be easy,” we answered. “It might even work out of the box,” we wrote in our reply. We were utterly oblivious that we were about to embark on an adventure deep in the belly of the Xen memory allocator and Linux address translation layers.

The first hurdle was the availability of low memory addresses. RPi4 has devices that can only access the first 1GB of RAM. The amount of memory below 1GB in Dom0 was not enough. Julien Grall solved this problem with a simple one-line fix to increase the memory allocation below 1GB for Dom0 on RPi4. The patch is now present in Xen 4.14.

“This lower-than-1GB limitation is uncommon, but now that it is fixed, it is just going to work.” We were wrong again. The Xen subsystem in Linux uses virt_to_phys to convert virtual addresses to physical addresses, which works for most virtual addresses but not all. It turns out that the RPi4 Linux kernel would sometimes pass virtual addresses that cannot be translated to physical addresses using virt_to_phys, and doing so would result in serious errors. The fix was to use a different address translation function when appropriate. The patch is now present in Linux’s master branch.

We felt confident that we finally reached the end of the line. “Memory allocations – check. Memory translations — check. We are good to go!” No, not yet. It turns out that the most significant issue was yet to be discovered. The Linux kernel has always had the concept of physical addresses and DMA addresses, where DMA addresses are used to program devices and could be different from physical addresses. In practice, none of the x86, ARM, and ARM64 platforms where Xen could run had DMA addresses different from physical addresses. The Xen subsystem in Linux is exploiting the DMA/physical address duality for its own address translations. It uses it to convert physical addresses, as seen by the guest, to physical addresses, as seen by Xen.

To our surprise and astonishment, the Raspberry Pi 4 was the very first platform to have physical addresses different from DMA addresses, causing the Xen subsystem in Linux to break. It wasn’t easy to narrow down the issue. Once we understood the problem, a dozen patches later, we had full support for handling DMA/physical address conversions in Linux. The Linux patches are in master and will be available in Linux 5.9.

Solving the address translation issue was the end of our fun hacking adventure. With the Xen and Linux patches applied, Xen and Dom0 work flawlessly. Once Linux 5.9 is out, we will have Xen working on RPi4 out of the box.

We will show you how to run Xen on RPi4, the real Xen hacker way, and as part of a downstream distribution for a much easier end-user experience.

HACKING XEN ON RASPBERRY PI 4

If you intend to hack on Xen on ARM and would like to use the RPi4 to do it, here is what you need to do to get Xen up and running using UBoot and TFTP. I like to use TFTP because it makes it extremely fast to update any binary during development.  See this tutorial on how to set up and configure a TFTP server. You also need a UART connection to get early output from Xen and Linux; please refer to this article.

Use the rpi-imager to format an SD card with the regular default Raspberry Pi OS. Mount the first SD card partition and edit config.txt. Make sure to add the following:

 

    kernel=u-boot.bin

    enable_uart=1

    arm_64bit=1

Download a suitable UBoot binary for RPi4 (u-boot.bin) from any distro, for instance OpenSUSE. Download the JeOS image, then open it and save u-boot.bin:

    xz -d 
openSUSE-Tumbleweed-ARM-JeOS-raspberrypi4.aarch64.raw.xz

   kpartx -a 
./openSUSE-Tumbleweed-ARM-JeOS-raspberrypi4.aarch64.raw

    mount /dev/mapper/loop0p1 /mnt
    cp /mnt/u-boot.bin /tmp

Place u-boot.bin in the first SD card partition together with config.txt. Next time the system boots, you will get a UBoot prompt that allows you to load Xen, the Linux kernel for Dom0, the Dom0 rootfs, and the device tree from a TFTP server over the network. I automated the loading steps by placing a UBoot boot.scr script on the SD card:

    setenv serverip 192.168.0.1

    setenv ipaddr 192.168.0.2

    tftpb 0xC00000 boot2.scr

    source 0xC00000

Where:
– serverip is the IP of your TFTP server

– ipaddr is the IP of the RPi4

Use mkimage to generate boot.scr and place it next to config.txt and u-boot.bin:

   mkimage -T script -A arm64 -C none -a 0x2400000 -e 0x2400000 -d boot.source boot.scr

Where:

– boot.source is the input

– boot.scr is the output

UBoot will automatically execute the provided boot.scr, which sets up the network and fetches a second script (boot2.scr) from the TFTP server. boot2.scr should come with all the instructions to load Xen and the other required binaries. You can generate boot2.scr using ImageBuilder.

Make sure to use Xen 4.14 or later. The Linux kernel should be master (or 5.9 when it is out, 5.4-rc4 works.) The Linux ARM64 default config works fine as kernel config. Any 64-bit rootfs should work for Dom0. Use the device tree that comes with upstream Linux for RPi4 (arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb). RPi4 has two UARTs; the default is bcm2835-aux-uart at address 0x7e215040. It is specified as “serial1” in the device tree instead of serial0. You can tell Xen to use serial1 by specifying on the Xen command line:

  console=dtuart dtuart=serial1 sync_console

The Xen command line is provided by the boot2.scr script generated by ImageBuilder as “xen,xen-bootargs“. After editing boot2.source you can regenerate boot2.scr with mkimage:

 mkimage -A arm64 -T script -C none -a 0xC00000 -e 0xC00000 -d boot2.source boot2.scr

XEN ON RASPBERRY PI 4: AN EASY BUTTON

Getting your hands dirty by building and booting Xen on Raspberry Pi 4 from scratch can be not only deeply satisfying but can also give you a lot of insight into how everything fits together on ARM. Sometimes, however, you just want to get a quick taste for what it would feel to have Xen on this board. This is typically not a problem for Xen, since pretty much every Linux distribution provides Xen packages and having a fully functional Xen running on your system is a mere “apt” or “zypper” invocation away. However, given that Raspberry Pi 4 support is only a few months old, the integration work hasn’t been done yet. The only operating system with fully integrated and tested support for Xen on Raspberry Pi 4 is LF Edge’s Project EVE.

Project EVE is a secure-by-design operating system that supports running Edge Containers on compute devices deployed in the field. These devices can be IoT gateways, Industrial PCs, or general-purpose ruggedized computers. All applications running on EVE are represented as Edge Containers and are subject to container orchestration policies driven by k3s. Edge containers themselves can encapsulate Virtual Machines, Containers, or Unikernels.

You can find more about EVE on the project’s website at http://projecteve.dev and its GitHub repo https://github.com/lf-edge/eve/blob/master/docs/README.md. The latest instructions for creating a bootable media for Raspberry Pi 4 are also available at:

https://github.com/lf-edge/eve/blob/master/docs/README.md

Because EVE publishes fully baked downloadable binaries, using it to give Xen on Raspberry Pi 4 a try is as simple as:

$ docker pull lfedge/eve:5.9.0-rpi-xen-arm64 # you can pick a different 5.x.y release if you like

$ docker run lfedge/eve:5.9.0-rpi-xen-arm64 live > live.raw

This is followed by flashing the resulting live.raw binary onto an SD card using your favorite tool.

Once those steps are done, you can insert the card into your Raspberry Pi 4, connect the keyboard and the monitor and enjoy a minimalistic Linux distribution (based on Alpine Linux and Linuxkit) that is Project EVE running as Dom0 under Xen.

As far as Linux distributions go, EVE presents a somewhat novel design for an operating system, but at the same time, it is heavily inspired by ideas from Qubes OS, ChromeOS, Core OS, and Smart OS. If you want to take it beyond simple console tasks and explore how to run user domains on it, we recommend heading over to EVE’s sister project Eden: https://github.com/lf-edge/eden#raspberry-pi-4-support and following a short tutorial over there.

If anything goes wrong, you can always find an active community of EVE and Eden users on LF Edge’s Slack channels starting with #eve over at http://lfedge.slack.com/ — we’d love to hear your feedback.

In the meantime – happy hacking!

Pushing AI to the Edge (Part Two): Edge AI in Practice and What’s Next

By Blog, LF Edge, Project EVE, Trend

Q&A with Jason Shepherd, LF Edge Governing Board member, Project EVE leader and VP of Ecosystem at ZEDEDA

Image for post

This content originally ran on the ZEDEDA Medium Blog – visit their website for more content like this.

In Part One of this two-part Q&A series we highlighted the new LF Edge taxonomy that publishes next week and some key considerations for edge AI deployments. In this installment our questions turn to emerging use cases and key trends for the future.

To discuss more on this budding space, we sat down with our Vice President of ecosystem development, Jason Shepherd, to get his thoughts on the potential for AI at the edge, key considerations for broad adoption, examples of edge AI in practice and some trends for the future.

What do you see as the most promising use cases for edge AI?

As highlighted in Part One, the reasons for deploying AI at the edge include balancing needs across the vectors of scalability, latency, bandwidth, autonomy, security and privacy. In a perfect world all processing would be centralized, however this breaks down in practice and the need for AI (and ML) at the edge will only continue to grow with the explosion of devices and data.

Hands down, computer vision is the killer app for edge AI today due to the bandwidth associated with streaming video. The ability to apply AI to “see” events in the physical world enables immense opportunity for innovation in areas such as object recognition, safety and security, quality control, predictive maintenance and compliance monitoring.

Considering retail — computer vision solutions will usher in a new wave of personalized services in brick and mortar stores that provide associates with real-time insights on current customers in addition to better informing longer term marketing decisions. Due to privacy concerns, the initial focus will be primarily around assessing shopper demographics (e.g., age, gender) and location but increasingly we’ll see personalized shopping experiences based on individual identity with proper opt-in (often triggered through customer loyalty programs). This includes a trend for new “experiential” shopping centers, for which customers expect to give up some privacy when they walk in the door in exchange for a better experience.

While Amazon Go stores have led the trend for autonomous shopping environments, the use of computer-vision enabled self-service kiosks for grab-and-go checkout is growing rapidly overall. Given the recent health concerns with COVID-19, providers are rapidly shifting to making these solutions contactless by leveraging gesture control, instead of requiring interaction with a keypad or touch screen.

Computer vision use cases will often leverage sensor fusion, for example with barcode scans or radio-frequency identification (RFID) technology providing additional context for decision making in retail inventory management and point of sale (POS) systems. A camera can tell the difference between a T-shirt and a TV, but not the difference between large and medium sizes of the same shirt design. Still, perhaps eventually an AI model will be able to tell you if you have bad taste in clothing!

Another key vertical that will benefit from computer vision at the edge is healthcare. At ZEDEDA, we’ve worked with a global provider that leverages AI models in their medical imaging machines, located within hospitals and provided as a managed service. In this instance, the service provider doesn’t own the network on which their machines are deployed so they need a zero-trust security model in addition to the right tools to orchestrate their hardware and software updates.

Another example where bandwidth drives a need for deploying AI at the IoT Edge is vibration analysis as part of a use case like predictive maintenance. Here sampling rates of at least 1KHz are common, and can increase to 8–10KHz and beyond because these higher resolutions improve visibility into impending machine failures. This represents a significant amount of continuously streaming data that is cost-prohibitive to send directly to a centralized data center for analysis. Instead, inferencing models will be commonly deployed on compute hardware proximal to machines to analyze the vibration data in real time and only backhauling events highlighting an impending failure.

Analysis for predictive maintenance will also commonly leverage sensor fusion by combining this vibration data with measurements for temperature and power (voltage and current). Computer vision is also increasingly being used for this use case, for example the subtle wobble of a spinning motor shaft can be detected with a sufficient camera resolution, plus heat can be measured with thermal imaging sensors. Meanwhile, last I checked voltage and current can’t be checked with a camera!

An example of edge AI served up by the Service Provider Edge is for cellular vehicle-to-everything (C-V2X) use cases. While latency-critical workloads such as controlling steering and braking will always be run inside of a vehicle, service providers will leverage AI models deployed on compute proximal to small cells in a 5G network within public infrastructure to serve up infotainment, Augmented Reality for vehicle heads-up displays and coordinating traffic. For the latter, these AI models can warn two cars that they are approaching a potentially dangerous situation at an intersection and even alert nearby pedestrians via their smartphones. As we continue to collaborate on foundational frameworks that support interoperability it will open up possibilities to leverage more and more sensor fusion that bridges intelligence across different edge nodes to help drive even more informed decisions.

We’re also turning to processing at the edge to minimize data movement and preserve privacy. When AI inferencing models are shrunk for use in constrained connected products or healthcare wearables, we can train local inferencing models to redact PII before data is sent to centralized locations for deeper analysis.

Who are the different stakeholders involved in accelerating adoption of edge AI?

Edge AI requires the efforts of a number of different industry players to come together. We need hardware OEMs and silicon providers for processing; cloud scalers to provide tools and datasets; telcos to manage the connectivity piece; software vendors to help productize frameworks and AI models; domain expert system integrators to develop industry-specific models, and security providers to ensure the process is secure.

In addition to having the right stakeholders it’s about building an ecosystem based on common, open frameworks for interoperability with investment focused on the value add on top. Today there is a plethora of choices for platforms and AI tools sets which is confusing, but it’s more of the state of the market than necessity. A key point of efforts like LF Edge is to work in the open source community to build more open, interoperable, consistent and trusted infrastructure and application frameworks so developers and end users can focus on surrounding value add. Throughout the history of technology open interoperability has always won out over proprietary strategies when it comes to scale.

In the long run, the most successful digital transformation efforts will be led by organizations that have the best domain knowledge, algorithms, applications and services, not those that reinvent foundational plumbing. This is why open source software has become such a critical enabler across enterprises of all sizes — facilitating the creation of de-facto standards and minimizing “undifferentiated heavy lifting” through a shared technology investment. It also drives interoperability which is key for realizing maximum business potential in the long term through interconnecting ecosystems… but that’s another blog for the near future!

How do people differentiate with AI in the long term?

Over time, AI software frameworks will become more standardized as part of foundational infrastructure, and the algorithms, domain knowledge and services on top will be where developers continue to meaningfully differentiate. We’ll see AI models for common tasks — for example assessing the demographics of people in a room, detecting license plate numbers, recognizing common objects like people, trees, bicycles and water bottles — become commodities over time. Meanwhile, programming to specific industry contexts (e.g. a specific part geometry for manufacturing quality control) will be where value is continually added. Domain knowledge will always be one of the most important aspects of any provider’s offering.

What are some additional prerequisites for making edge AI viable at scale?

In addition to having the right ecosystem including domain experts that can pull solutions together, a key factor for edge AI success is having a consistent delivery or orchestration mechanism for both compute and AI tools. The reality is that to date many edge AI solutions have been lab experiments or limited field trials, not yet deployed and tested at scale. PoC, party of one, your table is ready!

Meanwhile, as organizations start to scale their solutions in the field they quickly realize the challenges. From our experience at ZEDEDA, we consistently see that manual deployment of edge computing using brute-force scripting and command-line interface (CLI) interaction becomes cost-prohibitive for customers at around 50 distributed nodes. In order to scale, enterprises need to build on an orchestration solution that takes into account the unique needs of the distributed IoT edge in terms of diversity, resource constraints and security, and helps admins, developers and data scientists alike keep tabs on their deployments in the field. This includes having visibility into any potential issues that could lead to inaccurate analyses or total failure. Further, it’s important that this foundation is based on an open model to maximize potential in the long run.

Where is edge AI headed?

To date, much of the exploration involving AI at the edge has been focused on inferencing models — deployed after these algorithms have been trained with the scalable compute of the cloud. (P.S. for those of you who enjoy a good sports reference, think of training vs. inference as analogous to coaching vs. playing).

Meanwhile, we’re starting to see training and even federated learning selectively moving to the Service Provider and User Edges. Federated learning is an evolving space that seeks to balance the benefits of decentralization for reasons of privacy, autonomy, data sovereignty and bandwidth savings, while centralizing results from distributed data zones to eliminate regional bias.

The industry is also increasingly developing purpose-built silicon that can increase efficiencies amid power and thermal constraints in small devices and even support either training or inference and this corresponds with the shift towards pushing more and more AI workloads onto edge devices. Because of this, it’s important to leverage device and application orchestration tools that are completely agnostic to silicon, compared to offers from silicon makers that have a vested interest in locking you into their ecosystem.

Finally, we’ll see the lower boundary for edge AI increasingly extend into the Constrained Device Edge with the rise of “Tiny ML” — the practice of deploying small inferencing models optimized for highly constrained, microcontroller-based devices. An example of this is the “Hey Alexa” of an Amazon Echo that is recognized locally and subsequently opens the pipe to the cloud-based servers for a session. These Tiny ML algorithms will increasingly be used for localized analysis of simple voice and gesture commands, common sounds such as a gunshot or a baby crying, assessing location and orientation, environmental conditions, vital signs, and so forth.

To manage all of this complexity at scale, we’ll lean heavily on industry standardization, which will help us focus on value on top of common building blocks. Open source AI interoperability projects, such as ONNX, show great promise in helping the industry coalesce around a format so that others can focus on developing and moving models across frameworks and from cloud to edge. The Linux Foundation’s Trust over IP effort and emerging Project Alvarium will also help ease the process of transporting trusted data from devices to applications. This notion of pervasive data trust will lead to what I call the “Holy Grail of Digital” — selling and/or sharing data resources and services to/with people you don’t even know. Now this is scale!

In Closing

As the edge AI space develops, it’s important to avoid being locked into a particular tool set, instead opting to build a future-proofed infrastructure that accommodates a rapidly changing technology landscape and that can scale as you interconnect your business with other ecosystems. Here at ZEDEDA, our mission is to provide enterprises with an optimal solution for deploying workloads at the IoT Edge where traditional data center solutions aren’t applicable, and we’re doing it based on an open, vendor-neutral model that provides freedom of choice for hardware, AI framework, apps and clouds. We’re even integrating with major cloud platforms such as Microsoft Azure to augment their data services.

Reach out if you’re interested in learning more about how ZEDEDA’s orchestration solution can help you deploy AI at the IoT Edge today while keeping your options open for the future. We also welcome you to join us in contributing to Project EVE within LF Edge which is the open source foundation for our commercial cloud offering. The goal of the EVE community is to build the “Android of the IoT Edge” that can serve as a universal abstraction layer for IoT Edge computing — the only foundation you need to securely deploy any workload on distributed compute resources. To this end, a key next step for Project EVE is to extend Kubernetes to the IoT Edge, while taking into account the unique needs of compute resources deployed outside of secure data centers.

The success of AI overall — and especially edge AI — will require our concerted collaboration and alignment to move the industry forward while protecting us from potential misuse along the way. The future of technology is about open collaboration on undifferentiated plumbing so we can focus on value and build increasingly interconnected ecosystems that drive new outcomes and revenue streams. As one political figure famously said — “it takes a village!”

If you have questions or would like to chat with leaders in the project, join us on the LF Edge Slack  (#eve or #eve-help) or subscribe to the email list. You can check out the documentation here.

On the “Edge” of Something Great

By Akraino, Announcement, Baetyl, Blog, EdgeX Foundry, Fledge, Home Edge, LF Edge, Open Horizon, Project EVE, Secure Device Onboard, State of the Edge

As we kick off Open Networking and Edge Summit today, we are celebrating the edge by sharing the results of our first-ever LF Edge Member Survey and insight into what our focuses are next year.

LF Edge, which will celebrate its 2nd birthday in January 2021, sent the survey to our more than 75 member companies and liaisons. The survey featured about 15 questions that collected details about open source and edge computing, how members of the LF Edge community are using edge computing and what project resources are most valuable. 

Why did you chose to participate in LF Edge?

The Results Are In

The Top 3 reasons to participate in LF Edge are market creation and adoption acceleration, collaboration with peers and industry influence. 

  • More than 71% joined LF Edge for market creation and adoption acceleration
  • More than 57% indicated they joined LF Edge for business development
  • More than 62% have either deployed products or services based on LF Edge Projects or they are planned by for later this year, next year or within the next 3-5 years

Have you deployed products or services based on LF Edge Projects?

This feedback corresponds with what we’re seeing in some of the LF Edge projects. For example, our Stage 3 Projects Akraino and EdgeX Foundry are already being deployed. Earlier this summer, Akraino launched its Release 3 (R3) that delivers a fully functional open source edge stack that enables a diversity of edge platforms across the globe. With R3, Akraino brings deployments and PoCs from a swath of global organizations including Aarna Networks, China Mobile, Equinix, Futurewei, Huawei, Intel, Juniper, Nokia, NVIDIA, Tencent, WeBank, WiPro, and more. 

Additionally, EdgeX Foundry has hit more than 7 million container downloads last month and a global ecosystem of complementary products and services that continues to increase. As a result, EdgeX Foundry is seeing more end-user case studies from big companies like Accenture, ThunderSoft and Jiangxing Intelligence

Have you gained insight into end user requirements through open collaboration?


Collaboration with peers

The edge today is a solution-specific story. Equipment and architectures are purpose-built for specific use cases, such as 5G and network function virtualization, next-generation CDNs and cloud, and streaming games. Which is why collaboration is key and more than 70% of respondents said they joined LF Edge to collaborate with peers. Here are a few activities at ONES that showcase the cross-project and members collaboration. 

Additionally, LF Edge created a LF Edge Vertical Solutions Group that is working to enable easily-customized deployments based on market/vertical requirements. In fact, we are hosting an LF Edge End User Community Event on October 1 that provides a platform for discussing the utilization of LF Edge Projects in real-world applications. The goal of these sessions is to educate the LF Edge community (both new and existing) to make sure we appropriately tailor the output of our project collaborations to meet end user needs. Learn more.

Industry Influence

More than 85% of members indicated they have gained insights into end user requirements through open collaboration. A common definition of the edge is gaining momentum. Community efforts such as LF Edge and State of the Edge’s assets, the Open Glossary of Edge Computing, and the Edge Computing Landscape are providing cohesion and unifying the industry. In fact,  LF Edge members in all nine of the projects collaborated to create an industry roadmap that is being supported by global tech giants and start-ups alike.

 

 

Where do we go from here? 

When asked, LF Edge members didn’t hold back. They want more. They want to see more of everything – cross-project collaboration, end user events and communication, use cases, open source collaboration with other liaisons. As we head into 2021, LF Edge will continue to lay the groundwork for markets like cloud native, 5G, and edge for  more open deployments and collaboration.  

 

Introduction to Project EVE, an open-source edge node architecture

By Blog, Project EVE
Written by Brad Corrion, active member of the Project EVE community and Strategic Architect at Intel 
No alt text provided for this image

This article originally ran as a LinkedIn article last month. 

At the inaugural August ZEDEDA Transform conference, I participated in a panel discussion entitled “Edge OSS Landscape, Intro to Project EVE and Bridging to Kubernetes”. Project EVE, an LF Edge project, delivers an open-source edge node, on which applications are deployed as either containers or VMs. As some audience members noted, they were pleasantly surprised that we didn’t spend an hour talking explicitly about the Project EVE architecture. Instead, we considered many aspects of a container-first edge, community open-source investments, and whether technologies like Kubernetes can be useful for IoT applications.

The Edge Buzz

There’s a lot of buzz around the edge, and many see it as the next big thing since the adoption of cloud computing. Just as the working definition of the cloud has morphed over time and conceptualized as a highly scalable, micro-service-based application hosted on computing platforms around the globe, so has the edge come to represent a computing style. To embrace the edge means to place your computing power as close to the data that it is processing, to balance the cost and latency of moving high volumes of data across the network. Most have also used the transition to edge computing to adopt the same “cloud-native” technologies and processes to manage their compute workloads in a similar fashion, regardless of where the compute is deployed, be it cloud, data center, or some remote edge environment.

Turning the Problem on its Head

That last part is what enthuses me about the shift to edge computing: we move away from highly curated enterprise IT devices (whose management tends to prevent change and modification) and move towards cloud-like, dynamic, scalable assets (whose management technologies are designed for innovation and responding to ever-changing circumstances). This is the classic example of “pets vs. cattle“, but sprinkled in with IoT challenges like systems that are distributed at extreme ends of the world, with costly trip charges required to manage with an on-site technician. The solution turns the problem on its it head. It requires organizations, from IT to line of business, to adopt practices of agility and innovation, so that they can manage and deploy solutions to the edge as nimbly as they experiment and innovate in the cloud.

Next up, we discussed the benefits of utilizing open source software for projects like Project EVE. Making an edge node easy to deploy, secure to boot, and remotely manageable is not trivial, and it is not worth competing over. The community, including competitors, can create a solid, open-source edge node architecture, such as Project EVE, and all parties can benefit from the group investment. With a well-accepted, common edge architecture, innovators can focus instead on the applications, the orchestration, and the usability of the edge. Even using the same basic edge node architecture, there is more than ample surface area left to compete for service value, elegance, and solution innovation. Just as the typical investment around Kubernetes is allowing masses of projects and companies to improve on nearly every aspect of orchestration, without making each project re-invent the basics of orchestration, we don’t need tens of companies re-inventing secure device deployments, secure boot, and container protection. Get the basics done (the “boring stuff” as I called it) and focus on the specialization around it.

Can container first concepts, and projects like Kubernetes, be effective at the edge and solving IoT problems? Yes. No doubt, there are differences in using Kubernetes in the cloud from using it to manage edge nodes, and some of those challenges include limited power infrastructure, communications and connectivity, and cost considerations. However, these technologies are very adaptable. A container will run as happily on an IOT device, an IoT gateway, or an edge server. How you connect to and orchestrate the containers on those devices will vary. Most edges won’t need a local Kubernetes cluster. Distant Kubernetes infrastructures could remotely orchestrate a local edge node or local edge cluster.

Infrastructure as Code

A common theme in edge orchestration architectures is local orchestration, which provides enough power to a small orchestrator running at the edge to make decisions while offline from a central orchestrator. Projects like Open Horizon, which IBM recently open-sourced to the LF Edge, is designed to bridge traditional cloud orchestration to edge devices with a novel distributed policy engine. This distributed policy engine is executing and responding to changing conditions even when disconnected. Adopting an “infrastructure as code” mentality provides administrators a high degree of configuration and control of the target environment, even over remote networks. There is high confidence in the resulting infrastructure configurations, but with the variability as to “when” the changes are received due to bandwidth considerations. Could this be used on oil and gas infrastructure in the jungles of Ecuador? Yes. However, the challenge is in deciding which of the philosophies and projects are best suited to the needs of the situation.

If you find yourself architecting your edge node or using a simple Linux installation, while kicking the security can down the road, or are otherwise bogged down by remote manageability challenges when you’d rather be innovating and solving domain-specific problems, look to the open-source community. Specifically to projects like Project EVE, to give you a leg up for your edge architecture.

Please feel free to connect with me and start a conversation about this article. Here are some additional resources:

  • Explore how Intel is connecting open source communities with the retail market through the Open Retail Initiative
  • Be sure to explore Project EVE, Open Horizon, EdgeX Foundry and more projects at the LF Edge