Skip to main content
Category

eKuiper

LF Edge Releases Industry-Defining Edge Computing White Paper to Accelerate Edge/ IoT Deployments

By Akraino, Announcement, Baetyl, EdgeX Foundry, eKuiper, Fledge, Home Edge, LF Edge, Open Horizon, Project EVE, Secure Device Onboard, State of the Edge

Collaborative community white paper refines the definitions and nuances of open source edge computing across telecom, industrial, cloud, enterprise and consumer markets

 SAN FRANCISCO – June 24, 2022 –  LF Edge, an umbrella organization under the Linux Foundation that aims to establish an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system, today announced continued ecosystem collaboration via a new collaborative white paper, “Sharpening the Edge II: Diving Deeper into the LF Edge Taxonomy & Projects.” 

A follow-up to the LF Edge community’s original, collaborative 2020 paper which provides an overview of the organization and details the LF Edge taxonomy, high level considerations for developing edge solutions and key use cases,the new publication dives deeper into key areas of edge manageability, security, connectivity and analytics, and highlights how each project addresses these areas. The paper demonstrates maturation of the edge ecosystem and how the rapidly growing LF Edge community has made great progress over the past two years towards building an open, modular framework for edge computing. As with the first publication, the paper addresses  a balance of interests spanning the cloud, telco, IT, OT, IoT, mobile, and consumer markets.  

“With the growing edge computing infrastructure market set to be worth up to $800B by 2028, our LF Edge project communities are evolving,” said Jason Shepherd, VP Ecosystem, ZEDEDA  and former LF Edge Governing Board Chair. “This paper outlines industry direction through an LF Edge community lens. With such a diverse set of knowledgeable stakeholders, the report is an accurate reflection of a unified approach to defining open edge computing.” 

“I’m eager to continue to champion and spearhead the great work of the LF Edge community as the new board chair,” said Tina Tsou, new Governing Board chair, LF Edge.  “The Taxonomy white paper that demonstrates the accelerated community momentum seen by open source edge communities is really exciting and speaks to the power of open source.” 

The white paper, which is now available for download,  was put together as the result of broad community collaboration, spanning insights and expertise from subject matter experts across LF Edge project communities: Akraino, EdgeX Foundry, EVE, Fledge, Open Horizon, State of the Edge, Alvarium, Baetyl, eKuiper, and FIDO Device Onboard. 

ONE Summit North America 2022

Join the broader open source ecosystem spanning Networking, Edge, Access, Cloud and Core at ONE Summit North America, November 15-16 in Seattle, Wash. ONE Summit is the one industry event focused on best practices, technical challenges, and business opportunities facing decision makers across integrated verticals such as 5G, Cloud, Telco, and Enterprise Networking, as well as Edge, Access, IoT, and Core. The Call for Proposals is now open through July 8, 2022. Sponsorship opportunities are also available. 

 

About The Linux Foundation 

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation’s projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, and more.  The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

 

###

eKuiper Issues 1.4 Release & Discusses 2022 Roadmap

By Blog, eKuiper

Written by Jiyong Huang, Chair of the eKuiper Technical Steering Committee and Senior Software Engineer of EMQ

eKuiper, a Stage 1 (at-large) project under the LF Edge umbrella, is a Go language implementation of the lightweight stream processing engine which can run in various IoT edge usage scenarios for real-time data analysis. Being able to run near the data source at the edge, eKuiper can improve system response speed and security, saving network bandwidth and storage costs.

The eKuiper project usually has one release per quarter with new features and several patch versions afterwards. The current active release is v1.4 which is the biggest update after joining LF Edge! We have solid community contributions for thee release, including requirement collections, code contributions, testing and trials. So we are grateful to share the updates and hope they can benefit our users.

Rule Pipeline: Building Complex Businesses Logic with Flexibility

eKuiper uses SQL to define business logic, which lowers the threshold of development. Simple SQL statements can efficiently define business requirements such as filtering, aggregation, and data conversion in practical usage scenarios. However, for some complex scenarios, it is difficult to address by defining a single SQL statement; even if you can, the SQL statement itself is too complex and difficult to maintain.

Based on the new in-memory source and sink, the rule pipeline can connect multiple SQL rules easily, efficiently, and flexibly. The readability and maintainability of SQL statements can be improved when implementing complex business scenarios. The rules are connected with in-memory topics similar to the MQTT topics and support wildcard subscriptions, enabling an exceptionally flexible and efficient rules pipeline. While improving business expressiveness, rule pipelining can also improve runtime performance in certain complex scenarios. For example, multiple rules need to process data that has been filtered by a certain condition. By extracting that filtering condition as a predecessor rule, you can make the filtering calculated only once, significantly reducing the computation in the case of many rules.

Portable Plugin: Making Extensions Easier

The original version of eKuiper supported an extension scheme based on the Go native plug-in system, supporting individual extensions to source, sink and function (UDF). However, due to the limitations of the Go plugin system, writing and using plugins is not easy for users familiar with Go, let alone users of other languages. eKuiper has received a lot of feedback from users in the community about plugin development, operation and deployment, and various operational issues.

To balance development efficiency and runtime efficiency, v1.4.0 adds a new Portable plugin system to lower the threshold of plugin development. The new Portable plug-in is based on the nng protocol for inter-process communication and supports multiple languages, currently providing go and python SDKs, with more SDKs to be added in subsequent versions according to user requirements; simplifies the compilation/deployment process, and runs like a normal program written in various languages without additional restrictions. Due to the different operation mechanisms, portable plugin crashes will not affect eKuiper itself.

Native plug-ins and portable plug-ins can coexist. Users can choose the plug-in implementation or mix them according to their needs.

Shared Connections: Source/Sink Multiplexed Connections

eKuiper provides a rich set of sources and sinks to access and send results to external systems. Many of these sources and sinks are input/output pairs of the same external system type. For example, MQTT and EdgeX both have corresponding source and sink, and in the new version, users can do all connection-related configurations in the connection.yaml file; in the source/sink configuration, you can specify which connections to use without repeating the configuration. Shared connection instances reduce the additional consumption of multiple connections. In some scenarios, users may be limited in the number of connections and ports to external systems, and using shared connections can meet this limit. Also, based on shared connections, eKuiper can support connections to the EdgeX secure data bus.

Other enhancements

  • Support for configuration via environment variables
  • By default, eKuiper uses SQLite, an embedded database, to store metadata such as streams and rules, allowing for no external dependencies at runtime. In the new version, users can choose Redis as the metadata storage solution
  • Rule status returns error reason
  • Optimized SQL Runtime, reduce CPU usage up to 70% in a shared source user scenario
  • sink dynamic parameter support, e.g., MQTT sink can set the topic to a field value in the result so that the data received can be sent to a dynamic topic
  • Authentication support: user-configurable JWT-based authentication for REST API

2022 Roadmap

As a young project, eKuiper is still far from perfect. There are a long list of feature requests from the users and community. eKuiper will continue to grow in the new year and open to anyone who is interested to make edge computing powerful and easier to use at the same time. The team will focus on the four themes:

  • Enrich API and syntax
    • more SQL clause (in, between, like etc.)
    • streaming feature (dynamic table etc.)
    • more handy functions (changed, unnest, statistic functions)
  • Stability & Performance
    • Allow to select feature by build tag and provide a minimal core version
    • Incremental window
    • Continuously memory footprint and CPU optimization
  • Extension
    • Stabilize portable plugin runtime and support more languages
    • More http source type
  • Build & Deployment
    • Cluster and HA
    • Android support

Please check the 2022 Roadmap in our Github project for details, and stay tuned in with the eKuiper community.