June 2023

eKuiper Issues 1.10 Release Empowering Edge Computing with Advanced Analytics and Enhanced Features

By LF EdgeBlog, eKuiper, Project Release

eKuiper—a lightweight IoT data analytics and streaming software—is now available in its 1.10 release. eKuiper, an LF Edge project, migrates real-time cloud streaming analytics frameworks such as Apache Spark, Apache Storm and Apache Flink to the edge. eKuiper references these cloud streaming frameworks, incorporates any special requirements of edge analytics and introduces rule engine, which is based on Source, SQL (business logic) and Sink; rule engine is used for developing streaming applications at the edge.

We are thrilled to announce the highly anticipated release of eKuiper version 1.10.0, marking a significant milestone as our first double-digit release. This remarkable release boasts an extensive array of features and a growing community of contributors, for whom we express our deepest gratitude.

In this release, our top priority was enhancing the product’s core dependencies. Notably, we upgraded the Go language version to 1.20, ensuring optimal performance and compatibility. Additionally, eKuiper now extends support for the latest major version of Minnesota (v3) from EdgeX, further expanding its interoperability capabilities. While driving advancements in expressiveness, connectivity, and usability, we diligently focused on maintaining a delicate balance between feature richness and the ability to cater to edge deployments. Our commitment to delivering a streamlined experience for edge computing remains unwavering.

The 1.10 release introduces several significant new features, including:

Rule scheduling:
- Empower the edge to autonomously manage rule execution, allowing for enhanced flexibility and efficiency in edge computing environments.
EdgeX Minnesota support:
- Seamlessly integrate with the latest major version of EdgeX.
Enhanced file sink:
- Expand the capabilities of the file sink by supporting a wider range of file types, introducing rolling and compression features for efficient data storage and management.
Kafka sink:
- Introduce a new sink to write rule results directly to Kafka, enabling seamless integration with Kafka-based data pipelines and stream processing workflows.
SQL source/sink enhancement:
- Enhance the SQL source/sink functionality by introducing support for the new ClickHouse driver. Additionally, enable the configuration of maximum connections to optimize performance in scenarios with high connection volumes.
Batch strategy for sink:
- Introduce common properties for sinks to configure batch strategies, enabling efficient data handling and reducing I/O operations as needed.
Complex data extraction for sink:
- Simplify data transformation and processing by introducing common properties for sinks to extract specific data fields and optimize column selection, resulting in improved data processing performance.
Array type payload support:
- Enhance the source to support array type payloads, allowing for seamless handling and processing of complex data structures.
Unnest function:
- Introduce the unnest function, enabling the conversion of array types into multiple lines of data, facilitating more granular and detailed data processing.
Array and object functions:
- Expand the array and object functions library with over 10 additional functions, providing extensive support for complex data processing scenarios.
Dot notation for nested field access:
- Introduce dot notation to access nested fields within data structures, simplifying data retrieval and manipulation within complex nested hierarchies.
External state access in Redis:
- Enable the ability to read external state information from Redis, facilitating efficient data retrieval and integration with external applications.
SQL syntax enhancements:
- Enhance the SQL syntax by introducing support for expressions in array indexing, enabling more flexible and dynamic array operations within SQL queries.
Rule-related functions:
- Introduce additional functions to retrieve the current rule ID and enable pipeline delay, providing greater control and flexibility in rule execution and pipeline management.
Graph API for lookup table:
- Enabling efficient lookup table and lookup join in Graph API.

With this release, we reaffirm our dedication to empowering organizations with a cutting-edge, reliable, and user-friendly solution for harnessing the full potential of edge analytics and stream processing. Learn more about these and other features of eKuiper’s 1.10 release in the release notes.

What’s Next

In our upcoming version, we are thrilled to prioritize further advancements in the streaming and SQL runtime of eKuiper. Our focus will revolve around enriching the window functionality, particularly empowering the event window to cater to a wider range of scenarios. To ensure seamless integration and compatibility, we are committed to adding and refining the SQL syntax to align with standard SQL specifications. This will provide users with a familiar and intuitive experience when working with eKuiper, while enabling greater flexibility and ease of use.

Stay tuned for more updates and join us on this exciting journey as we continue to shape the future of edge computing with eKuiper.

June 16, 2023

Key New Usability and Security Features in Major EdgeX V3 Release Help It Stand Out as the Leading Open Source Edge Data Platform

By James ButcherBlog, EdgeX Foundry, Project Release

I am delighted to announce the release of the latest and greatest version of EdgeX Foundry the open source edge data platform. Version 3.0 is our most significant upgrade to date and I firmly believe will bring much value to edge solution developers in a variety of industrial sectors.

As the EdgeX TSC chair and product manager of IOTech’s commercial EdgeX-based solutions, I’d like to explain what EdgeX V3 brings and describe how the platform continues to be the key framework upon which companies rely when developing their commercial edge software solutions.

A jump of major version is significant for EdgeX, not only because it indicates a new set of key features for its user base, but also because it shows the mature, yet evolutionary outlook of the base platform and the associated commercial edge products that are available in the ecosystem.

EdgeX Reminder

EdgeX Foundry is an open source edge data platform, managed and hosted under the Linux Foundation Edge umbrella. More specifically, EdgeX is a highly flexible and scalable architecture that allows easy interoperability between edge devices and applications. It is made possible thanks to an ecosystem of community collaborators (including Intel, IOTech, Canonical, Eaton, Beechwoods and many others) representing a variety of industries.

Minnesota

To help in its identification, each release of EdgeX is provided a codename based on a geographical location. The previous version was named Levski (located in Bulgaria). The next stopping point on our virtual world tour is Minnesota! Aptly named, in part for being a key industrial center of the United States, EdgeX Minnesota helps to further serve the needs of OT/IT integration at the edge.

The major jump in version number lets us make more significant updates to the framework and allows us the freedom to make changes that are otherwise prohibited by our minor version compatibility rules. Of course, we try to be backwards compatible where possible, but after two years of V2 minor releases, there are some key updates and usability benefits we can now bring in…

Easier, Common Configuration

The main theme of Minnesota is usability. In particular, the way the user configures EdgeX has been much simplified and improved. Version 3.0 of EdgeX introduces a common configuration pattern for the EdgeX microservices. Since many config options (logging, telemetry, security, database and message bus settings) apply across multiple microservices, we’ve added the ability to configure these in a single common location. Some microservices are likely to have common config requirements, e.g., the southbound device services or northbound application services, so we’ve provided a layered design where users can configure settings at their chosen level.

This will save developers and deployers of EdgeX systems a lot of time and reduce maintenance effort. It will be much easier to make config changes, rerun the system and be confident that the settings have all taken effect appropriately. Take a look at the latest EdgeX docs for the details.

Reduced Config Formats

The jump in major version has also allowed us to reduce the number of different config file formats in EdgeX. Removing TOML and using only YAML and JSON going forward helps reduce what a new user has to learn. YAML and JSON can be derived from each other, so we let users pick their favorite and work with that. Again, usage of the platform becomes easier and more maintainable.

Service Authentication

Another big feature for 3.0 is the addition of microservice authentication within EdgeX. When run in secure mode, the microservices now require a validated authentication token before they are allowed to communicate. Thanks to the efforts of the EdgeX Security Working Group, this is all built into the platform with tokens issued by the Secret Store service. The security advancements here also allow version 3 to switch to a much lighter-weight API Gateway service than was used previously, which helps a lot with our ongoing effort to minimize the footprint of the platform.

Other V3 Updates

Of course, we also have V3 versions of the EdgeX APIs. We do have some minor changes there including updates to help with the automatic discovery of devices, but the APIs are mature so there aren’t the major updates we had with the previous major jump. You can check out the full set of changes in the EdgeX V3 migration guide. Full release notes are here.

Using EdgeX V3 and Joining in

We’re expecting that the next version after this (3.1 or Napa) planned for Q4 this year, will become the new Long Term Support (LTS) version of EdgeX. Those users wishing to jump from the previous LTS (2.1 or Jakarta) should start to become familiar with the new features. There are some breaking changes, so we encourage you to start to get to know V3 sooner rather than later. The features put into EdgeX Minnesota will serve as the basis for other features for the next few years. For example, using URLs to point to configuration and profiles that could come from anywhere and be shared is only possible because of the common configuration feature added now.

So please do go ahead and download EdgeX V3. A good place to get going is the Getting Started Guide and the lively EdgeX discussions board is a great place to chat and leave feedback.

June 5, 2023