Skip to main content
search
Category

Project Release

eKuiper Issues 1.10 Release Empowering Edge Computing with Advanced Analytics and Enhanced Features

By Blog, eKuiper, Project Release

eKuiper—a lightweight IoT data analytics and streaming software—is now available in its 1.10 release. eKuiper, an LF Edge project, migrates real-time cloud streaming analytics frameworks such as Apache Spark, Apache Storm and Apache Flink to the edge. eKuiper references these cloud streaming frameworks, incorporates any special requirements of edge analytics and introduces rule engine, which is based on Source, SQL (business logic) and Sink; rule engine is used for developing streaming applications at the edge.

We are thrilled to announce the highly anticipated release of eKuiper version 1.10.0, marking a significant milestone as our first double-digit release. This remarkable release boasts an extensive array of features and a growing community of contributors, for whom we express our deepest gratitude. 

In this release, our top priority was enhancing the product’s core dependencies. Notably, we upgraded the Go language version to 1.20, ensuring optimal performance and compatibility. Additionally, eKuiper now extends support for the latest major version of Minnesota (v3) from EdgeX, further expanding its interoperability capabilities. While driving advancements in expressiveness, connectivity, and usability, we diligently focused on maintaining a delicate balance between feature richness and the ability to cater to edge deployments. Our commitment to delivering a streamlined experience for edge computing remains unwavering.

The 1.10 release introduces several significant new features, including:

  • Rule scheduling:
    • Empower the edge to autonomously manage rule execution, allowing for enhanced flexibility and efficiency in edge computing environments.
  • EdgeX Minnesota support:
    • Seamlessly integrate with the latest major version of EdgeX.
  • Enhanced file sink:
    • Expand the capabilities of the file sink by supporting a wider range of file types, introducing rolling and compression features for efficient data storage and management.
  • Kafka sink:
    • Introduce a new sink to write rule results directly to Kafka, enabling seamless integration with Kafka-based data pipelines and stream processing workflows.
  • SQL source/sink enhancement:
    • Enhance the SQL source/sink functionality by introducing support for the new ClickHouse driver. Additionally, enable the configuration of maximum connections to optimize performance in scenarios with high connection volumes.
  • Batch strategy for sink:
    • Introduce common properties for sinks to configure batch strategies, enabling efficient data handling and reducing I/O operations as needed.
  • Complex data extraction for sink:
    • Simplify data transformation and processing by introducing common properties for sinks to extract specific data fields and optimize column selection, resulting in improved data processing performance.
  • Array type payload support:
    • Enhance the source to support array type payloads, allowing for seamless handling and processing of complex data structures.
  • Unnest function:
    • Introduce the unnest function, enabling the conversion of array types into multiple lines of data, facilitating more granular and detailed data processing.
  • Array and object functions:
    • Expand the array and object functions library with over 10 additional functions, providing extensive support for complex data processing scenarios.
  • Dot notation for nested field access:
    • Introduce dot notation to access nested fields within data structures, simplifying data retrieval and manipulation within complex nested hierarchies.
  • External state access in Redis:
    • Enable the ability to read external state information from Redis, facilitating efficient data retrieval and integration with external applications.
  • SQL syntax enhancements:
    • Enhance the SQL syntax by introducing support for expressions in array indexing, enabling more flexible and dynamic array operations within SQL queries.
  • Rule-related functions:
    • Introduce additional functions to retrieve the current rule ID and enable pipeline delay, providing greater control and flexibility in rule execution and pipeline management.
  • Graph API for lookup table:
    • Enabling efficient lookup table and lookup join in Graph API.

With this release, we reaffirm our dedication to empowering organizations with a cutting-edge, reliable, and user-friendly solution for harnessing the full potential of edge analytics and stream processing. Learn more about these and other features of eKuiper’s 1.10 release in the release notes.

What’s Next

In our upcoming version, we are thrilled to prioritize further advancements in the streaming and SQL runtime of eKuiper. Our focus will revolve around enriching the window functionality, particularly empowering the event window to cater to a wider range of scenarios. To ensure seamless integration and compatibility, we are committed to adding and refining the SQL syntax to align with standard SQL specifications. This will provide users with a familiar and intuitive experience when working with eKuiper, while enabling greater flexibility and ease of use.

Stay tuned for more updates and join us on this exciting journey as we continue to shape the future of edge computing with eKuiper.

June 16, 2023

Key New Usability and Security Features in Major EdgeX V3 Release Help It Stand Out as the Leading Open Source Edge Data Platform

By Blog, EdgeX Foundry, Project Release

I am delighted to announce the release of the latest and greatest version of EdgeX Foundry the open source edge data platform. Version 3.0 is our most significant upgrade to date and I firmly believe will bring much value to edge solution developers in a variety of industrial sectors.

As the EdgeX TSC chair and product manager of IOTech’s commercial EdgeX-based solutions, I’d like to explain what EdgeX V3 brings and describe how the platform continues to be the key framework upon which companies rely when developing their commercial edge software solutions.

A jump of major version is significant for EdgeX, not only because it indicates a new set of key features for its user base, but also because it shows the mature, yet evolutionary outlook of the base platform and the associated commercial edge products that are available in the ecosystem.

EdgeX Reminder

EdgeX Foundry is an open source edge data platform, managed and hosted under the Linux Foundation Edge umbrella. More specifically, EdgeX is a highly flexible and scalable architecture that allows easy interoperability between edge devices and applications. It is made possible thanks to an ecosystem of community collaborators (including Intel, IOTech, Canonical, Eaton, Beechwoods and many others) representing a variety of industries.

Minnesota

To help in its identification, each release of EdgeX is provided a codename based on a geographical location. The previous version was named Levski (located in Bulgaria). The next stopping point on our virtual world tour is Minnesota! Aptly named, in part for being a key industrial center of the United States, EdgeX Minnesota helps to further serve the needs of OT/IT integration at the edge.

The major jump in version number lets us make more significant updates to the framework and allows us the freedom to make changes that are otherwise prohibited by our minor version compatibility rules. Of course, we try to be backwards compatible where possible, but after two years of V2 minor releases, there are some key updates and usability benefits we can now bring in…

Easier, Common Configuration

The main theme of Minnesota is usability. In particular, the way the user configures EdgeX has been much simplified and improved. Version 3.0 of EdgeX introduces a common configuration pattern for the EdgeX microservices. Since many config options (logging, telemetry, security, database and message bus settings) apply across multiple microservices, we’ve added the ability to configure these in a single common location. Some microservices are likely to have common config requirements, e.g., the southbound device services or northbound application services, so we’ve provided a layered design where users can configure settings at their chosen level.

This will save developers and deployers of EdgeX systems a lot of time and reduce maintenance effort. It will be much easier to make config changes, rerun the system and be confident that the settings have all taken effect appropriately. Take a look at the latest EdgeX docs for the details.

Reduced Config Formats

The jump in major version has also allowed us to reduce the number of different config file formats in EdgeX. Removing TOML and using only YAML and JSON going forward helps reduce what a new user has to learn. YAML and JSON can be derived from each other, so we let users pick their favorite and work with that. Again, usage of the platform becomes easier and more maintainable.

Service Authentication

Another big feature for 3.0 is the addition of microservice authentication within EdgeX. When run in secure mode, the microservices now require a validated authentication token before they are allowed to communicate. Thanks to the efforts of the EdgeX Security Working Group, this is all built into the platform with tokens issued by the Secret Store service. The security advancements here also allow version 3 to switch to a much lighter-weight API Gateway service than was used previously, which helps a lot with our ongoing effort to minimize the footprint of the platform.

Other V3 Updates

Of course, we also have V3 versions of the EdgeX APIs. We do have some minor changes there including updates to help with the automatic discovery of devices, but the APIs are mature so there aren’t the major updates we had with the previous major jump. You can check out the full set of changes in the EdgeX V3 migration guide.  Full release notes are here.

Using EdgeX V3 and Joining in

We’re expecting that the next version after this (3.1 or Napa) planned for Q4 this year, will become the new Long Term Support (LTS) version of EdgeX. Those users wishing to jump from the previous LTS (2.1 or Jakarta) should start to become familiar with the new features. There are some breaking changes, so we encourage you to start to get to know V3 sooner rather than later. The features put into EdgeX Minnesota will serve as the basis for other features for the next few years. For example, using URLs to point to configuration and profiles that could come from anywhere and be shared is only possible because of the common configuration feature added now.

So please do go ahead and download EdgeX V3. A good place to get going is the Getting Started Guide and the lively EdgeX discussions board is a great place to chat and leave feedback.

June 5, 2023

Efficiently Collect, Transform and Transit Your Data With eKuiper 1.9 Release

By Blog, eKuiper, Project Release

eKuiper—a lightweight IoT data analytics and streaming software—is now available in its 1.9.0 release. eKuiper, an LF Edge project, migrates real-time cloud streaming analytics frameworks such as Apache Spark, Apache Storm and Apache Flink to the edge. eKuiper references these cloud streaming frameworks, incorporates any special requirements of edge analytics and introduces rule engine, which is based on Source, SQL (business logic) and Sink; rule engine is used for developing streaming applications at the edge.

eKuiper 1.9 release continues to enhance the source/sink connectors to make it easier to connect and transmit data with lower bandwidth. The community has also enhanced the data transformation ability to flexibly encode and compress any part of your data. The 1.9 release adds a number of significant new features, among them are

  • Multiple neuron connection to analyze collected data from multiple IOT gateways together
  • MQTT sink/source compression/decompression support, save bandwidth for edge cloud communication
  • HTTPPull source & REST sink support dynamic token based authentication, connect to more services out of box
  • More transformation and compression functions added, handle your data flexibly
  • Partial data export/import, migrate only interested rules and the dependencies
  • Run python plugin in conda virtual environment, separate the python runtime env

Learn more about these and other features of eKuiper’s 1.9 release in the release notes.

What’s next

In the next release, the community will adapt to the EdgeX Foundry‘s Minnesota version, while exploring the use of external states such as Redis states to achieve persistent states.

April 13, 2023

Baetyl New Release Integrates With eKuiper and Delivers Edge to More Devices

By Baetyl, Blog, eKuiper, Project Release

Baetyl—an LF Edge project that extends cloud computing, data and service seamlessly to edge devices— has released v2.4.3 release. With the efforts of many active contributors, new functions have been added, and some existing functions have been continuously optimized since the previous v2.3.0 release. These new features continue to follow the cloud-native philosophy and build an open, secure, scalable, and controllable intelligent edge computing platform.

Compared to the previous Baetyl v2.3.0 releases, the new features and optimizations in v2.4.3 include:

  • Device management functionality has been refactored with the addition of a device template interface, support for calculating OT data collection values, support for IEC-104 protocol, updated OPC-UA and Modbus drivers, and updated driver-node binding logic;
  • Support for Windows platform has been added with the ability to generate Windows – platform images for the Baetyl edge main module;
  • Remote invocation has been implemented, allowing for remote access to specified edge services with results returned from the cloud;
  • New container mode with eKuiper as an optional system application;
  • New baetyl-rule module supports HTTP Target;
  • Adaptation to the highest K3s 1.24.4 version;
  • Fix workload type creation failure bug;

These new features are available and now you can view the release note here. Other features can be further explored by the developers, and Baetyl will continue to improve and optimize its functionality.

Integration with eKuiper

Baetyl uses eKuiper as a system optional application on the edge-side for stream processing and data analytics. The collaboration and integration with eKuiper increase the linkage between LF Edge projects and promote innovation. Platform and version adaptation enables Baetyl to run on more devices. The optimization of device management and new driver support prepare for the access of more devices with different protocols. From the point of eKuiper, this means you can deploy eKuiper more quickly and conveniently.

In the new version, the integration between Baetyl and eKuiper makes the following changes, including:

  • set mqtt client as Baetyl broker; 
  • mount eKuiper’s data file to the host to ensure no configuration loss; 
  • add k8s service to the eKuiper application to enable calling eKuiper’s open API from the host as well as from within the cluster to enable edge configuration changes; 
  • built-in eKuiper as an optional application in the Baetyl framework, so that eKuiper can be installed directly through Baetyl, eliminating the need for separate installation; 

After the integration, the Baetyl framework enhances the ability of edge message processing, while users can use Baetyl’s ability to use eKuiper more easily and quickly.

What’s next?

For future releases, the project is working on strengthening the management of non-intelligent devices at the edge, including providing

  • more comprehensive management functions for device models, devices, and drivers on the cloud management platform, 
  • a software gateway management module on the edge to support the ability of devices to connect through various industrial drivers,
  • a unified northbound connection protocol blink for access implementation.

The Baetyl project is also further expanding and optimizing the implementation of cloud storage at the underlying level, providing support for database storage outside of k8s crds. It’ll also enhance the integration with K8s cloud-native, providing more edge cloud-native capabilities for access, such as providing the ability to view edge description information, and so on.

March 27, 2023

Home Edge Eucalyptus (vE) Release Enhances Home Data Privacy and Adds Android Support

By Blog, Home Edge, Project Release

Home Edge—an open source edge computing framework for home devices—is now available in its Eucalyptus (vE) release. Home Edge, an LF Edge project, is a robust, reliable and intelligent home edge computing open source framework and ecosystem running on a variety of devices in the home. To accelerate the deployment of the edge computing services ecosystem successfully, the Home Edge Project provides users with an interoperable, flexible, and scalable edge computing services platform with a set of APIs that can also run with libraries and runtimes.

“I am excited to share the availability of vE version of Home Edge on Github,” said Suresh L C, Home Edge Technical Steering Committee (TSC) Co-Chair. “The new Home Edge release adds Android execution support with a secure, robust and end-to-end framework for intelligent service offloading in smart home scenarios, which in turn provides data privacy with low latency response.”

The vE release adds a number of significant new features, among them are

  • Open Source Security Foundation (OpenSSF) Badge
    • Home Edge has achieved the OpenSSF best practices Gold badge and the OpenSSF scorecard has also been integrated. Go-Project is made compliant by incorporating necessary documentation/quality fixes.
  • Data synchronization to Cloud endpoints – MQTT
    • Independent MQTT based cloud synchronization mechanism as enhancement to vD
    • API’s to send and receive data from and to between Service application and Cloud via Home Edge
    • MQTT broker would be configured at the cloud (Cloud agnostic)
    • Home Edge acts as MQTT client to exchange data 
    • TLS secure mode for all the data exchange
  • Platform enhancement
    • The support for the execution of Home Edge on Android has been added to Home Edge Android
    • The base code currently supports device/service discovery and service offload to Linux device
    • Service offload to Android from Linux is not supported in this release
  • Dependent bot integration
    • Bot to check on the updates on dependent libraries has been integrated
    • The bot has been configured to run once every month in the first week
  • Code enhancements
    • Fixed TxT Record parsing as per mDNS protocol
    • Refactored service list assignment logic TxT record
    • Modified get score API from GET to POST in line with API design rules
    • API to add/delete events from database based on event ID added for DataStorage
    • Code coverage increased by adding more test cases
    • Modified response for Ping request to Pong
  • Auto numerical tagging of code when significant changes are incorporated

Learn more about these and other features of Home Edge vE release in the release notes.

What’s next

For the next steps, the community will work on strengthening the Android version of Home Edge so that the features are in line with the Go version.

If you are using or evaluating Home Edge, please let the project TSC know and join the TSC meetings. Your feedback on the project is greatly appreciated!

November 11, 2022

Edgex Foundry Goes Full Message Bus With Its Latest EdgeX 2.3 “Levski” Release

By Blog, EdgeX Foundry, Project Release
EdgeX Foundry now fully embraces decoupled message bus communication within the platform

I am delighted to announce today the availability of the EdgeX Foundry 2.3 release which is codenamed Levski. This is the project’s 11th official release since its inception in 2017 and includes some significant new features and benefits to users that I’ll explain in this blog.

Having been elected as the chair of the EdgeX Technical Steering Committee (TSC) earlier this summer, it’s great to be able to announce the news of this release but it really is the technical expertise and dedication of the EdgeX development team that helps bring these releases to the wider community. Thanks again for everyone’s effort here. Great to be part of a group of many different companies and individuals working together to make great open source edge software.

What is Levski?

First let me give mention to the Levski codename. As you may know, each EdgeX release is named after a certain place or location in the world, with the specific place chosen each time significant contributors to the project. This version was named by two long term EdgeX contributors, Diana Atanasova and Malini B

handaru, both from VMware. For version 2.3 and the letter “L”, we come back over to Europe because Levski is a large mountain in the beautiful European country of Bulgaria. Bulgaria is also Diana’s homeland so it’s nice to be able to recognize that with the naming of this release.

New in EdgeX 2.3 – More about the Message Bus

Perhaps the biggest single new feature is the enhancement to support the delivery of commands via the EdgeX message bus.

We’ve previously made great strides in EdgeX V2 by delivering data from the southside (from devices and sensors) to the northside (to the core and application layers) via an internal message bus. This release also adds the support in the other direction, i.e., from the northside (the application layers and core layers) down to the southside (the devices and sensors) on the same message bus. Previously the southbound communication was exclusively via REST. Moving to support message bus-based communication in both directions is a big advancement and brings key benefits in terms of reduced latency and increased scalability. The asynchronous communication and the QoS-based control that you get from the message bus implementations that you can choose adds delivery guarantees and retransmissions of messages as needed.

Message Bus

More Run-Time Data

Another key area of development for Levski has been the focus on providing EdgeX users with more live information about how the system is running. EdgeX 2.3 adds System or “Control Plane” Events that can provide live updates as to what is happening. For example, users can receive notifications that new devices are added or that there has been a network disconnection from a specific device.

Somewhat related to System Events are the Telemetry Metrics. In contrast to events though, the metrics provide numeric information relating to how the platform is operating. Examples include the number of data readings that are persisted by core data or the number of secrets or tokens that are stored. Building of what was delivered previously, Levski adds more metrics across all of the services.

All of this information can be collected and reported however the user sees fit. In addition, the System Events and Telemetry Metrics mechanisms are available for users to add their own metrics as needed.

Many other updates and enhancements

There are many other new additions in this last release cycle including:

  • Availability of NATS as alternative implementation of the internal EdgeX Message Bus. NATS is a popular and lightweight protocol with native delivery of EdgeX messages and potential advantages in high availability (HA) use cases
  • Authenticated access to the MQTT message bus
  • Securing of the Consul registry service with access tokens
  • Passing status at the Open Source Security Foundation (OpenSSF)
  • Initial construction of the EdgeX STRIDE threat model
  • Improving the EdgeX development process with a new Use Case Requirements (UCRs) phase in the design procedure

See the release notes here for full details of what the Levski release provides.

New adopters and use cases for EdgeX

Running as a mature, stage 3 Linux Foundation project, there are now many users and adopters of EdgeX technology around the world. That includes users who download and deploy the open source EdgeX code, but also users of commercial products (including ours at IOTech) that are based on EdgeX. Take a look at some of the companies who are users of EdgeX. There are also a good set of presentation videos from different adopters of the technology.

One of my personal aims as TSC chair is to help encourage the adoption and wider use of EdgeX. Thanks to Building System Integrators (BSI) who recorded a talk at a recent TSC meeting. It would be great to hear from more companies who would like to do something similar. Please do reach out to me to arrange it.

What’s coming next?

Each release marks a busy period where of course we finalize the current version, but already we are looking forward at what we can achieve in the next release. Codenamed Minnesota and expected to be version 3.0, we are looking to add new features to help with the scalable configuration of EdgeX. Tune into the Minnesota technical planning conference that we are running next week where we will scope out what we can do. As always, attendance and contributions are very welcome.

Thanks again to everyone who had a hand in this release, and we look forward to more successful releases in the future.

November 9, 2022

eKuiper Issues 1.7.0 Release, Adding Lookup Table, Analytic Functions and More

By Blog, eKuiper, Project Release

eKuiper—a lightweight IoT data analytics and streaming software—is now available in its 1.7.0 release. eKuiper, an LF Edge project, migrates real-time cloud streaming analytics frameworks such as Apache Spark, Apache Storm and Apache Flink to the edge. eKuiper references these cloud streaming frameworks, incorporates any special requirements of edge analytics and introduces rule engine, which is based on Source, SQL (business logic) and Sink; rule engine is used for developing streaming applications at the edge.

eKuiper 1.7.0 continues to improve the expressiveness of the rules by providing the lookup table, updatable sink, and more SQL syntax and functions. The 1.7.0 release adds a number of significant new features, among them are

  • Stream batch combined computation
    • Not all data changes frequently, even in real-time computation. In some cases, you may need to supplement stream data with externally stored static data. For example, user metadata may be stored in a relational database, and the stream data only has data that changes in real time, requiring a connection between the stream data and the batch data in the database to enrich the data. In the new version, eKuiper adds the new Lookup Table concept for binding external static data, which can be joined with stream data in the rules to realize the operation of stream-batch combination. Additionally, with the support of updatable sink, eKuiper can process CDC data and update the batch storage.
  • Enhanced analysis capabilities
    • Analytic functions refer to functions that perform state-related analysis by saving state, allowing users to complete some of the common stateful calculations without complex time windows or custom functions. In the new version, we have added the Partition By syntax to the analytic functions, allowing users to perform partitioned stateful calculations based on the dimensions defined by Partition By clause. We also added a new analysis function latest to get the latest value of data. It is used to collect unstructured data whose columns are not fixed and can be automatically stitched to calculate the complete data.
  • Expanding the connectivity ecosystem 
    • On one hand, we continue to add built-in and extended source/sink by adding Httppush source, which allows users to push data to eKuiper via HTTP protocol; adding built-in Redis lookup source, which supports using Redis as external lookup table; adding Influx V2 sink, which supports writing to InfluxDB 2.x version. At the same time, Memory, SQL, and Redis source are adapted to support lookup tables; Memory, SQL, and Redis sink are adapted to be updatable.
  • Enhance the Ops experience
    • The new release refactors the external connection configuration API to enable easier connection resource management. Another major update comes from the bulk import/export and initialization of streams and rules to facilitate the migration of eKuiper instances.

Learn more about these and other features of eKuiper’s 1.7.0 release in the release notes.

What’s next

In the next release, the community will work on enhancing graph API to support more nodes, such as switch node and script node, to provide a more powerful and easy-to-use rule pipeline composing tool. A new function to run tensor flow lite model will be provided to easily integrate with AI models. Check out the evolving Github 1.8.0 milestones and we welcome you to add your wisdom.

November 9, 2022
Close Menu