ARM CPU for Cost-Effective Apache Kafka at the Edge and Cloud

ARM CPUs often outperform x86 CPUs in scenarios requiring high energy efficiency and lower power consumption. These characteristics make ARM preferred for edge and cloud environments. This blog post discusses the benefits of using Apache Kafka alongside ARM CPUs for real-time data processing in edge and hybrid cloud setups, highlighting energy-efficiency, cost-effectiveness, and versatility. A wide range of use cases are explored across industries, including manufacturing, retail, smart cities and telco.

Apache Kafka at the Edge and Hybrid Cloud

Apache Kafka is a distributed event streaming platform that enables building real-time streaming data pipelines and applications by providing capabilities for publishing, subscribing to, storing, and processing streams of records in a scalable and fault-tolerant way.

Various examples exist for Kafka deployments on the edge. These use cases are related to several of the above categories and requirements, such as low hardware footprint, disconnected offline processing, hundred of locations, and hybrid architectures.

Use Cases for Apache Kafka at the Edge

I have worked with enterprises across industries and the globe on the following scenarios:

• Public Sector: Local administration in each city, smart city projects including public transportation, traffic management, integration of various connected car platforms from different carmakers, cybersecurity (including IoT use cases such as capturing and processing camera images)
• Transportation / Logistics / Railway / Aviation: Track and trace, Kafka in the trains for offline and local processing / storage, traveller information (delayed or canceled flight / train / bus), real-time loyalty platforms (class upgrade, lounge access)
• Manufacturing (Automotive, Aerospace, Semiconductors, Chemical, Food, and others): IoT aftermarket customer services, OEM in machines and vehicles, embedding into standard software such as ERP or MES systems, cybersecurity, a digital twin of devices/machines/production lines/processes, production line monitoring in factories for predictive maintenance/quality control/production efficiency, operations dashboards and line wellness (on-site for the plant manager, and aggregated global KPIs for executive management), track&trace and geofencing on the shop floor
• Energy / Utility / Oil & Gas: Smart home, smart buildings, smart meters, monitoring of remote machines (e.g., for drilling, windmills, mining), pipeline and refinery operations (e.g., predictive failure or anomaly detection)
• Telecommunications / Media: OSS real-time monitoring/problem analysis/metrics reporting/root cause analysis/action response of the network devices and infrastructure (routers, switches, other network devices), BSS customer experience and OTT services (mobile app integration for millions of users), 5G edge (e.g., street sensors)
• Healthcare: Track & trace in the hospital, remote monitoring, machine sensor analytics
• Retailing / Food / Restaurants / Banking: Customer communication, cross-/up-selling, loyalty system, payments in retail stores, perpetual inventory, Point-of-Sale (PoS) integration for (local) payments and (remote) CRM integration, EFTPOS (Electronic funds transfer at point of sale)

Benefits for Kafka at the Edge AND in the Cloud

Deploying the same technology in hybrid environments is not a new idea. Project teams see tremendous benefits when using Kafka at the edge and in the data center or cloud:

• Same APIs, concepts, development tools and testing
• Same architecture for streaming, storing, processing and connecting systems, even if at very different scale
• Real-time synchronization between multiple environments included out-of-the-box via the Kafka protocol

Let’s explore how ARM CPUs fit into this discussion.

What is ARM CPU?

An ARM CPU refers to a family of CPUs based on the Advanced RISC Machine (ARM) architecture, which is a type of Reduced Instruction Set Computing (RISC) architecture. ARM CPUs ave a reputation for their high performance, power efficiency, and low cost. These characteristics make them particularly popular in mobile devices such as smartphones, tablets, and an increasingly wide range of other devices like IoT (Internet of Things) gadgets, servers, and even desktop computers.

The ARM architecture performs operations with a smaller number of computer instructions, allowing it to achieve high performance with lower power consumption compared to more complex instruction set computing (CISC) architectures like x86 used by Intel and AMD CPUs. This efficiency is a key advantage for battery-powered devices, where energy conservation is critical.

ARM Holdings, the company behind the ARM architecture, does not manufacture CPUs but licenses the architecture to other companies. These companies can then implement their own ARM-based processors, potentially customizing them for specific needs. This licensing model has led to a wide adoption of ARM processors across various segments of the technology industry.

ARM32 vs. ARM64

ARM architectures come in different versions, primarily distinguished by their instruction set architectures and addressing capabilities. The most commonly referenced are ARMv7 and ARMv8 (also called AArch64) correspond to 32-bit and 64-bit processing capabilities, respectively.

Newer hardware for industrial PCs and home computers incorporates ARMv8 (64-bit). It is the foundation for smartphones, tablets, servers, and processors like Apple’s A-series chips in iPhones and iPads. Even the cloud providers use the ARM architecture to build new processors for cloud computing, like Amazon’s Graviton. ARMv8 processors can run both 32-bit and 64-bit applications, offering greater versatility and performance.

Key Features and Benefits of ARM CPUs

The key features and benefits of ARM CPUs include:

• Power Efficiency: Their design allows for significant power savings, extending battery life in portable devices.
• Performance: While historically seen as less powerful than their x86 counterparts, modern ARM processors offer competitive performance, especially in multi-core configurations.
• Customization: Companies can license the ARM architecture and customize their own chips, allowing for optimized processors that meet specific product requirements.
• Ecosystem: A broad adoption across mobile, embedded, and increasingly in server and desktop markets ensures a robust ecosystem of software and development tools.

ARM CPUs are central to the development of mobile computing and are becoming more important in other areas, including edge computing, data centers, and as part of the shift towards more energy-efficient computing solutions.

Why ARM CPUs at the Edge (e.g., for Industrial IoT)?

ARM architecture is favored for edge computing, including Industrial IoT. It provides high power efficiency and performance within compact form factors. These characteristics ensure devices can handle compute-intensive tasks locally. Only relevant data is transmitted to the cloud, which saves bandwidth and decreases latency.

The efficiency of ARM CPUs is crucial for industrial applications where real-time processing and long battery life are essential. ARM’s versatility and low power consumption make it ideal for the diverse needs of edge computing in various environments.

For instance, in manufacturing, ARM-powered sensors on machines enable predictive maintenance by monitoring conditions like vibration and temperature. These sensors process data locally, offering real-time alerts on potential failures, reducing downtime, and saving costs. ARM’s efficiency supports widespread deployment, making it ideal for continuous, autonomous monitoring in industrial environments.

Why ARM in the Cloud?

ARM’s efficiency and performance advantages are driving its adoption in cloud computing. ARM-based processors, like Amazon’s AWS Graviton, offer an attractive mix of high performance and lower power consumption compared to traditional x86 CPUs. This efficiency translates into cost savings and reduced environmental impact for cloud service providers and their customers.

AWS Graviton, specifically designed for cloud workloads, exemplifies how ARM architecture can optimize operations in data centers, enhancing the performance of web servers, containerized applications, and microservices at a lower cost. This shift towards ARM in the cloud represents a significant move towards more energy-efficient and cost-effective data center operations.

Apache Kafka on ARM – A Match Made in Heaven for Edge and Cloud Workloads

Using ARM architecture together with Apache Kafka, a distributed streaming platform, offers several advantages, especially in scenarios that demand high throughput, scalability, and energy efficiency.

1. Energy Efficiency and Cost-Effectiveness: ARM processors are known for their low power consumption, which makes them cost-effective for running distributed systems like Kafka. Deploying Kafka on ARM-based servers can reduce operational costs, particularly in large-scale environments where energy consumption can significantly affect the budget.
2. Scalability: Kafka handles large volumes of data and high throughput, characteristics that align well with the scalability of ARM processors in cloud environments. ARM’s efficiency enables scaling out Kafka clusters more economically, allowing for the processing of streaming data in real-time without incurring high energy or hardware costs.
3. Edge Computing: Kafka is a common choice for real-time data processing and aggregation in edge computing scenarios. ARM’s dominance in IoT and edge devices makes it a natural fit for these use cases. Running Kafka on ARM enables efficient data processing closer to the source, reducing latency and bandwidth usage by minimizing the need to send large volumes of data to central data centers.
4. Eco-Friendly Solutions: With growing environmental concerns, ARM’s energy efficiency contributes to more sustainable computing solutions. Deploying Kafka on ARM can be part of an eco-friendly strategy for organizations looking to minimize their carbon footprint.
5. Innovative Use Cases: Combining Kafka with ARM opens up new possibilities for innovative applications in IoT, real-time analytics, and mobile applications. The efficiency of ARM allows for cost-effective experimentation and deployment of new services that require real-time data processing and streaming capabilities.

Examples and Case Studies for Kafka at the Edge

Overall, the combination of ARM and Apache Kafka supports the development of efficient, scalable, and sustainable data processing architectures, particularly suited for modern applications that require real-time performance with minimal energy consumption.

For several use cases, architectures and case studies about data streaming at the edge and hybrid cloud, check out my related articles:

• Use Cases and Architectures for Kafka at the Edge
• Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics and Retailing
• Infrastructure Checklist for Apache Kafka at the Edge
• Kafka for Real-Time Replication between Edge and Hybrid Cloud
• Apache Kafka in Air-Gapped Zero-Trust Environments with Data Diode / Unidirectional Gateway

Most of these blog posts are a few years old. But they are as relevant today as at the time of writing them. Actually, the official support of ARM CPU at the edge completely changes the conversations about challenges and solutions of deploying Kafka on edge infrastructure. The deployment of Kafka at the edge was never easier. If you buy a new Industrial PC (IPC) today, it will have enough hardware power to run Kafka and its ecosystem for data integration and stream processing easily.

Confluent Platform on ARM Infrastructure for Edge Deployments

Confluent Platform is Confluent’s data streaming platform for self-managed deployments of Apache Kafka. Most deployments operate in a traditional data center. However, this is more and more shifting to deploy at the edge, i.e., outside of a data center, too.

Since version 7.6., Confluent Platform officially supports ARM64 Linux architectures. Confluent Platform’s architecture allows you to run it wherever your IT systems are, across a global footprint. This includes running it as a mission-critical cluster in data centers, but also on edge sites like retail stores, ships or factories, or as a single broker on edge devices.

Confluent itself recognized the benefits of ARM64 CPUs: They moved the entire AWS fleet for the fully managed Confluent Cloud to ARM-based images in the past months.

The Confluent Server Broker powered by Apache Kafka enables end-to-end data pipelines:

• Collecting data from any source
• Persists the data in the event storage with separate compute and storage
• Processes the events with stream processing
• Share data with downstream applications
• Replicate selected events across the WAN through the native Kafka protocol via Cluster Linking.

Now, you can deploy this in production on low-cost, small-footprint ARM64 architecture infrastructure at the edge and also synchronize with a data center or cloud Kafka cluster.

Kafka + ARM = Cost-Effective and Sustainable

The article outlined the synergistic relationship between Apache Kafka and ARM CPUs. It enables efficient, scalable, and sustainable data processing architectures for edge and hybrid cloud environments.

The adoption of ARM in cloud computing marks a significant shift towards more sustainable and performance-optimized computing solutions. The combination of Kafka and ARM CPUs is poised to drive innovation in real-time analytics, IoT, and mobile applications. A few great examples:

• AWS Graviton to operate Kafka cost-efficient in the public cloud.
• Confluent Platform’s compatibility and support for ARM64 architectures at the edge.

The sustainability of energy-efficient ARM CPUs is a perfect segue to the data streaming article “Green Data, Clean Insights: How Kafka and Flink Power ESG Transformations“.

Do you already use ARM processors in your edge or cloud Kafka environment? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.