Apache Kafka for Industrial IoT and Manufacturing 4.0

This post explores use cases and architectures for processing data in motion with Apache Kafka in Industrial IoT (IIoT) across verticals such as automotive, energy, steel manufacturing, oil&gas, cybersecurity, shipping, logistics. Use cases include predictive maintenance, quality assurance, track and track, real-time locating system (RTLS), asset tracking, customer 360, and more. Examples include BMW, Bosch, Baader, Intel, Porsche, and Devon.

Why Kafka Is a Key Piece of the Evolution for Industrial IoT and Manufacturing

Industrial IoT was a mess of monolithic and proprietary technologies in the last decades. Modbus, Siemens S7, SCADA, and similar “concepts” controlled the industry. Vendors locked in enterprises by intentionally building incompatible products without open interfaces. These systems still run on Windows XP or similar non-supported outdated operating systems and without security in mind.

Fortunately, this is completely changing. Apache Kafka and its ecosystem play a key role in the IIoT evolution. System integration and data processing get an open architecture with a scalable, reliable infrastructure.

I speak to customers in this industry every week across the globe. Very different challenges, use cases, and innovative ideas originate. I covered this topic a lot in the past, already.

Related Content about Kafka and Industrial IoT

Check out my other related blog posts for Kafka in IIoT and Manufacturing. Learn about use cases and architecture for deployments at the edge (i.e., outside the data center), the relation between Kafka and other IoT standards like MQTT or OPC-UA, and how to build a modern, open and scalable data historian.

I want to highlight one post as it is superimportant for any discussion around shop floors, PLCs, machines, robots, cars, and any other embedded systems: Kafka and other IT software are NOT hard real-time.

This post here “just” shares my latest presentation on this topic, including the slide deck and on-demand video recording. Before we get there, let’s summarize the current scenarios for Kafka in Industrial IoT in one concrete example.

Requirements for Industrial IoT: Everywhere, Complete, Cloud-native!

Let’s take a look at one specific example. The following picture depicts the usage of event streaming in combination with other OT and IT technologies in the shipping industry:

This is an interesting example because it shows many challenges and requirements of many Industrial IoT real-world scenarios across verticals:

• Everywhere: Industrial IoT is not possible only in the cloud. The edge is impossible to avoid because manufacturing produces tangible goods. Integration between the (often disconnected) edge and the data center is essential for many use cases.
• Complete: Industrial IoT is mission-critical. Stability with zero downtime, security, and safety are crucial across verticals. The only realistic option is a robust, battle-tested enterprise-grade solution to realize IIoT use cases.
• Cloud-native: Automation, scalability, decoupled agile applications, and flexibility regarding technologies and products are required for enterprises to stay competitive. Not just in the cloud, but also at the edge! Not all use cases required a critical, scalable solution, though. For instance, a single broker for data processing and storage is sufficient in a disconnected drone.

A unique value of Kafka is that you can use one single technology for scalable real-time messaging, storage and caching, continuous stateless and stateful data processing, and data integration with the OT and IT world. This is especially important at the edge where the hardware is constrained, and the network is limited. It is much easier to operate and much more cost-efficient to deploy one single infrastructure instead of glue together a best-of-breed like you often do in the cloud.

With this introduction, let’s now share the slide deck and video recording to talk about all these points in much more detail.

Slide Deck: Kafka for Industrial IoT and Manufacturing 4.0

Here is the slide deck:

Video Recording: Connect All the Things

Here is the video recording:

 

Apache Kafka for an open, scalable, flexible IIoT Architecture

Industrial IoT was a mess of monolithic and proprietary technologies in the last decades. Fortunately, Apache Kafka is completely changing many industrial environments. An open architecture with a scalable, reliable infrastructure changes how systems are integrated and how data is processed in the future.

What are your experiences and plans in IIoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.