Data Streaming from Smart Factory to Cloud

A smart factory organizes itself without human intervention to produce the desired products. Data integration of IoT protocols, data correlation with other standard software like MES or ERP, and sharing data with independent business units for reporting or analytics is crucial for generating business value and improving the OEE. This blog post explores how data streaming powered by Apache Kafka helps connect and move data to the cloud at scale in real-time, including a case study from BMW and a simple lightboard video about the related enterprise architecture.

The State of Data Streaming for Manufacturing in 2023

The evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations require modern, open, and scalable information sharing. Data streaming allows integrating and correlating data in real-time at any scale. Trends like software-defined manufacturing and data streaming help modernize and innovate the entire engineering and sales lifecycle.

I have recently presented an overview of trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, and Siemens. A complete slide deck and on-demand video recording are included:

This blog post explores one of the enterprise architectures and case studies in more detail: Data streaming between edge infrastructure (like a smart factory) and applications in the data center or public cloud.

What is a Smart Factory? And how does Data Streaming help?

Smart Factory is a term from research in manufacturing technology. It refers to the vision of a production environment in which manufacturing plants and logistics systems primarily organize themselves without human intervention to produce the desired products.

The technical basis is cyber-physical systems, i.e., physical manufacturing objects and virtual images in a centralized system. Digital Twins often play a crucial role in smart factories for simulation, engineering, condition monitoring, predictive maintenance, and other tasks.

In the broader context, the Internet of Things (IoT) is the foundation of a smart factory. Communication between the product (e.g., workpiece) and the manufacturing system continues to be part of this future scenario: The product brings its manufacturing information in machine-readable form, e.g., on an RFID chip. This data controls the product’s path through the production system and the individual production steps. Other transmission technologies, such as WLAN, Bluetooth, color coding, or QR codes, are also being experimented with.

Data streaming helps connect high-volume sensor data from machines, PLCs, robots, and other IoT devices. Integrating and pre-processing the events with data streaming is a prerequisite for data correlation with information systems like the MES or ERP (that might run at the edge or more often in the cloud). The latter is possible in real-time at scale with stream processing. The de facto standard for data streaming is Apache Kafka and its ecosystems, like Kafka Stream and Kafka Connect.

BMW Group: Data from 30 Smart Factories Streamed to the Cloud

BMW Group needed to make all data generated by its 30+ production facilities and worldwide sales network available in real-time to anyone across the global business.

The data ingested by BMW from its smart factories into the cloud with data streaming enables simple access to the data for visibility and new automation applications by any business unit.

The Apache Kafka ecosystem facilitates the decoupling between logistics and production systems. Transparent data flows and the flexibility of building innovative new services are possible with this access to events from everywhere in the company.

Stability is vital in manufacturing across the supply chain. This begins with Tier 1 and Tier 2 suppliers up to the aftersales and service management. Direct integration from the shop floor to serverless Confluent Cloud on Azure ensures a mission-critical data streaming environment for data pipelines between edge and cloud.

The use case enables reliable data sharing across the logistics and supply chain processes for BMW’s global plants.

Data streaming enables:

• Access to information about the right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence process optimization
• Improved overall equipment effectiveness (OEE) for a lot of critical applications
• Last-mile integration between Apache Kafka and IoT standards like OPC-UA

Read more about BMW’s success story for IoT and cloud-native data streaming.

Lightboard Video: How Data Streaming Connects Smart Factory and Cloud

Here is a five-minute lightboard video that describes how data streaming helps with the integration between production facilities (or any other edge environments) and the cloud:

If you liked this video, make sure to follow the YouTube channel for many more lightboard videos across all industries.

IoT and Edge are not contradictory to Cloud and Data Streaming

The BMW case study shows how you can build reliable real-time synchronization between smart factories and cloud applications. However, there are more options. For more case studies, check out the free “The State of Data Streaming in Manufacturing” on-demand recording or read the related blog post.

MQTT is combined with Kafka regularly if the use case requires supporting bad networks or millions of IoT clients. Another alternative is data streaming at the edge with highly available Kafka clusters on industrial PCs, e.g., for air-gapped environments, or embedded single Kafka brokers, e.g., deployment in a machine.

Humans are still crucial for the success of a smart factory. Improving the OEE requires a smart combination of software, robots, and people. Augmented Reality leveraging Data Streaming is an excellent example. VR/AR platforms like Unity enable remote services, training, or simulation. Apache Kafka is the foundation for real-time data sharing across these different technologies and interfaces.

How do you leverage data streaming in your manufacturing use cases? Do you deploy at the edge, in the cloud, or both? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.