Apache Kafka and MQTT are a perfect combination for many IoT use cases. This blog series covers the pros and cons of both technologies. Various use cases across industries, including connected vehicles, manufacturing, mobility services, and smart city are explored. The examples use different architectures, including lightweight edge scenarios, hybrid integrations, and serverless cloud solutions. This post is part two: Connected Vehicles and V2X applications.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles (THIS POST): MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City: MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Connected Vehicles and V2X

Vehicle-to-everything (V2X) is communication between a vehicle and any entity that may affect, or may be affected by, the vehicle. It is a vehicular communication system that incorporates other more specific types of communication as V2I (vehicle-to-infrastructure), V2N (vehicle-to-network), V2V (vehicle-to-vehicle), V2P (vehicle-to-pedestrian), V2D (vehicle-to-device), and V2G (vehicle-to-grid). The main motivations for V2X are road safety, traffic efficiency, energy savings, and better driver experience.

V2X includes various use cases. The following picture from 3G4G shows some examples :

Business Point of View for Connected Vehicles

From a business perspective, the following diagram from Frost & Sullivan explains the use cases for connected vehicles very well:

Technical Point of View for V2X and Connected Vehicles

A few things to point out from a technical perspective:

• MQTT + Kafka provides a scalable real-time infrastructure for high volumes of data in motion in milliseconds with end-to-end processing between 10 and 20ms. This is good enough for the integration with backend IT systems and almost all mobility services.
• MQTT and Kafka are not used for hard real-time and deterministic embedded systems.
• Some safety-critical V2X use cases require other communication technologies such as 5G New Radio (NR) / NR C-V2X sidelink to directly connect vehicles or vehicles and local infrastructure (e.g. traffic lights). There is no need for an intermediary cellular network or radio access network (RAN).
• Example: A self-driving car executes all its algorithms like image processing and decision making within the car in embedded systems. These use cases require deterministic behavior and hard real-time. Communication with 3rd party such as emergency services, traffic routing, parking, etc., connects to backend systems for data correlation (close to the edge or far away in a cloud data center). Real-time in milliseconds – or sometimes even seconds – is good enough in these cases.
• Not every application is for tens or hundreds of thousands of connected vehicles. For instance, a real-time locating system (RTLS) is a perfect example for realizing use cases in logistics in transportation. This can be geofencing within a plant or regional global track&trace. “Real-Time Locating System (RTLS) with Apache Kafka for Transportation and Logistics” explores this use case in more detail.

The following sections focus on use cases that require real-time (but not hard real-time) data integration and processing at scale with 24/7 uptime between vehicles, networks, infrastructure, and applications.

Architecture: MQTT and Kafka for Connected Vehicles

Let’s take a look at an example: Remote control and command of a car. This can be simple scenarios like opening your car trunk from a remote location with your digital key for the mailman or more sophisticated use cases like the payment process for buying a new feature via OTA (over the air) update.

The following diagram shows an architecture for V2X leveraging MQTT and Kafka:

A few notes on the above architecture:

• The MQTT and Kafka clusters run in a Kubernetes environment.
• Kubernetes allows the deployment across data centers and multiple cloud providers with a single “template”.
• Bi-directional communication is guaranteed in reliable, scalable infrastructure end-to-end in real-time.
• The MQTT clients from cars and mobile devices communicate with the MQTT cluster. This allows connecting hundreds of thousands of interfaces and support of bad networks.
• Kafka is the integration backbone for connected vehicles and mobile devices. Use cases include streaming ETL, correlation of the data in stateful business applications, or ingestion into other IT applications, databases, and cloud services.

V2X with MQTT and Kafka in a 5G Infrastructure

The following diagram shows the above use cases around connected vehicles from the V2X perspective:

The infrastructure is separated into three categories and networks:

• The edge (vehicles, devices) using local processing and remote integration via 5G.
• MEC (multi-access edge computing) region for low-latency use cases. This example leverages AWS Wavelength for combining the power of 5G with cloud services and Confluent Platform for processing data in motion at scale.
• The public cloud infrastructure using AWS and Confluent Cloud for all other cloud-native applications.

The integration between the edge and the IT world depends on the requirements. In this example, we use mostly MQTT but also HTTP for the integration with the Kafka cluster. The connectivity to other IT applications happens via Kafka-native interfaces such as Kafka clients, Kafka Connect, or Confluent’s Cluster Linking (for the bi-directional replication between the AWS Wavelength zone and the AWS cloud region).

Direct communication between vehicles or vehicles and pedestrians requires deterministic behavior and ultra-low latency. Hence, this communication does not use technologies like MQTT or Kafka. Technologies like 5G Sidelink were invented for these requirements.

Let’s now look at two-real world examples for connected vehicles.

Example: MQTT and Kafka for Millions of Connected Cars @ Autonomic

Autonomic built the Transportation Mobility Cloud (TMC), a standard way of accessing connected vehicle data and sending remote commands. This platform provides the foundation to build smart mobility applications related to driver safety, preventive maintenance, fleet management.

Autonomic built a solution with MQTT and Kafka to connect millions of cars. MQTT forwards the car data in real-time to Kafka to distribute the messages to the different microservices and applications in the platform.

This is a great example of combining the benefits of MQTT and Kafka. Read the complete case study from HiveMQ for more details.

Example: Kafka as Car Data Collector @ Audi

Audi started its journey for connected cars a long time ago to collect data from hundreds of thousands of cars in real-time. The car data is collected and processed in real-time with Apache Kafka. The following diagram shows the idea:

As you can imagine, tens of potential use cases exist to reduce cost, improve the customer experience, and increase revenue. The following is the example of a real-time service to find a free parking lot:

Watch Audi’s Kafka Summit keynote for more details about the infrastructure and use cases.

Slide Deck – Kafka for Connected Vehicles and V2X

Here is a slide deck covering this topic in more detail:

Kafka + MQTT = Connected Vehicles and V2X

In conclusion, Apache Kafka and MQTT are a perfect combination for V2X and connected vehicles. It makes so many new IoT use cases possible!

Follow this blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.