Apache Kafka and MQTT are a perfect combination for many IoT use cases. This blog series covers the pros and cons of both technologies. Various use cases across industries, including connected vehicles, manufacturing, mobility services, and smart city are explored. The examples use different architectures, including lightweight edge scenarios, hybrid integrations, and serverless cloud solutions. This post is part five: Smart City and 5G.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City (THIS POST): MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Smart City and 5G

A smart city is an urban area that uses different types of electronic Internet of Things (IoT) sensors to collect data and then use insights gained from that data to manage assets, resources, and services efficiently.

A smart city provides many benefits for civilization and city management. Some of the goals are:

• Improved Pedestrian Safety
• Improved Vehicle Safety
• Proactively Engaged First Responders
• Reduced Traffic Congestion
• Connected / Autonomous Vehicles
• Improved Customer Experience
• Automated Business Processes

I covered the use cases in more detail in the post “Event Streaming with Kafka as Foundation for a Smart City“. For a specific 5G example, check out “Building a Smart Factory with Apache Kafka and 5G Campus Networks“.

Let’s now explore the relation of Kafka and MQTT for smart city use cases.

Architecture: MQTT and Kafka for a Smart City

The following architecture shows an infrastructure deployed at a stadium:

In this example, both MQTT and Kafka are deployed close to the stadium. For instance, AWS Wavelength is an innovative infrastructure option to build low latency 5G use cases. The connected “regular AWS cloud region” is still used for use cases that do not require low latency.

The combination of Kafka and MQTT enables connectivity and real-time data processing for various use cases:

• Parking information and smart navigation.
• Location-based shopping and restaurant experiences, including innovative scenarios such as monitoring of queues and geofencing.
• Integration of loyalty platforms to earn rewards and points.
• Live information about the game or concert
• Lottery drawing experiences while watching a sports game.

The possibilities are endless. Integration with 1st and 3rd party applications will create completely new opportunities to improve the customer experience, increase safety, and improve operational efficiency.

The stadium example is a particular scenario to explore the added value of processing data in motion. Let’s take a look at other real-world examples that leverage MQTT and Kafka.

Example: Cloud-based Traffic Control Systems @Berlex

The Swedish company Berlex designs and manufactures new ways to improve traffic safety.

Berlex provides cloud-based portable traffic signals. Their innovative R6 traffic signal is one of the first mobile traffic signals controlled by a cloud-based service. Berlex’s connected solution allows customers to monitor the new traffic signals on a smartphone, computer, or tablet anytime and from anywhere. MQTT enables real-time information delivery and constant monitoring.

The cloud-based service reduces the time that their customers need to spend in dangerous traffic work zones. The system enables customers to carry out numerous tasks such as checking the battery status of a traffic signal or performing an inspection remotely, with no need for risky and time-consuming on-site intervention.

Each portable R6 traffic signal is equipped with a radar that allows the signal to see traffic. Sensors within the signals publish detailed information on the current status of the signal as MQTT data. The Berlex Connect cloud service captures the continuous stream of MQTT data from each signal and shares the information with the appropriate subscribers.

To prevent interruption of the traffic signal operation, high availability is essential for the system. Berlex customers monitor the real-time information on individual portals with customized user roles that fit their specific use case.

Read the complete case study from HiveMQ for more details about this successful smart city project.

Example: The Life of Citizens as a Stream of Events @ NAV

NAV (Norwegian Work and Welfare Department) currently distributes more than one-third of the national budget to Norway or abroad citizens. NAV assists people through all phases of life within work, family, health, retirement, and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them.

In most countries, each person has to apply for these services resulting in many tasks handled manually by various caseworkers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our caseworkers and our users. By streaming a person’s life events through our Kafka pipelines, NAV revolutionized the way users experience government services and the way the employees work:

NAV and the government as a whole have access to vast amounts of data about the citizens, reported by health institutions, employers, various government agencies, or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. The data is ingested into streams using Kafka, Streams API, and Java microservices. NAV distributes and acts on events about birth, death, relationships, employment, income, and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services the government already knows are needed.

NAV chose Confluent Platform to implement to get valuable insight from life and business events. Security is a key concern. Compliance with GDPR is essential for the success of this project.

More details about NAV’s Kafka usage in their Kafka Summit presentation.

Kafka + MQTT = Smart City

In conclusion, Apache Kafka and MQTT are a perfect combination for smart city and 5G use cases. Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.