IoT and Data Streaming with Kafka for a Tolling Traffic System with Dynamic Pricing

In the rapidly evolving landscape of intelligent traffic systems, innovative software provides real-time processing capabilities, dynamic pricing and new customer experiences, particularly in the domains of tolling, payments and safety inspection. With the increasing complexity of road networks and the need for efficient traffic management, these organizations are embracing cutting-edge technology to revolutionize traffic and logistics systems. This blog post delves into success stories from Quarterhill and DKV Mobility providing traffic and payment systems for tolls. Data streaming powered by Apache Kafka has been pivotal in the journey towards building intelligent traffic systems in the cloud.

Traffic System for Tolls: Use Case, Challenges, and Business Models

Tolling systems are integral to modern infrastructure by providing a mechanism for funding road maintenance and expansion. The primary use case for tolling systems is to efficiently manage and collect tolls from vehicles using roadways. This involves roadside tracking, back-office accounting, and payment processing. However, the implementation of such systems is loaded with challenges.

Use Cases and Business Models for Tolling

Various business models have emerged to provide comprehensive tolling and payment solutions that integrate technology and data-driven strategies to optimize operations and revenue generation:

1. Roadside Tracking and Data Collection: At the core of modern tolling systems is the integration of IoT devices for roadside tracking. These devices capture essential data, such as vehicle identification, speed, and lane usage. This data is crucial for calculating tolls accurately and in real-time. The business model here involves deploying and maintaining a network of sensors and cameras that ensure seamless data collection across toll points.
2. Back-Office Accounting and Payment Processing: A robust back-office system is essential for processing toll transactions, managing accounts, and handling payments. This includes integrating with financial institutions for payment processing and ensuring compliance with financial regulations. The business model focuses on providing a secure and efficient platform for managing financial transactions, reducing administrative overhead, and enhancing customer satisfaction through streamlined payment processes.
3. Dynamic Pricing Models: To optimize revenue and manage traffic flow, tolling systems can implement dynamic pricing models. These models adjust toll rates based on real-time traffic conditions, time of day, and demand. By leveraging data analytics and machine learning, toll operators can predict traffic patterns and set prices that encourage optimal road usage. The business model here involves using data-driven insights to maximize revenue while minimizing congestion and improving the overall driving experience.
4. Interoperability and Cross-Agency Collaboration: Vehicles often travel across multiple tolling jurisdictions, causing interoperability between different toll agencies. Business models in this area focus on creating partnerships and agreements that allow for seamless data exchange and revenue sharing. This ensures that tolls are accurately attributed and collected, regardless of jurisdiction. This enhances the user experience and operational efficiency.
5. Subscription and Membership Models: Some tolling systems offer subscription or membership models that provide users with benefits such as discounted rates, priority access to express lanes, or bundled services. This business model aims to build customer loyalty and generate steady revenue streams by offering value-added services and personalized experiences.
6. Public-Private Partnerships (PPPs): Many tolling systems are developed and operated through collaborations. These leverage the strengths of both sectors, with the public sector providing regulatory oversight and the private sector offering technological expertise and investment. The business model focuses on sharing risks and rewards. This strategy ensures sustainable and efficient tolling operations.

Challenges of Traffic Systems

Intelligent tolling systems create lots of challenges for the project teams:

1. Integration with IoT Devices: Tolling systems rely heavily on IoT devices for roadside tracking. These devices generate vast amounts of data that need to be processed in real-time to ensure accurate toll collection.
2. Interoperability: Ensuring interoperability between different systems is crucial with vehicles crossing state lines and using multiple toll agencies.
3. Data Management: Managing and processing the data generated by IoT devices and various backend IT systems such as a CRM in a scalable and reliable manner is complex.
4. Static Pricing: Implementing innovative revenue-generating use cases such as dynamic pricing on express lanes requires real-time data processing to adjust toll rates based on current traffic conditions.

As you might expect, implementing and deploying intelligent tolling systems requires the use of modern, cloud-natives technologies. Conventional data integration and processing solutions, like databases, data lakes, ETL tools, or API platforms, lack the necessary capabilities. Therefore, data streaming becomes essential…

Data Streaming with Apache Kafka and Flink for Intelligent Traffic Systems

The ability to process data in real-time is crucial for ensuring efficient and accurate toll collection. Data streaming has emerged as a transformative technology that addresses the unique challenges faced by tolling systems, particularly in integrating IoT devices and implementing dynamic pricing models.

Apache Kafka became the de facto standard for data streaming. Apache Flink emerges as standard for stream processing. These technologies can help to implement tolling use cases:

1. Real-Time Toll Collection: Tolling systems rely on IoT devices to capture data from vehicles as they pass through toll points. This data includes vehicle identification, time of passage, and lane usage. Real-time processing of this data is essential to ensure that tolls are accurately calculated and collected without delay from IoT devices.
2. Dynamic Pricing Models: To optimize traffic flow and revenue, tolling systems can implement dynamic pricing models. These models adjust toll rates based on current traffic conditions, time of day, and other factors. Data streaming enables the continuous analysis of traffic data, allowing for real-time adjustments to pricing.
3. Interoperability Across Agencies: Vehicles often travel across multiple tolling jurisdictions, requiring seamless interoperability between different toll agencies. Data streaming facilitates the real-time exchange of data between agencies, ensuring that tolls are accurately attributed and collected regardless of jurisdiction.

Key Benefits of Data Streaming with Apache Kafka and Flink

Data streaming with Kafka and Flink makes a tremendous difference for building a next generation traffic system:

• Real-Time Processing: Data streaming technologies like Apache Kafka and Apache Flink enable the real-time processing of data from IoT devices. Kafka acts as the backbone for data ingestion, capturing and storing streams of data from roadside sensors and devices. Flink provides the capability to process and analyze these data streams in real-time, ensuring that tolls are calculated and collected accurately and promptly.
• Scalability: Tolling systems must handle large volumes of data, especially during peak traffic hours. Kafka’s distributed architecture allows it to scale horizontally, accommodating the growing data demands of expanding traffic networks. This scalability ensures that the system can handle increased data loads without compromising performance.
• Reliability: Kafka’s robust architecture provides a reliable mechanism for tracking and processing data. It ensures that every message from IoT devices is captured and processed, reducing the risk of errors in toll collection. Kafka’s ability to replay messages also allows for recovery from potential data loss, ensuring data integrity.
• Flexibility: By decoupling data processing from the underlying infrastructure, data streaming offers the flexibility to adapt to changing business needs. Kafka’s integration capabilities allow it to connect with various data sources and sinks. Flink’s stream processing capabilities enable complex event processing and real-time analytics. This flexibility allows tolling systems to develop and incorporate new technologies and business models as needed.

Quarterhill – Tolling and Enforcement with Dynamic Pricing

Quarterhill is a company that specializes in intelligent traffic systems, focusing on two main areas: tolling and safety/inspection. The company provides comprehensive solutions for managing tolling systems, which include roadside tracking, back-office accounting, and payment processing.

By integrating IoT devices and leveraging data streaming technologies, Quarterhill optimizes toll collection processes, implements dynamic pricing models, and ensures interoperability across different toll agencies to optimize revenue generation while ensuring smooth traffic flow.

I had the pleasure of doing a panel conversation with Josh LittleSun, VP Delivery of Quarterhill at Confluent’s Data in Motion Tour Chicago 2024.

Quarterhill’s Product Portfolio

Quarterhill’s product portfolio encompasses a comprehensive range of solutions designed to enhance traffic management and transportation systems. As you can see, many of these products are inherently designed for data streaming.

• Roadside technologies include tools for congestion charging, performance management, insights and analytics, processing systems, and lane configuration, all aimed at optimizing road usage and efficiency.
• Commerce and mobility platforms offer analytics, toll interoperability, a mobility marketplace, back-office solutions, and performance management, facilitating seamless transactions and mobility services.
• Safety and enforcement solutions focus on ensuring compliance and safety for commercial vehicles, with features like maintenance, e-screening, tire anomaly detection, weight compliance, and commercial roadside technologies.
• Smart Transportation solutions provide multi-modal data and intersection management, improving the coordination and flow of various transportation modes.
• Data Solutions feature video-based systems, traffic recording systems, in-road sensor systems, and cloud-based solutions, offering advanced data collection and analysis capabilities for informed decision-making and maintenance.

How Quarterhill Built an Intelligent Traffic System in the Cloud with Data Streaming and IoT

Quarterhill’s journey towards building an intelligent traffic system began with the realization that traditional monolithic architectures did not meet the demands of modern tolling systems. The company embarked on a transformation journey, moving from monolith to microservices and adopting data streaming as a core component of their architecture.

Key Components of Quarterhill’s Intelligent Traffic System

1. Fully Managed Confluent Cloud on GCP: By leveraging Confluent Cloud on Google Cloud Platform (GCP) as its data streaming platform, Quarterhill could focus on solving business problems rather than managing infrastructure. This shift allowed for greater agility and reduced operational overhead.
2. Data Streaming Instead of Google Pub/Sub: Quarterhill chose data streaming over Google Pub/Sub because of its ability to provide various use cases beyond ingestion into the data lake, including real-time processing of transactional workloads and integration with IoT devices.
3. Direct Connection to the Cloud via MQTT, HTTP, and Connectors: IoT devices connect directly to Kafka using protocols like MQTT and HTTP. Connectors facilitate data integration and processing.
4. Edge Servers for Data Aggregation: In some cases, edge servers are used to aggregate data before sending it to the cloud. This option optimizes bandwidth usage and ensuring low-latency processing.
5. Consumers: Elastic, BigQuery, Custom Connectors: Data is consumed by various systems, including Elastic for search and analytics, Google BigQuery for data warehousing, and custom connectors for specific use cases.

Benefits Realized with a Fully Managed Data Streaming Platform

• Elasticity and high throughput: The ability to scale with traffic volume ensures that tolling systems can handle peak loads without degradation in performance.
• Resiliency and accuracy: The reliability of data streaming ensures that toll collection is accurate and resilient to failures.
• Cost savings and efficiency: By moving to a fully managed cloud solution for the data streaming platform with Confluent Cloud, Quarterhill achieved significant cost savings (TCO) and reduced the demand for in-house resources.

DKV Mobility: On-the-Road Payments and Solutions

DKV Mobility stands as a leading European B2B platform specializing in on-the-road payments and solutions. With a robust customer base of over 300,000 active clients spanning over 50 service countries, DKV Mobility has revolutionized the way businesses manage their on-the-road expenses. The platform enables real-time payments and transaction processing, providing valuable insights for businesses on the move.

DKV Mobility’s comprehensive services cover a wide range of needs, including refueling, electric vehicle (EV) charging, toll solutions, and vehicle services. The platform supports approximately 468,000 EV charge points, 63,000 fuel service stations, and 30,000 vehicle service stations, ensuring that businesses have access to essential services wherever they operate. Through its innovative solutions, DKV Mobility enhances operational efficiency and cost management for businesses across Europe.

If you are interested in how DKV Mobility transitioned from open source Kafka to fully managed SaaS and how they leverage stream processing with Kafka Streams, check out the DKV Mobility success story.

IoT Connectivity with MQTT and HTTP + Data Streaming with Apache Kafka = Next Generation Traffic System

Quarterhill’s intelligent traffic system for tolls and DKV Mobility’s real time on-the-road payment solution exemplify the transformative power of a data streaming platform using Apache Kafka in modern infrastructure to solve a specific business problem. Related scenarios, such as logistics and supply chain, can benefit from such a foundation and connect to existing data products for new business models or B2B data exchanges with partners.

By embracing a cloud-native, microservices-based architecture, Quarterhill and DKV Mobility have not only overcome the challenges of traditional tolling and payment systems but have also set a new standard for efficiency and innovation in the industry. Use cases such as IoT sensor integration and dynamic pricing are only possible with data streaming.

As these companies continue to leverage stream processing with Kafka Streams and explore new technologies like Apache Flink and data governance solutions, the future of intelligent traffic systems looks promising. The potential is huge to further enhance safety, efficiency of payments and customer experiences, and revenue generation on roadways.

How do you leverage data streaming in your enterprise architecture? How do you connect to IoT interfaces? What is your data processing strategy? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.