Middleware Archives - Kai Waehner

Industrial IoT Middleware OT IT Bridge between Edge and Cloud with Apache Kafka and Flink

1.2K views
11 minute read

Industrial IoT Middleware for Edge and Cloud OT/IT Bridge powered by Apache Kafka and Flink

ByKai Waehner
20. September 2024
No comments

As industries continue to adopt digital transformation, the convergence of Operational Technology (OT) and Information Technology (IT) has become essential. The OT/IT Bridge is a key concept in industrial automation to connect real-time operational processes with business-oriented IT systems ensuring seamless data flow and coordination. By leveraging Industrial IoT middleware and data streaming technologies like Apache Kafka and Flink, businesses can achieve a unified approach to managing both production processes and higher-level business operations to drive greater efficiency, predictive maintenance, and streamlined decision-making.

20.0K views
12 minute read

The Past, Present and Future of Stream Processing

ByKai Waehner
20. March 2024
No comments

Stream processing has existed for decades. The adoption grows with open source frameworks like Apache Kafka and Flink in combination with fully managed cloud services. This blog post explores the past, present and future of stream processing, including the relation of machine learning and GenAI, streaming databases, and the integration between data streaming and data lakes with Apache Iceberg.

Data Streaming with Apache Kafka at Airlines - Lufthansa Case Study

5.7K views
5 minute read

How Lufthansa uses Apache Kafka for Middleware and Analytics

ByKai Waehner
24. September 2023
No comments

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. The coronavirus was just a piece of the challenge. This post explores how Lufthansa leverages data streaming powered by Apache Kafka as cloud-native middleware for mission-critical data integration projects and as data fabric for AI/machine learning scenarios such as real-time predictions in fleet management. An interactive conversation with Lufthansa as an on-demand video is added at the end as a highlight if you want to learn more.

Event Streaming as the Data Hub for Crypto NFT and Metaverse

4.1K views
19 minute read

Apache Kafka as Data Hub for Crypto, DeFi, NFT, Metaverse – Beyond the Buzz

ByKai Waehner
4. February 2022
One comment

Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This blog post explores the relationship between crypto technologies and modern enterprise architecture. I discuss how event streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let’s skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.

16.3K views
13 minute read

When to use Apache Camel vs. Apache Kafka?

ByKai Waehner
28. January 2022
4 shares
4 comments

Should I use Apache Camel or Apache Kafka for my next integration project? The question is very valid and comes up regularly. This blog post explores both open-source frameworks and explains the difference between application integration and event streaming. The comparison discusses when to use Kafka or Camel, when to combine them, when not to use them at all. A decision tree shows how you can quickly qualify out one for the other.

Stream Exchange for Data Sharing with Apache Kafka in a Data Mesh

8.7K views
10 minute read

Streaming Data Exchange with Kafka and a Data Mesh in Motion

ByKai Waehner
14. November 2021
No comments

Data Mesh is a new architecture paradigm that gets a lot of buzzes these days. This blog post looks into this principle deeper to explore why no single technology is the perfect fit to build a Data Mesh. Examples show why an open and scalable decentralized real-time platform like Apache Kafka is often the heart of the Data Mesh infrastructure, complemented by many other data platforms to solve business problems.

Apache Kafka for Government and Citizen Services in the Public Sector

2.1K views
5 minute read

Apache Kafka in the Public Sector – Part 3: Government and Citizen Services

ByKai Waehner
15. October 2021
No comments

The public sector includes many different areas. Some groups leverage cutting-edge technology, like military leverage. Others like the public administration are years or even decades behind. This blog series explores both edges to show how data in motion powered by Apache Kafka adds value for innovative new applications and modernizing legacy IT infrastructures. This is part 3: Use cases and architectures for Government and Citizen Services.

Innovation in Financial Services and Open Banking with Apache Kafka

8.2K views
5 minute read

Apache Kafka in the Financial Services Industry

ByKai Waehner
18. January 2021
No comments

The rise of event streaming in financial services is growing like crazy. Continuous real-time data integration and processing are mandatory for many use cases. Apache Kafka is deployed across the financial services business departments for mission-critical transactional workloads and big data analytics. High scalability, high reliability, and an elastic open infrastructure are the key reasons for the success of Kafka. This blog post explores different use cases, architectures, and real-world examples in the FinServ sector.