Kai Waehner

314 posts
builds cloud-native event streaming infrastructures for real-time data processing and analytics
Apache Kafka Confluent for Data Ingestion into Snowflake Databricks BigQuery Data Warehouse
Read More
  • 10.7K views
  • 10 minute read

Data Streaming for Data Ingestion into the Data Warehouse and Data Lake

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 2: Data Streaming for Data Ingestion into the Data Warehouse and Data Lake.
Read More
0
0
0
Data Warehouse vs Data Lake vs Data Streaming Comparison
Read More
  • 8.4K views
  • 9 minute read

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 1: Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Read More
0
0
0
Request Response Data Exchange with Apache Kafka vs CQRS and Event Sourcing
Read More
  • 25.1K views
  • 10 minute read

When to use Request-Response with Apache Kafka?

How can I do request-response communication with Apache Kafka? That’s one of the most common questions I get regularly. This blog post explores when (not) to use this message exchange pattern, the differences between synchronous and asynchronous communication, the pros and cons compared to CQRS and event sourcing, and how to implement request-response within the data streaming infrastructure.
Read More
0
0
1
How to do Error Handling in Data Streaming
Read More
  • 63.3K views
  • 15 minute read

Error Handling via Dead Letter Queue in Apache Kafka

Recognizing and handling errors is essential for any reliable data streaming pipeline. This blog post explores best practices for implementing error handling using a Dead Letter Queue in Apache Kafka infrastructure. The options include a custom implementation, Kafka Streams, Kafka Connect, the Spring framework, and the Parallel Consumer. Real-world case studies show how Uber, CrowdStrike, Santander Bank, and Robinhood build reliable real-time error handling at an extreme scale.
Read More
0
0
0
Kafka in Healthcare - Open API and Omnichannel Data Streaming
Read More
  • 1.9K views
  • 4 minute read

Open API and Omnichannel with Apache Kafka in Healthcare

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Open API and Omnichannel. Examples include Care.com and Invitae.
Read More
0
0
0
JMS Message Queue vs Apache Kafka Comparison
Read More
  • 12.4K views
  • 19 minute read

Comparison: JMS Message Queue vs. Apache Kafka

Comparing JMS-based message queue (MQ) infrastructures and Apache Kafka-based data streaming is a widespread topic. Unfortunately, the battle is an apple-to-orange comparison that often includes misinformation and FUD from vendors. This blog post explores the differences, trade-offs, and architectures of JMS message brokers and Kafka deployments. Learn how to choose between JMS brokers like IBM MQ or RabbitMQ and open-source Kafka or serverless cloud services like Confluent Cloud.
Read More
0
0
0
Cybersecurity in Crypto and FinTech with Data Streaming and Apache Kafka
Read More
  • 3.6K views
  • 5 minute read

Apache Kafka in Crypto and FinServ for Cybersecurity and Fraud Detection

The insane growth of the crypto and fintech market brings many unknown risks and successful cyberattacks to steal money and crypto coins. This post explores how data streaming with the Apache Kafka ecosystem enables real-time situational awareness and threat intelligence to detect and prevent hacks, money loss, and data breaches. Enterprises stay compliant with the law and keep customers happy in any innovative Fintech or Crypto application.
Read More
0
0
0
Machine Learning and Data Science with Apache Kafka in Healthcare
Read More
  • 3.3K views
  • 4 minute read

Machine Learning and Data Science with Kafka in Healthcare

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Machine Learning and Data Science. Examples include Recursion and Humana.
Read More
0
0
0
Disaster Recovery and Resiliency with Apache Kafka Data Streaming
Read More
  • 2.6K views
  • 3 minute read

Disaster Recovery with Kafka across the Edge and Hybrid Cloud (QCon Talk)

I spoke at QCon London in April 2022 about building disaster recovery and resilient real-time enterprise architectures with Apache Kafka. This blog post summarizes the use cases, architectures, and real-world examples. The slide deck and video recording of the presentation is included as well.
Read More
0
0
0
Real Time Analytics and Machine Learning with Apache Kafka in Healthcare
Read More
  • 3.2K views
  • 4 minute read

Real Time Analytics with Apache Kafka in the Healthcare Industry

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. This is part four: Real-Time Analytics. Examples include Cerner, Celmatix, CDC/Centers for Disease Control and Prevention.
Read More
0
0
0