Browsing Tag

middleware

40 posts
Apache Flink - Overkill for Simple Stateless Stream Processing
Read More
  • 1.6K views
  • 8 minute read

Apache Flink: Overkill for Simple, Stateless Stream Processing and ETL?

Discover when Apache Flink is the right tool for your stream processing needs. Explore its role in stateful and stateless processing, the advantages of serverless Flink SaaS solutions like Confluent Cloud, and how it supports advanced analytics and real-time data integration together with Apache Kafka. Dive into the trade-offs, deployment options, and strategies for leveraging Flink effectively across cloud, on-premise, and edge environments, and when to use Kafka Streams or Single Message Transforms (SMT) within Kafka Connect for ETL instead of Flink.
Read More
0
0
0
Virgin Australia Journey with Apache Kafka - Innovation in the Airline and Aviation Industry
Read More
  • 1.1K views
  • 5 minute read

Virgin Australia’s Journey with Apache Kafka: Driving Innovation in the Airline Industry

Data streaming with Apache Kafka and Flink is transforming the airline industry, enabling real-time efficiency and exceptional customer experiences. Virgin Australia exemplifies this innovation to modernize its Flight State Engine and overhaul its loyalty program. By embracing event-driven architecture, the airline has improved operational reliability and personalized services, setting a benchmark for aviation digitalization.
Read More
0
0
0
Snowflake with Apache Kafka and Iceberg Connector
Read More
  • 5.8K views
  • 8 minute read

Snowflake Data Integration Options for Apache Kafka (including Iceberg)

The integration between Apache Kafka and Snowflake is often cumbersome. Options include near real-time ingestion with a Kafka Connect connector, batch ingestion from large files, or leveraging a standard table format like Apache Iceberg. This blog post explores the alternatives and discusses its trade-offs. The end shows how data streaming helps with hybrid architectures where data needs to be ingested from the private data center into Snowflake in the public cloud.
Read More
0
0
0
Top Use Cases and Architectures for Data Streaming with Apache Kafka in 2023
Read More
  • 6.1K views
  • 7 minute read

Top 5 Data Streaming Trends for 2023

Data Streaming is one of the most relevant buzzwords in tech to build scalable real-time applications in the cloud and innovative business models. Do you wonder about my predicted TOP 5 data streaming trends in 2023 to set data in motion? Check out the following presentation and learn what role Apache Kafka plays. Learn about decentralized Data Mesh, cloud-native lakehouse, data sharing, improved user experience, and advanced data governance.
Read More
0
0
0
Request Response Data Exchange with Apache Kafka vs CQRS and Event Sourcing
Read More
  • 31.9K views
  • 10 minute read

When to use Request-Response with Apache Kafka?

How can I do request-response communication with Apache Kafka? That’s one of the most common questions I get regularly. This blog post explores when (not) to use this message exchange pattern, the differences between synchronous and asynchronous communication, the pros and cons compared to CQRS and event sourcing, and how to implement request-response within the data streaming infrastructure.
Read More
0
0
1
How to do Error Handling in Data Streaming
Read More
  • 83.7K views
  • 15 minute read

Error Handling via Dead Letter Queue in Apache Kafka

Recognizing and handling errors is essential for any reliable data streaming pipeline. This blog post explores best practices for implementing error handling using a Dead Letter Queue in Apache Kafka infrastructure. The options include a custom implementation, Kafka Streams, Kafka Connect, the Spring framework, and the Parallel Consumer. Real-world case studies show how Uber, CrowdStrike, Santander Bank, and Robinhood build reliable real-time error handling at an extreme scale.
Read More
0
0
0
JMS Message Queue vs Apache Kafka Comparison
Read More
  • 16.8K views
  • 19 minute read

Comparison: JMS Message Queue vs. Apache Kafka

Comparing JMS-based message queue (MQ) infrastructures and Apache Kafka-based data streaming is a widespread topic. Unfortunately, the battle is an apple-to-orange comparison that often includes misinformation and FUD from vendors. This blog post explores the differences, trade-offs, and architectures of JMS message brokers and Kafka deployments. Learn how to choose between JMS brokers like IBM MQ or RabbitMQ and open-source Kafka or serverless cloud services like Confluent Cloud.
Read More
3
0
0
Disaster Recovery and Resiliency with Apache Kafka Data Streaming
Read More
  • 3.2K views
  • 3 minute read

Disaster Recovery with Kafka across the Edge and Hybrid Cloud (QCon Talk)

I spoke at QCon London in April 2022 about building disaster recovery and resilient real-time enterprise architectures with Apache Kafka. This blog post summarizes the use cases, architectures, and real-world examples. The slide deck and video recording of the presentation is included as well.
Read More
0
0
0
Streaming ETL with Apache Kafka in Healthcare
Read More
  • 5.1K views
  • 4 minute read

Streaming ETL with Apache Kafka in the Healthcare Industry

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. This is part three: Streaming ETL. Examples include Babylon Health and Bayer.
Read More
0
0
0
Data Streaming with Apache Kafka in the Healthcare Industry
Read More
  • 5.6K views
  • 4 minute read

Apache Kafka in the Healthcare Industry

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores real-world examples of data streaming with Apache Kafka to increase efficiency, reduce cost, and improve the human experience across the healthcare value chain including pharma, insurance, providers, retail, and manufacturing. This is part one: Overview.
Read More
0
0
0