Browsing Tag

Data Integration

9 posts
Apache Flink - Overkill for Simple Stateless Stream Processing
Read More
  • 2.7K views
  • 8 minute read

Apache Flink: Overkill for Simple, Stateless Stream Processing and ETL?

Discover when Apache Flink is the right tool for your stream processing needs. Explore its role in stateful and stateless processing, the advantages of serverless Flink SaaS solutions like Confluent Cloud, and how it supports advanced analytics and real-time data integration together with Apache Kafka. Dive into the trade-offs, deployment options, and strategies for leveraging Flink effectively across cloud, on-premise, and edge environments, and when to use Kafka Streams or Single Message Transforms (SMT) within Kafka Connect for ETL instead of Flink.
Read More
0
0
0
Virgin Australia Journey with Apache Kafka - Innovation in the Airline and Aviation Industry
Read More
  • 1.6K views
  • 5 minute read

Virgin Australia’s Journey with Apache Kafka: Driving Innovation in the Airline Industry

Data streaming with Apache Kafka and Flink is transforming the airline industry, enabling real-time efficiency and exceptional customer experiences. Virgin Australia exemplifies this innovation to modernize its Flight State Engine and overhaul its loyalty program. By embracing event-driven architecture, the airline has improved operational reliability and personalized services, setting a benchmark for aviation digitalization.
Read More
0
0
0
Airport and Airlines Digitalization with Data Streaming using Apache Kafka and Flink
Read More
  • 4.6K views
  • 10 minute read

The Digitalization of Airport and Airlines with IoT and Data Streaming using Kafka and Flink

The vision for a digitalized airport includes seamless passenger experiences, optimized operations, consistent integration with airlines and retail stores, and enhanced security through the use of advanced technologies like IoT, AI, and real-time data analytics. This blog post shows the relevance of data streaming with Apache Kafka and Flink in the aviation industry to enable data-driven business process automation and innovation while modernizing the IT infrastructure with cloud-native hybrid cloud architecture.
Read More
0
0
0
Snowflake and Apache Kafka Data Integration Anti Patterns Zero Reverse ETL
Read More
  • 5.3K views
  • 9 minute read

Snowflake Integration Patterns: Zero ETL and Reverse ETL vs. Apache Kafka

Snowflake is a leading cloud-native data warehouse. Integration patterns include batch data integration, Zero ETL and near real-time data ingestion with Apache Kafka. This blog post explores the different approaches and discovers its trade-offs. Following industry recommendations, it is suggested to avoid anti-patterns like Reverse ETL and instead use data streaming to enhance the flexibility, scalability, and maintainability of enterprise architecture.
Read More
0
0
0
Read More
  • 23.6K views
  • 12 minute read

Apache Kafka + Vector Database + LLM = Real-Time GenAI

Generative AI (GenAI) enables advanced AI use cases and innovation but also changes how the enterprise architecture looks like. Large Language Models (LLM), Vector Databases, and Retrieval Augmentation Generation (RAG) require new data integration patterns. Data streaming with Apache Kafka and Apache Flink processes incoming data sets in real-time at scale, connects various platforms, and enables decoupled data products.
Read More
0
0
3
Data Streaming with Apache Kafka at Airlines - Lufthansa Case Study
Read More
  • 6.8K views
  • 5 minute read

How Lufthansa uses Apache Kafka for Middleware and Analytics

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. The coronavirus was just a piece of the challenge. This post explores how Lufthansa leverages data streaming powered by Apache Kafka as cloud-native middleware for mission-critical data integration projects and as data fabric for AI/machine learning scenarios such as real-time predictions in fleet management. An interactive conversation with Lufthansa as an on-demand video is added at the end as a highlight if you want to learn more.
Read More
0
0
0
Disaster Recovery and Resiliency with Apache Kafka Data Streaming
Read More
  • 3.4K views
  • 3 minute read

Disaster Recovery with Kafka across the Edge and Hybrid Cloud (QCon Talk)

I spoke at QCon London in April 2022 about building disaster recovery and resilient real-time enterprise architectures with Apache Kafka. This blog post summarizes the use cases, architectures, and real-world examples. The slide deck and video recording of the presentation is included as well.
Read More
0
0
0
Reverse ETL Anti Pattern vs Event Streaming with Apache Kafka
Read More
  • 7.7K views
  • 9 minute read

When to Use Reverse ETL and when it is an Anti-Pattern

This blog post explores why software vendors (try to) introduce new solutions for Reverse ETL, when Reverse ETL is really needed, and how it fits into the enterprise architecture. The involvement of event streaming to process data in motion is a key piece of Reverse ETL for real-time use cases.
Read More
1
0
0
Cyber Threat Intelligence with Apache Kafka and SIEM SOAR Machine Learning
Read More
  • 3.6K views
  • 6 minute read

Kafka for Cybersecurity (Part 3 of 6) – Cyber Threat Intelligence

This blog series explores use cases and architectures for Apache Kafka in the cybersecurity space, including situational awareness, threat intelligence, forensics, air-gapped and zero trust environments, and SIEM / SOAR modernization. This post is part three: Cyber Threat Intelligence.
Read More
0
0
0