Browsing Tag

Kafka Connect

29 posts
Snowflake with Apache Kafka and Iceberg Connector
Read More
  • 4.2K views
  • 8 minute read

Snowflake Data Integration Options for Apache Kafka (including Iceberg)

The integration between Apache Kafka and Snowflake is often cumbersome. Options include near real-time ingestion with a Kafka Connect connector, batch ingestion from large files, or leveraging a standard table format like Apache Iceberg. This blog post explores the alternatives and discusses its trade-offs. The end shows how data streaming helps with hybrid architectures where data needs to be ingested from the private data center into Snowflake in the public cloud.
Read More
0
0
0
GenAI Demo with Kafka, Flink, LangChain and OpenAI
Read More
  • 9.1K views
  • 10 minute read

GenAI Demo with Kafka, Flink, LangChain and OpenAI

Generative AI (GenAI) enables automation and innovation across industries. This blog post explores a simple but powerful architecture and demo for the combination of Python, and LangChain with OpenAI LLM, Apache Kafka for event streaming and data integration, and Apache Flink for stream processing. The use case shows how data streaming and GenAI help to correlate data from Salesforce CRM, searching for lead information in public datasets like Google and LinkedIn, and recommending ice-breaker conversations for sales reps.
Read More
0
0
0
Apache Kafka vs Redpanda Comparison
Read More
  • 19.4K views
  • 18 minute read

When to choose Redpanda instead of Apache Kafka?

Data streaming emerged as a new software category. It complements traditional middleware, data warehouse, and data lakes. Apache Kafka became the de facto standard. New players enter the market because of Kafka’s success. One of those is Redpanda, a lightweight Kafka-compatible C++ implementation. This blog post explores the differences between Apache Kafka and Redpanda, when to choose which framework, and how the Kafka ecosystem, licensing, and community adoption impact a proper evaluation.
Read More
0
0
0
Disaster Recovery and Resiliency with Apache Kafka Data Streaming
Read More
  • 3.0K views
  • 3 minute read

Disaster Recovery with Kafka across the Edge and Hybrid Cloud (QCon Talk)

I spoke at QCon London in April 2022 about building disaster recovery and resilient real-time enterprise architectures with Apache Kafka. This blog post summarizes the use cases, architectures, and real-world examples. The slide deck and video recording of the presentation is included as well.
Read More
0
0
0
Streaming ETL with Apache Kafka in Healthcare
Read More
  • 4.8K views
  • 4 minute read

Streaming ETL with Apache Kafka in the Healthcare Industry

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. This is part three: Streaming ETL. Examples include Babylon Health and Bayer.
Read More
0
0
0
Apache Kafka in the Insurance Industry
Read More
  • 8.4K views
  • 7 minute read

Apache Kafka in the Insurance Industry

The rise of data in motion in the insurance industry is visible across all lines of business including life, healthcare, travel, vehicle, and others. Apache Kafka changes how enterprises rethink data. This blog post explores use cases and architectures for insurance-related event streaming. Real-world examples from Generali, Centene, Humana, and Tesla show innovative data integration and stream processing in real-time.  
Read More
1
0
2
Apache Kafka in Aviation Industry including Airlines Airports Manufacturing Retail GDS
Read More
  • 6.2K views
  • 8 minute read

Apache Kafka in the Airline, Aviation and Travel Industry

Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. Coronavirus is just a piece of the challenge. This post explores use cases, architectures, and references for Apache Kafka in the aviation industry, including airline, airports, global distribution systems (GDS), aircraft manufacturers, and more.
Read More
2
0
2
Supply Chain Optimization with Apache Kafka and SCM
Read More
  • 4.6K views
  • 8 minute read

Apache Kafka for Supply Chain Management (SCM) Optimization

Supply Chain optimization leveraging Event Streaming with Apache Kafka. See real-world use cases and architectures from Walmart, BMW, Porsche, and other enterprises to improve the Supply Chain Management (SCM) processes. Automation, robustness, flexibility, real-time, decoupling, data integration, and hybrid deployments…
Read More
1
0
0