Kai WaehnerApache Kafka and Enterprise Service Bus (ESB) are complementary, not competitive!
Apache Kafka is much more than messaging in the meantime. It evolved to a streaming platform including Kafka Connect, Kafka Streams, KSQL and many other open source components. Kafka leverages events as a core principle. You think in data flows of events and process the data while it is in motion. Many concepts, such as event sourcing, or design patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture.
Kafka is unique because it combines messaging, storage, and processing of events all in one platform. It does this in a distributed architecture using a distributed commit log and topics divided into multiple partitions.
ETL and ESB have excellent tooling, including graphical mappings for doing complex integration with legacy systems and technologies such as SOAP, EDIFACT, SAP BAPI, COBOL, etc. (Trust me, you don’t want to write the code for this.)
Therefore, existing MQ and ESB solutions, which already integrate with your legacy world, are not competitive to Apache Kafka. Rather, they are complementary!
Read more details about this question in my Confluent blog post:
Apache Kafka® vs. Enterprise Service Bus (EBS) | Confluent
As always, I appreciate any feedback, comments or criticism.
2 comments
Hello Kai,
Thanks for this post. I am still slightly confused (not because of your article though) why Kafka isn’t an “Enhanced/Improved ESB” vs. Traditional ESB? I suppose you could use Kafka as ESB, BUT it’s an anti-pattern and Kafka is meant to decentralise ESB-like architecture..
I understand the confusion. This is a tough topic 🙂 The short answer is that an ESB requires to implement all the integration logic in the central ESB infrastructure. Contrarily, Kafka decouples the systems and the integration logic can be built in each domain/application/service. Check out the article “Microservices, Apache Kafka, and Domain-Driven Design (DDD)” for more details about this discussion.