When (Not) to Choose Google Managed Service for Apache Kafka?

Qualifying out a technology is often the easier option. Why evaluate a service if it does not solve the requirements? Let’s explore when NOT to use Kafka at all, and specifically when the Google Apache Kafka service is probably NOT the right choice for you.

When NOT to use Apache Kafka?

Apache Kafka has overlaps with technologies like a message broker (like IBM MQ, TIBCO or RabbitMQ), other streaming analytics platforms, and it actually is a database, too. But Apache Kafka is not an allrounder to solve every problem.

Apache Kafka is NOT:

• A replacement for your favorite database, data warehouse or data lake. Instead it complements and integrates with these platforms.
• An analytics platform for AI/ML model training, even though model scoring is often done within the streaming platform for critical or low-latency use cases.
• A proxy for thousands of clients in bad networks.
• An API Management solution, even though you can connect REST/HTTP producers and consumers against Kafka.
• An IoT gateway, even though direct integration with IoT protocols like MQTT or OPC-UA is possible.
• Hard real-time for safety-critical embedded workloads.

Read the thorough analysis “When NOT to use Apache Kafka?” for more details. Or watch this YouTube video:

When to Choose ANOTHER Kafka instead of Google’s?

If Apache Kafka is the right choice for your project, you still have plenty of options.

Here are a few criteria that let you easily disqualify out Google Cloud Managed Service for Apache Kafka (MSK):

• Non-GCP: If your use case requires on-premise, multi-cloud, hybrid cloud or edge deployments, then you need another offering.
• Critical SLAs: If you need 24/7 critical support and consulting expertise, a dedicated Kafka vendor like Confluent is the better choice. Kafka is not just for analytics, but shines for transactional workloads, too. Google’s Managed Apache Kafka service is not GA yet. This will probably happen in the second half of 2024. Hence, don’t even consider it for critical applications before GA.
• Serverless: A managed service is not always a truly managed service. The future will show where Google goes with Kafka. But right now, Google Apache Kafka is not serverless like e.g., Confluent Cloud. You pay for capacity pricing and cluster capacity management is required. Amazon even created a second service Amazon MSK Serverless to handle this issue with its traditional MSK offering. Learn why Amazon MSK is NOT always the right choice and adopt the learnings to Google.
• Complete Data Streaming Platform: A data streaming platform requires more than just messaging: data integration with first and third party systems, stream processing for continuous data correlation, flexible (long-term) retention with Tiered Storage, data governance, and more. The future will show us where Google’s Kafka service goes. A detailed article compares Kafka offerings as car engine, complete car and self-driving car level 5. Google is a car, but not (yet) a Porsche (complete luxury car) and not yet a Google Waymo (self-driving car level 5). Google Apache Kafka even misses basic features for data streaming best practices, like defining data contracts in schemas for building data products in good data quality.

The Evolution of Data Streaming is NOT Stopping…

If you did not qualify out Kafka in general or Google Apache Kafka in particular yet, that’s great. Start evaluating Google Cloud’s Managed Service for Apache Kafka (MSK) service and compare it against self-managed open source Kafka and other semi-managed or fully-managed Kafka cloud services on GCP.

As we look ahead, the future possibilities for data streaming are boundless, promising more agile, intelligent, and real-time insights into the ever-increasing streams of data. I recently looked at the past, present and future of stream processing.

I often get the question if I am worried about the emerging competition as I work for Confluent where we “only do data streaming”?

No, I am not! Actually, the new Google Apache Kafka cloud service is great news for the industry! Data Streaming established itself as a new software category. Research analysts like Forrester and IDG already created dedicated waves and comparisons. What could be better than working with the people that invented Kafka and the company that created this software category across all industries and continents? And competition is always good for innovation, too.

Real-time data beats slow data. That’s true in almost every use case. At Confluent, we are now ~3000 people working only on one thing: Data Streaming. I think we should celebrate this Google announcement and look forward to more mass adoption of data streaming around the world.

And as a strategic Google partner, customers can

1. leverage GCP credits to consume Confluent Cloud
2. leverage GCPs security and private networking infrastructure
3. integrate via fully managed connectors into various GCP services like Google Big Query or Google Cloud Storage and third party cloud solutions like MongoDB, Snowflake or Databricks.

Are you excited about the new Google Cloud Managed Service for Apache Kafka (MSK) service? Or do you use still plan to use open source Kafka or another cloud service like Confluent Cloud? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.