Apache Kafka for Telco-OTT (Telecom Sector) and Media Applications

Current IT architectures in the telecom and media sector are not able to satisfy business needs because of their high complexity, lack of flexibility, and low level of automation. The biggest hurdle to overcome with digital transformation is to understand that it isn’t just a simple technology challenge – it covers every part of the telco business! This blog post explores next-generation architectures for the telecom and media sector with the Apache Kafka ecosystem to build Telco-OTT (Over the Top) services.

Apache Kafka in the Telecom Sector (OSS, BSS, Middleware, OTT)

A previous blog post covered the use cases of Event Streaming and Apache Kafka in the telecom industry: “Event Streaming and Apache Kafka in Telco Industry“. Follow that post or the related “whiteboard on Youtube about Apache Kafka and the telecom sector” to learn about various use cases like the following:

Telco-OTT (Over-The-Top) – 3rd Party Telecom and Media Services

Telco-OTT (Over-The-Top, OTT) is a particular field in the overall architecture in Telco and Media enterprises:

• OSS (Operations Support System), as part of the telecommunication infrastructure, ensures services are working.
• BSS (Business Support System), as part of the telecommunication infrastructure, ensures that they can be provided to an actual customer
• OTT Applications use the existing telecommunication infrastructure and provide better cost and/or features and/or convenience.

An example is the easiest way to understand this better.

OTT Applications for Messaging and Chat (SMS, RCS, WhatsApp, WeChat)

Messaging applications are a great example of Telco-OTT. Easy to understand for everybody, but pretty powerful tools.

Here are some examples of messaging apps:

• SMS (short message service) by telco providers: Text format message.
• WhatsApp by Facebook: Above + group chat, gif/stickers, photos, videos, audio, location, contact information, and ‘walkie-talkie’ services.
• WeChat by Tencent: Above + payment + various other partner integrations

The telecom sector had huge revenue coming from simple text messages via SMS. No competition existed for a long time. Telcos were able to charge ~20 Cent per message. A billion-dollar business emerged.

The internet and app economy changed this massively:

People-to-People (P2P) SMS shrunk significantly. Application-to-people (A2P) is lower today, too.

The telecom sector has the following options to act when a competing service becomes successful:

• Partnership: No control over the service and potentially damage their reputation and customer relationships.
• Development of own services: No skills for building the service and too late anyway.
• Blocking OTT services: Losing revenue for traffic and customers.

Not good options, right? The best answer is to innovate early; before competitors take over.

In this case, the answer from the telco providers was RCS (rich communication services)… RCS enabled additional features to simple text, such as sending images and videos. But it came “a little bit late” and was much more expensive compared to “almost free” apps like WhatsApp and WeChat. (yes, I know, if the product is free, then you are the product – but this is another discussion…)

Next-Generation Telecom Architecture

The above example shows the main problem in the Telecommunications Industry. The telecom sector needs to completely change their strategy and overall infrastructure to stay competitive and innovative. This change is the only way to “bridge the gap with the Over-the-Top (OTT) internet providers and survive in the digital age“:

Microservices Architecture for the Telecom Sector to Compete with 3rd Party OTT Applications

The telco infrastructure grew over the decades. It is inflexible, monolithic, and complex. The trend at most telco companies (I talked to in the last 12 months across the world) only sees one direction for the future: Build a flexible, open, scalable Telco architecture to process massive volumes of data in real-time.

The capability to decouple applications is where Apache Kafka shines with its distributed architecture providing a combination of messaging and storage to enable real domain-driven design (DDD):

A microservices architecture decouples applications and allows the integration between legacy and modern applications, infrastructure, and technologies. This is once again where Apache Kafka shines as scalable real-time data integration and middleware with any Telco infrastructure, using low-level interfaces such as TCP, Syslog or SNMP, business integrations with a CRM, EMS, NMS, IMS, or even mainframe offloading and replacement.

Hybrid Architectures as Key for Success in the Telco Industry

Hybrid cloud infrastructure is key to success in the telecommunications industry. The network infrastructure will always be on-premise and at the edge, while big data analytics, customer relations, and other business applications can run in a central data center or at a cloud provider:

Apache Kafka and its ecosystem provide various architecture patterns for distributed, hybrid, edge, and global deployments.

Telco architectures often combine Kafka with other Telco-specific applications and services from leading providers in the telecom sector, such as Amdocs, Ericsson, or Huawei.

Disney Hotstar – OTT in Real-Time for Millions of Cricket Fans in India

Let’s now take a look at a specific example for building an OTT service with requirements for real-time processing at a massive scale.

Disney+ Hotstar (known as Hotstar outside India), is an Indian over-the-top streaming service. In 2018, during the Indian Premier League, Hotstar introduced “Watch N Play”, a real-time cricket prediction game in which over 33 million unique users answered over 2 billion questions, won more than 100 million rewards, built with Kafka as the backbone.

In the game, the user guesses the outcome of the next ball. If he/she guesses right before the actual result, they score points to climb up the ladder and receive rewards along the way. Supporting potentially millions of users with differing stream times & device latencies, Hotstar used topics to separate logical streams & partitions provide up to supporting 1M requests/second and more.

This is an impressive OTT service built with Apache Kafka and its ecosystem, isn’t it? Check out more details in Hotstar’s Kafka Summit talk “Scaling for India’s Cricket Hungry Population“.

OTT at Netflix and Tencent with Kafka

In addition to the above Disney+ Hotstar example, I want to quickly refer to two more OTT and media examples:

• Kafka at Netflix provides its business teams critical insights that help plan, determine spending, and account for all Netflix content. Kafka acts as a bridge for all point-to-point and Netflix Studio wide communications. The Netflix Studio Productions and Finance Team embraces distributed governance as the way of architecting systems.
• Kafka at Tencent is responsible for integrating Tencent’s internet, social, and content platforms. Kafka handles 10 trillion+ messages per day. This includes many major products – from the well-known QQ, QZone, video, App Store, news, and browser, to relatively new members of the ecosystem such as live broadcast, anime, and movies.

Time to Change for Traditional Players in the Telecom and Media Sector

Deloitte has defined four scenarios for 2030. Every company in the telco industry needs to think about they want to go:

This necessary change affects any company working in the telco industry. It is time to change – no matter if you work on projects in OSS, BSS, OTT, IMS, Middleware, 3rd party services, or any other telecom domain.

Open Source MANO (OSM) for NFV and OSS Interoperability

A great example of this change in the telecom sector is the project “Open Source MANO (OSM)” for management and orchestration in OSS environments. This open-source project formed by ETSI (European Telecommunication Standards Institute) to “address challenges associated with orchestration, interoperability and performance optimization between different NFV (Network Functions Virtualization) and OSS systems”:

“The unified message bus of Open Source MANO is implemented with Apache Kafka. This bus allows asynchronous communication between OSM components and enables the introduction of new modules that can be easily pluggable.”

(please note that I don’t like the term ‘bus’ because it depicts Kafka as a messaging bus even though it is an event streaming platform including messaging, storage, processing, and integration capabilities)

The Evolution of the Telco Industry with Apache Kafka

The following slide deck and video recording cover the evolution of Kafka in the telco industry, including use cases, architectures, and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Apache Kafka and Event Streaming for Innovation in the Telecom Sector

Current IT architectures in the telecom sector are not able to satisfy business needs because of their high complexity, lack of flexibility, and low level of automation.

Event Streaming with Apache Kafka and its ecosystem provides a scalable, reliable, and flexible infrastructure to process massive volumes in real-time. It enables real decoupling, plus powerful data integration and processing capabilities. Many telco enterprises build new infrastructures around Kafka.

This blog post focused on next-generation architecture to build Telco-OTT services. Disney+ Hotstar is an imposing example. But no matter in which part of the Telco industry you work, change and innovation are essential to staying competitive and innovative in the telecom sector.

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!