Event Streaming and Apache Kafka in Telco Industry

Event Streaming is a hot topic in Telco Industry. In the last few months, I have seen various projects leveraging Apache Kafka and its ecosystem to implement scalable real time infrastructure in OSS and BSS scenarios. This blog post covers the reasons for this trend. The end shows a whiteboard video recording exploring the different use cases for event streaming in telcos in detail.

The Evolution of the Telecommunications Industry

The telecommunications industries within the sector of information and communication technology is made up of all telecommunications / telephone companies and internet service providers. It plays the crucial role in the evolution of mobile communications and the information society.

Traditional telephone calls continue to be the industry’s biggest revenue generator in Telecommunications Industry. But thanks to advances in network technology, Telecom today is less about voice and increasingly about text (messaging, email) and images (e.g. video streaming).

High-speed internet access for computer-based data applications such as broadband information services and interactive entertainment, is pervasive. Digital subscriber line (DSL) is the main broadband telecom technology. The fastest growth comes from (value-added) services delivered over mobile networks.

Telco Architecture – OSS, BSS and Cloud

The traditional telco landscape is separated into OSS (Operations Support Systems), BSS (Business Support Systems) and OSS-BSS-Integration.

OSS track network inventory, assets and provisioning of services. BSS deal with customer relationship management (CRM) and processes such as taking orders, processing bills, and collecting payments. Here is an example of such a Telecommunications Industry landscape:

(Overview about OSS / BSS Landscape)

Modern architectures (have to) include one more aspect: Cloud. Like in almost all other traditional industries, telcos move towards cloud. The consequence is a hybrid architecture. Telco have to stay hybrid forever because a core part of their business is infrastructure. A combination of on premises data centers, edge processing and multi cloud architectures is the new normal in Telecommunications Industry.

Telco Business – CDR (Call Detail Record), Customer 360, Media and more

The business in Telecommunications Industry did not change that much from an end user perspective yet. Many basic concepts are still the same. OSS and BSS work together. Call Detail Records (CDR) metadata from phone calls is still processed and analysed to monitor the infrastructure and sell services to customers. Smartphone contracts and media offerings are  still pretty similar like ten years ago (you just get higher data limits per month).

However, the telecom business for the traditional telecommunications industry is changing significantly these days. It has to change. Otherwise, competitors like Apple or Netflix take over most of the business.

I like how this picture from Huawei describes these changes of the digital transformation in the telco industry:

The trend goes towards an open ecosystem, partnering, context-specific personalized offerings, multichannel, improved customer service and new innovative products.

These trends create a lot of challenges for the technical implementation. Traditional technologies which were used in the last 20 years do not work to realize this digital transformation. Requirements include real time processing at scale, decoupled and flexible applications and infrastructure, and still a reliable system with zero down time and zero data loss.

The combination of these requirements makes clear why more and more Telco projects and infrastructure are built with event streaming in mind and leveraging the de facto standard Apache Kafka and its ecosystem.

Use Cases for Apache Kafka in Telco

Apache Kafka is used in various very different telco projects in OSS, BSS, OSS-BSS-Integration. Many different architectures are used, included edge, data center, hybrid and multi cloud. Let’s take a look at some examples for use cases and architectures.

Middleware / Central Event Streaming Platform

• Central Event Streaming Platform as core infrastructure for integration and new applications
• Integration on all levels: OSS, BSS-OSS-Integration, BSS
• OSS – Separate clusters are used in each country / OpCo for integration with syslog etc.
• Integration with legacy and modern systems (including integration to existing legacy middleware)
• Move Data from OSS Fixed, Mobile, Cable into various systems
• Spanning different business units (TV, Analytics, Video Platform, Sales and Services)
• Integration with external partners (today via APIs, future: Kafka-native)
• Merger after acquisitions

Monitoring (Hardware, Performance, Security)

• Real time monitoring of their network devices (routers, switches, other network devices)
• Outside infrastructure: Million miles of fiber and coax, and over 40 million in home devices (severe weather to power grid outages to construction-related disruptions)
• Proactive monitoring: Real time monitoring, problem analysis, metrics reporting and action response
• Nationwide distributed IT infrastructure for all districts -> Fully-managed Security Information and Event Management
• Smarter network planning and deployment, as well as more precise ROI-based investment decisions, taking into account the profitability of each radio site based on customers’ actual and predicted profitability
• SIEM / SOAR: Mission critical cyber defence (typically a combination with other products like Elastic, Splunk, etc.)

Data Distribution

• Event Hub across the whole telco organisation for automation and virtualization of transports to enable data-driven infrastructure decisions
• Abstracted customer layer for all customer facing apps (Mobile App, Web Site, other CRM Systems)
• Next generation media distribution network. Provides alarm correlation to allow identifying network faults and reroute traffic in real time.
• Share information about network performance back to external customers
• API Gateway for network providers
• Accelerate the simplification and automation of standard processes, in both operational and support areas. These include IT and network operations, customer management back office functions and all other administrative activities
• Message archiving service for customers (traders specifically) that want fully searchable archive of all messages (delivered through all channels).

Data Processing

• CDR (Call Detail Record) processing: Technical ETL and business applications (range of scenarios like including upsell, churn analysis)
• Data consolidation issues with many data sources, late arriving events, data out of sync and high SLA guarantees
• Text messaging service to let you send and receive text and multimedia messages seamlessly on your smartphone, tablet, computer or the web using your mobile number; including rate limiting of SMS, MMS and RCS (Rich Communication Services) requests specifically for seasonal spikes; including redirect of RCS via SMS when delivery ability for the RCS message is not present
• Personalized messages and recommendations are built and then delivered to customer inboxes

Business Applications

• Microservices: Infrastructure for agile and decoupled app development
• BSS: Data engine, customer engagement engine, access broker, dynamic business engine, information backbone, back office systems, customer 360, master data management, …
• Customer 360: Track every interaction that the customer has with any channel
• Supply Chain Management (SCM): Real Time Inventory. Support real time information down to the store level
• Billing, Accounts, Lines of Service Data. Relieve Ops Burden
• Make the mobile app and digital marketing channels over time become the main customer acquisition and management platform
• B2B products like Inventory / Asset Tracking, Fleet Management
• Connected Car solutions to retailers for aftersales, including vehicle analytics for drivers, Infotainment, vehicle diagnostics, personal assistance, SOS, remote access, convenience features
• Connected Car solution platform for auto manufacturers to automate vehicle analytics for insurance companies and fleet management, tracking sales; vehicle performance; consumer demographics, etc.

Hybrid Infrastructure and Cloud Migration

• Consume data from on premises and monitor in cloud
• Aggregation cluster: Hub-and-spoke architecture with several regional data centers capturing data to local Kafka clusters and then sending metrics info to the main cluster
• Edge processing (on different levels: OpCo, data center, retail store, …)

Media

• Pass all data through a single source of truth (real time, highly scalable) for decoupling and quick innovation of building new services
• IP-to-MAC: Handle all incoming MAC address registration requests from set-top boxes (to check account entitlements to make sure they are a valid customer)
• Managing content metadata / epg
• Processing pay per view bookings
• Fraud use cases (MAC address cloning, sharing of streaming account details, SMS Spam detection, etc); leveraging machine learning in real time at scale for predictions
• Digital natives (like Netflix) built completely new businesses that were born digital. Everything in a digitally-native business is an event
• Context-specific advertising in real time based on user data (including product upsell, coupons, and other revenue creation with ad partners)

10 Reasons for Event Streaming (i.e. Apache Kafka) in Telco Industry

10 Reasons why telco projects leverage Event Streaming and Apache Kafka:

1. Real Time
2. Scalable
3. Cost Reduction
4. 24/7 – Zero downtime, zero data loss
5. Decoupling – Storage, Domain Driven Design (DDD), Reprocessing of events
6. Data processing and stateful client applications
7. Integration
8. Hybrid Architecture – On Premises, multi cloud, edge computing
9. Fully managed cloud
10. No vendor locking

These characteristics are not just for telco business but relevant in general, of course. However, in the video below, I cover how they map to the telco industry and OSS / BSS use cases.

Obviously, Apache Kafka alone has a lot of strong characteristics. But to realize the digital transformation successfully, you also need expertise (services and support) and tools (development, operations and monitoring) to set up, operate and scale such an infrastructure 24/7 mission-critical.

Slide Deck

The following slide deck covers the evolution of Apache Kafka in the telecom sector, including use cases, architectures and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Live Whiteboard

The following whiteboard explains why and how different companies in the telco industry leverage Event Streaming and Apache Kafka in OSS, BSS, OSS-BSS-Integration and Cloud / Hybrid Architectures. I cover in detail how the above 10 characteristics can used in your next telco project:

Video Recording – Telco Use Cases and Architectures

Here is a webinar where I talk about use cases, architectures and technologies around Apache Kafka in the telecom sector:

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!