Apache Kafka in the Financial Services Industry

The rise of event streaming in financial services is growing like crazy. Continuous real-time data integration and processing are mandatory for many use cases. Many business departments in the financial services sector deploy Apache Kafka for mission-critical transactional workloads and big data analytics. High scalability, high reliability, and an elastic open infrastructure are the key reasons for Kafka’s success. This blog post explores different use cases, architectures, and real-world examples in the FinServ sector.

FinServ Enterprise Reality: Innovate or be disrupted!

There is no way around it: The new business reality is very different from the last decades:

• Technology was a support function in the past.
• Innovation required for growth.
• “Good enough” to run on yesterday’s data.
• Technology is the business.
• Innovation is required for survival.
• Yesterday’s data = failure.
• Modern, real-time data infrastructure is required.

Only two options exist for enterprises in the finance sector: Innovate or be disrupted!

The New FinServ Enterprise Reality  – Every Company is a Software Company

Please take a look at your favorite traditional bank and how its market cap looks like compared to new FinTech companies such as Robinhood, Stripe, Square, or Revolut.

Some traditional companies re-invented themselves to focus on innovative new products and great customer experience to stay competitive. Software is eating the world, including the finance sector.

Here are a few examples:

• Capital One: 10,000 of 40,000 employees are software engineers
• Goldman Sachs: 1.5B (billion!) lines of code across 7,000+ applications
• JPMorgan Chase & Co: Employs over 50,000 people in technology and has $10B+ technology spend.

Most successful post-modern companies in the finance sector heavily rely on Apache Kafka. This is true for emerging fintechs but also for (some) traditional banks.

Apache Kafka in Financial Services

Various use cases emerged to deploy event streaming with Apache Kafka in the finance industry. This includes mission-critical transactional workloads like payment processing or regulatory reporting and big data analytics projects leveraging Machine Learning, data lakes, etc.

Examples for Real-World Deployments

Here are a few companies leveraging Apache Kafka for banking projects:

Check past Kafka Summit video recordings and slides for details about use cases and architectures of these companies from the finance sector.

Here are a few concrete examples:

• Capital One: Becoming truly event-driven – offering a service other parts of the bank can use.
• ING: Significantly improved customer experience – as a differentiator + Fraud detection and cost savings.
• Nordea: Able to meet strict regulatory requirements around real-time reporting + cost savings.
• Paypal: Processing 400+ Billion events per day for user behavioral tracking, merchant monitoring, risk & compliance, fraud detection, and other use cases.
• Royal Bank of Canada (RBC): Mainframe off-load, better CX & fraud detection – brought many parts of the bank together
• 10X Banking: Cloud-native and open core-banking platform to implement a next-generation FinServ platform
• Robinhood: Commission-free stock trading using a mobile app and website.

This is just a concise list of companies in the financial sector using Apache Kafka as an event streaming platform for their business’s heart. Plenty of other examples are available by tens of global banks leveraging Apache Kafka for many use cases.

Kafka makes your Business Real-Time

The huge advantage is that Kafka allows decoupling your applications and infrastructure in a domain-driven design (DDD). Each microservice can use its own technology or product but leverage the same data (with security and privacy in mind, of course):

It is great to see that many FinServ companies do not just leverage Kafka in their applications but also contribute to the community. For instance, Robinhood published Faust: A stream processing library, porting Kafka Streams’ ideas from Java to Python.

This is a great example of a microservice architecture and the freedom of technology choice: Robinhood did not want to use Java for (some) applications and chose Python instead. No problem with Kafka as the brokers are dumb. The data processing and business logic happen in the clients in your favorite programming language.

And to be clear: Financial services are important in every company! Payments, orders, fraud, and similar transactional and analytical data rely on FinServ applications and the integration with partners.

Slides – The Rise of Event Streaming in FinServ

The following slide deck goes into more detail. Learn about the rise of event streaming in the financial services industry. Kafka is adopted in more and more scenarios:

Kafka in Banking for Middleware, Mainframe, Machine Learning, Open API, and more

The following links share additional content related to many banking and FinServ use cases and architectures:

• Use cases and architectures for Apache Kafka across industries
• Apache Kafka for Machine Learning in Banking and FinServ
• Middleware modernization with Apache Kafka vs. MQ, ETL, ESB
• Mainframe integration, offloading, and replacement with Apache Kafka
• Kafka and API Management for Open Banking API
• Edge Computing – Deployment of Apache Kafka in the Bank Branch
• XML Processing with Apache Kafka for Legacy Integration

Please check them out to learn more about the usage of event streaming with Kafka and its ecosystem across various business units in the Finserv sector.

Last but not least, please be aware that the term “real-time” is used in many contexts and can have different meanings. Read “Kafka is NOT hard real-time” to understand why Kafka is used in most banking projects, but not for the specific use case of trading in microseconds.

Software is Eating the Banks and FinServ Industry

Software is eating the world, including financial services. Continuous real-time data integration and processing are mandatory for many use cases. Apache Kafka is deployed across industries for mission-critical transactional workloads and big data analytics. No matter if you need to integrate with legacy systems, process mission-critical payment data, or build batch reports and analytic models, Kafka is a predominant choice as part of the architecture. Hybrid, edge, and multi-cloud deployments of Kafka are the new black.

What are your experiences and plans for event streaming in the financial services industry? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.