Apache Kafka in the Public Sector – Part 5: National Security and Defense

The public sector includes many different areas. Some groups leverage cutting-edge technology, like military leverage. Others like the public administration are years or even decades behind. This blog series explores how the public sector leverages data in motion powered by Apache Kafka to add value for innovative new applications and modernizing legacy IT infrastructures. This is part 5: Use cases and architectures for national security, cybersecurity, defense, and military.

Blog series: Apache Kafka in the Public Sector and Government

This blog series explores why many governments and public infrastructure sectors leverage event streaming for various use cases. Learn about real-world deployments and different architectures for Kafka in the public sector:

1. Life is a Stream of Events
2. Smart City
3. Citizen Services
4. Energy and Utilities
5. National Security (THIS POST)

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts once published.

As a side note: If you wonder why healthcare is not on the above list. Healthcare is another blog series on its own. While the government can provide public health care through national healthcare systems, it is part of the private sector in many other cases.

National Security and Defense

National security or national defense is the security and defense of a nation-state, including its citizens, economy, and institutions, is a duty of government. Originally conceived as protection against military attack, national security is now widely understood to include non-military dimensions, including the security from terrorism, minimization of crime, economic security, energy security, environmental security, food security, cyber-security, etc. Similarly, national security risks include, in addition to the actions of other nation-states, action by violent non-state actors, by narcotic cartels, and by multinational corporations, and also the effects of natural disasters.

Cybersecurity – The Threat is REAL!

Cybersecurity has become a real threat due to the ongoing digital transformation. Networking, communication, connectivity, open standards, and” always-on” principles provide significant benefits and innovation, but also new cyber threats. The US’s Colonial Pipeline ransomware attack in May 2021 is just one of many successful attacks in the past few quarters. There is no real doubt that the number of attacks will go up significantly in the following months and years.

Supply chain attacks make the threat even bigger. Even if your software is secure, a single loophole in a tiny 3rd party software can provide the attack surface into the whole company:

Defeating Cybersecurity with Apache Kafka

Threat detection, incident management, and proactive or predictive countermeasures are only possible with real-time data correlation and processing.

I won’t repeat myself. You can read all the details in my separate blog series about the success of event streaming with Apache Kafka to provide real-time cybersecurity at scale:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3: Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

TL;DR: Data in motion HAS TO BE the backbone of cybersecurity infrastructure:

National Security powered by Apache Kafka

While I work a lot with customers from the government and public administration, success stories are scarce. All of this is even more true for national security areas. Hence, contrary to the other posts of this blog series, I can only talk about a use case and architecture without giving a concrete example from the real world. Sorry for that 🙂

Nevertheless, I have a great example of sharing: Confluent presented an edge and hybrid demo for smart soldiers together with a partner at the AUSA 2021, an annual event from the association of the united states army.

Edge Computing (Soldier) and Replication to the Data Center (Command Post)

The enterprise architecture looks very similar to other Kafka edge deployments and hybrid architectures from a high level. Hence, my infrastructure checklist for Kafka at the Edge applied in national security use cases, too.

The following diagram shows our national security demo use case and architecture:

Kudos to my colleagues Jeffrey Needham and Michael Peacock, who built the demo leveraging the Kafka ecosystem for national security.

So, what’s happening in this use case?

• Soldiers wear a small compute “thing” running the whole “lightweight” single broker Kafka infrastructure.
• An MQTT client consumes sensor data from the environment and other remote locations.
• The collected sensor data is processed continuously with Kafka-native stream processing (Kafka Streams or ksqlDB). The Kafka broker stores the events and truly decouplings producers and consumers.
• Confluent Cluster Linking replicates relevant curated information to the command post in real-time (as long as internet connectivity is available).
• The command post is a small data center running a mission-critical Kafka cluster. The aggregation of all the data from all soldiers and other interfaces (vehicles, weather information, etc.) is correlated in real-time to provide situational awareness.
• Relevant data is replicated from command posts via Cluster Linking to a remote location (like a military data center). Here, analytics and other traditional IT workloads use the information for different use cases.

TL;DR: The project demonstrates vast benefits. The open infrastructure leverages the same components and technologies across the edge (soldiers), small on-site data centers (command posts), and large remote data centers. Reliable data integration and processing provide all the capabilities for situational awareness in real-time in a national security scenario end-to-end at the edge and remote locations.

NASA enables real-time data from Mars

Let’s end this blog series with one more exciting use case from the public sector. It is not directly related to national security. But who knows, maybe this will be relevant for attack scenarios in the future when the aliens attack.

The National Aeronautics and Space Administration (NASA) is an independent agency of the U.S. federal government responsible for the civilian space program, as well as aeronautics and space research.

NASA enables real-time data from Mars with the help of Apache Kafka. Real-time data extends into the far frontiers via its Deep Space Network (DSN). Data grows exponentially from spacecraft and other systems. Real-time data enables NASA for responsive citizen engagement, real-time situational awareness, anomaly detection, event-driven missions, and security operations. The global fabric of Kafka clusters allows for real-time sharing, event streaming, and other combinations of both real-time and historical data.

More details about NASA’s Kafka usage are available in a great article from the Federal News Network.

Data in Motion for National Security and Defense

National security is relevant across all areas in the public sector. This post showed an example from the military. However, situational awareness in real-time is needed everywhere, as the colonial pipeline attack and many other ransomware stories proved in the last months.

Event Streaming with Apache Kafka provides the unique capability of using a single technology across the edge and hybrid cloud architectures for real-time data integration and processing for National Security. Even disconnected or air-gapped environments are supported. Learn more about data in motion for cybersecurity in my dedicated blog series about Apache Kafka for Cybersecurity across Industries.

How do you leverage event streaming in the public sector? Are you working on any national security or cybersecurity projects? What technologies and architectures do you use? What projects did you already work on or are in the planning? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.