Streaming Data Processing with Apache Flink and Spark
Podcast · Streaming Data Processing with Apache Flink and Spark

Real Time Stream Processing with Apache Flink

by ZumiLabs · 22m
0:00 22m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

You know when you swipe your credit card, you just expect this immediate flawless precision, right? You tap your phone, the terminal beeps, and the transaction is approved.

Host 2

Yeah, it feels totally binary.

Host 1

Exactly. But think about what actually happens in that like four hundred millisecond window between the tap and the beep.

Host 2

No, it is a massive invisible war. I mean, in that fraction of a second, your request hits a payment gateway, routes to an issuing bank, and then runs through this absolute gauntlet of complex logic.

Host 1

A gauntlet. I like that.

Host 2

Yeah, because the system is, you know, cross-referencing your current physical location against your last known purchase. It's calculating your transaction velocity, basically how many times you spent money in the last hour.

Host 1

Right.

Host 2

And it's running massive machine learning models to detect fraud signatures based on global threat intelligence all at once.

Host 1

And it does all of that before you can even pull your hand back from the terminal. Because I mean, if it takes three seconds, you get annoyed, and if it takes ten seconds, you just assume the system is broken. We demand that level of processing. But we rarely stop to consider the actual data infrastructure required to make those instantaneous decisions possible.

Host 2

We really don't. We just take it for granted.

Host 1

We do. So welcome to today's deep dive. We are pulling from a really fascinating stack of sources today. We've got technical AWS and Confluent blogs, open source documentation, academic papers, and real world architectures from some of the biggest cybersecurity and e-commerce giants out there.

Host 2

It's a lot of ground to cover.

Host 1

It is.

Host 2

But our mission is to map out the nervous system of the modern internet. And the engine sitting right at the core of all this real time processing is called Apache Flink.

Host 1

Right.

Host 2

And you know, to understand why Flink is the engine of choice for these massive architectures, we have to look at the limitations it was actually built to overcome.

Host 1

Okay. Lay it on me.

Host 2

So for decades, the industry standard was batch processing. A business would collect this massive chunk of data all day, dump it into a data warehouse, and then just run a heavy processing job overnight.

Host 1

Well, the obvious flaw there is that you are always operating in the past.

Host 2

Exactly.

Host 1

I mean, if you run a batch job on a financial ledger to detect, say anomalous wire transfers at midnight, you might successfully identify that a user's account was drained.

Host 2

Sure, you'll find the breach.

Host 1

But the money is already gone.

Host 2

Right. You're doing an autopsy rather than preventing a crime. And that's where Flink comes in. It actually originated as a research project called Stratosphere at the Technical University of Berlin.

Host 1

Oh, nice.

Host 2

Yeah. And it operates on a fundamentally different paradigm. It is a distributed processing engine designed specifically for stateful stream processing.

Host 1

Stateful stream processing. Let's wait, let's pause on the word stateful, because stream processing itself isn't a completely new concept, right?

Host 2

No, not at all.

Host 1

Like there are stateless systems out there that handle streams, but they treat every single event in isolation. If an event comes in, it processes it and basically just forgets about it. Flink is different because it remembers.

Host 2

Flink maintains what we call per key state.

Host 1

What does that mean in practice?

Host 2

So if you are processing a stream of credit card transactions, Flink can hold the running total of a specific user's spend or the sequence of locations they've purchased from directly in memory.

Host 1

Oh wow. So it doesn't have to pause and send a query to some external database task with that user did five minutes ago.

Host 2

Exactly. The context is held locally. It's just sitting there, ready to be evaluated the very millisecond the next event arrives.

Host 1

Okay, but keeping all that context in memory sounds incredibly resource intensive. I mean, if we are talking about a global platform tracking millions of active users, RAM is not infinite.

Host 2

Not even close.

Host 1

So how does it maintain that state without the servers simply running out of memory and crashing?

Host 2

That is where Flink's use of local state backends comes into play. Specifically a technology called RocksDB.

Host 1

RocksDB. Right. Got it.

Host 2

When you define state in Flink, it is partitioned by a key, like a user ID. Each worker node in a Flink cluster is only responsible for the state of the specific keys assigned to it.

Host 1

So it divides up the workload.

Host 2

Yeah, exactly. It keeps the hottest, most active state in its local RAM for instant access. And when the state grows too large, ROXDB intelligently spills the less frequently accessed data to the local disk of that specific node.

Host 1

Oh that's clever. So you get the speed of in memory processing with the capacity of disk storage.

Host 2

Yep. And all without the latency of a network call to an external database.

Host 1

Which is huge. But that brings up a massive logistical problem. Time.

Host 2

Oh time's the hardest part.

Host 1

Right. Because data is pouring in from mobile networks across the globe. You might have a user swiping a card in a major city with super fast 5G and another user making a mobile purchase on a spotty 3G connection out in a rural area.

Host 2

Very common scenario.

Host 1

So those events are going to arrive at the Flink cluster totally out of order. If Flink is processing things the millisecond they arrive, how does it handle the fact that the data stream is essentially a scrambled timeline?

Host 2

Well Flink solves this by decoupling two concepts of time. First there is processing time, which is the literal wall clock time the server receives the data.

Host 1

Okay.

Host 2

And then there's event time, which is the timestamp generated on the user's device, the exact moment the action occurred. Flink prioritizes event time and it manages the chaos of network latency using things called watermarks.

Host 1

Okay, the sources mention watermarks, but the mechanics of them are a bit dense. If I'm understanding this correctly, a watermark isn't actually a piece of data itself, is it?

Host 2

No, it's not data.

Host 1

It's like a metadata signal injected into the stream that establishes a baseline of time.

Host 2

Yeah, think of a watermark as a system-wide declaration. A watermark with a timestamp of, say, 10.05 AM flows through the stream and basically tells all the processing operators, hey, you can safely assume that we've received all events that occurred up to 1005 AM.

Host 1

Oh, I see.

Host 2

So it's telling the system you can now finalize any calculations or time windows for that period. It is essentially a configurable grace period for out-of-order data.

Host 1

I see a vulnerability there though. Because if you set that grace period too tight, you drop legitimate events that were just lagging on a slow network.

Host 2

You definitely could.

Host 1

But if an event arrives five minutes after the watermark has already passed, what actually happens to it? I mean, Flink already finalized the calculation and moved on. Is that late data just lost forever?

Host 2

It's not lost, but it does require specific handling. So Flink allows developers to define allowed lateness for a window.

Host 1

Allowed lateness.

Host 2

Right. Which will actually reopen a finalized calculation, update the result with the late event, and then send out a corrected update downstream.

Host 1

Wait, it can just reopen it and fix it on the fly?

Host 2

It can. And if an event is so late that it passes even the allowed lateness threshold, Flink routes it to a side output.

Host 1

What's a side output?

Host 2

It's essentially a quarantine stream for severely delayed data, ensuring it is preserved for later auditing or you know, batch reconciliation, without bottlenecking the real-time pipeline.

Host 1

Okay, let's push this even further. We have a distributed system running across thousands of cores, holding massive amounts of partition state and constantly calculating time windows based on these watermarks.

Host 2

That's a lot of moving parts.

Host 1

It is. And in the real world, hardware fails. A server rack loses power.

Host 2

All the time.

Host 1

So if the stream is literally never ending and you lose a worker node, how do you ever recover without losing data or worse, processing the same data twice when the node reboots?

Host 2

You are touching on Flink's absolute crown jewel here. Exactly once semantics. Flink guarantees that every single piece of data is reflected in the final state exactly once.

Host 1

How is that even possible?

Host 2

Well, it achieves this without stopping the stream using a mechanism based on the Chandy Lamport algorithm. Flink calls it asynchronous barrier snapshotting, or ABS for short.

Host 1

Okay, ABS. Because, you know, in a traditional database, you just lock the tables, take a snapshot back up, and then unlock them.

Host 2

Right, very straightforward.

Host 1

But you can't lock a continuous stream of millions of events per second.

Host 2

You can't, which is why ABS is so elegant. Flink's master coordinator, which is called the job manager, regularly injects these lightweight markers called barriers directly into the data streams at the source.

Host 1

Are the barriers data?

Host 2

No, these barriers do not carry data. They act as logical dividing lines. They just flow downstream right alongside the actual events.

Host 1

Okay, I'm picturing it.

Host 2

So when a worker node called a task manager receives a barrier, it knows that all events before that barrier belong to the current snapshot, and all events after it belong to the next one.

Host 1

So the worker takes a snapshot of its local state at that exact moment. It's rocks DB files, it's running totals and ships that snapshot to durable storage, like an Amazon S3 bucket.

Host 2

Yes, exactly. And then it immediately forwards the barrier to the next operator downstream. The stream literally never halts.

Host 1

That is brilliant.

Host 2

Because the barriers flow naturally with the data, Flink generates a globally consistent snapshot of the entire distributed pipeline at a precise logical moment in time.

Host 1

So if a node just bursts into flames.

Host 2

Right. If a node dies, Flink simply spins up a new node, pulls the last successful snapshot from S3, rewinds the data sources to the exact offset of that snapshot, and just resumes processing.

Host 1

Okay. That paints a really clear picture of the engine itself, but I mean Flink can't operate in a vacuum. If it's performing these highly complex stateful calculations and taking asynchronous snapshots, we really need to look at the blueprint of the wider architecture.

Host 2

Yeah, the whole ecosystem.

Host 1

Exactly. How is data actually being fed into the system fast enough to make it useful?

Host 2

So in a modern streaming architecture, Flink sits as the processing middleware, flanked by sources and syncs. The most common ingestion source is a distributed message queue like Apache Kafka.

Host 1

Kafka. We hear about that one all the time.

Host 2

Yeah, Kafka acts as the high-throughput buffer. It absorbs massive spikes in traffic, say millions of users logging in simultaneously, and feeds those events to Flink at a rate the cluster can actually digest.

Host 1

Kafka makes total sense for event logs, but businesses still run on traditional relational databases, right? Like MySQL or Postgres.

Host 2

Absolutely.

Host 1

And you can't just run continuous SQL queries against your primary production database to feed a stream. I mean, you'd crush the database under the query load.

Host 2

You would destroy it. And that introduces another critical ingestion method, Flink CDC, or change data capture.

Host 1

Change data capture. How does that avoid the query load?

Host 2

Well, rather than querying the database tables, Flink CDC uses tools like Debezium to tap directly into the database's transaction log, like the MySQL bin log, for example.

Host 1

Oh, the internal log.

Host 2

Right. Every time a row is inserted, updated, or deleted, the database writes that change to its internal log anyway. Flink just reads that log directly.

Host 1

That's sneaky.

Host 2

It is. It extracts the raw changes the millisecond they happen without adding any query load to the database itself, and then turns those database changes into a continuous stream of events.

Host 1

That is incredibly efficient.

Host 2

So once Flink ingests the data from Kafka or CDC, the task managers execute the transformations. But what happens to the output? It doesn't just sit in Flink. No, the output is continuously pushed to downstream syncs.

Host 1

Syncs. Got it.

Host 2

A sync might be a fast analytical database like Elasticsearch to power a live dashboard. It could be a caching layer like Redis that instantly updates the inventory counter on a retail website.

Host 1

Or what about triggering an action?

Host 2

Crucially, yes. A sync can be an application webhook that triggers a real-world automated action based entirely on the result of Flink's calculations.

Host 1

Which brings us perfectly to the first real-world scenario we need to dissect today: cybersecurity. Specifically SIEM, security information and event management. We're talking about defending enterprise networks where latency is literally the enemy.

Host 2

Right, and the historical approach to SEAM is a perfect example of the limitations of batch processing we were discussing earlier.

Host 1

How so?

Host 2

Well, traditional SEAM ingests mountains of data: firewall logs, active directory logins, endpoint telemetry, web proxy events, and it writes all of that data into a centralized database. Security analysts then write complex queries to search that database for indicators of compromise.

Host 1

But if you are querying a database, you are inherently looking backward. Even if you automate those queries to run, say, every 10 minutes, a sophisticated attacker moves laterally across a network in seconds.

Host 2

In a heartbeat, yeah.

Host 1

So by the time your query identifies a suspicious login, the attacker has already exfiltrated the data. The alert is just noise at that point.

Host 2

Exactly. It just leads to severe alert fatigue. Flink changes the paradigm here by introducing complex event processing or CEP. With Flink CEP, you aren't writing queries against stored data. You are defining behavioral patterns, and Flink evaluates the live stream against those patterns in memory.

Host 1

Okay, let's use a brute force attack as a concrete example. Say an attacker is trying to guess an employee's password. The pattern is pretty obvious. Multiple failed logins followed by a success. How does Flink identify that without querying the history of that specific user?

Host 2

So you deploy a CEP pattern into the Flink cluster. You specify the conditions. You say, look for a sequence of five failed authentication events for a single user ID, followed immediately by a successful authentication event for that same user ID.

Host 1

Okay.

Host 2

But only if the successful login originates from an IP address we've never seen that user connect from before. And crucially, you constrain this pattern with a time window, say all occurring within two minutes.

Host 1

And because Flink maintains state, it just creates a lightweight state machine for every user currently logging in.

Host 2

Exactly. The first failed login opens the state machine. The second failed login advances it.

Host 1

And if two minutes pass.

Host 2

Flink just drops the state, freeing up memory. But if the attacker hits the full sequence, five fails, one success from a new IP within the window, Flink detects the breach the exact millisecond that final successful login event hits the pipeline.

Host 1

That is wild.

Host 2

Yeah. And the academic papers on the slide project, the system of logs intrusion detection engine detail exactly this architecture. They pipe Palo Alto Network's firewall logs and threat connect intelligence into Kafka, and Flink continuously evaluates the streams.

Host 1

But wait, detecting the sequence is only half the battle, right? To know if an IP address is actually malicious or to know if a user's outbound bandwidth has suddenly spiked beyond their normal baseline. Flink has to cross-reference the live event against threat intelligence feeds.

Host 2

It does.

Host 1

I want to push back on the mechanics of that for a second. Threat intelligence databases are massive. They contain millions of known bad IP addresses or file hashes.

Host 2

Very true.

Host 1

If Flink is processing millions of firewall logs a second, and it has to verify the IP address of every single log against a massive threat list, you have a massive bottleneck. You can't make millions of network calls to an external database every second.

Host 2

You absolutely cannot, which is why Flink utilizes temporal joins.

Host 1

Temporal joins.

Host 2

Yeah. Flink doesn't ask an external database for the threat intelligence. It actually consumes the threat intelligence feed as its own separate data stream and stores the entire lookup table locally in its partition state backends.

Host 1

Oh. So the worker node processing a specific subnet's firewall logs has the relevant threat intelligence for that subnet sitting right there in its local RAM or RocksDB instance.

Host 2

Exactly. But it's called a temporal join because time is a primary key. Threat intelligence changes constantly. An IP address might be malicious on Tuesday, but reassigned to a legitimate service by Thursday.

Host 1

Right.

Host 2

So when a firewall log from Wednesday flows through Flink, it joins that log against the state of the threat intelligence table as it existed on Wednesday at the exact moment the log was generated.

Host 1

Wow.

Host 2

It ensures the correlation is historically accurate, preventing false positives caused by race conditions.

Host 1

And because the detection happens in milliseconds, the output can actually be actionable. We aren't just sending an email to a dashboard for an analyst to check later.

Host 2

No. No.

Host 1

Flink pushes the output to a sync that triggers automated remediation. It can fire an API call to the firewall to immediately drop all packets from that IP, or it can revoke the user's session tokens across the entire corporate network, locking the attacker out mid-stride.

Host 2

Yeah, you move completely from forensic analysis to real-time automated defense.

Host 1

Which is incredible. But the really interesting thing is that the underlying logic there, stateful pattern matching across multiple streams of events, is completely agnostic to the industry, isn't it?

Host 2

Oh, totally. The exact same infrastructure used to block a cyber attack is currently revolutionizing retail through what we call the customer 360 view.

Host 1

Because retail struggles with the exact same latency issues as cybersecurity.

Host 2

They really do.

Host 1

I mean, if I walk into a physical brick and mortar store, buy an expensive pair of hiking boots, and use my loyalty card, the point of sale system records that transaction, but in a legacy architecture, that transaction just sits in a local database until a batch job runs overnight to update the central CRM.

Host 2

Which creates a massive blind spot for the retailer. I mean, you are an active, highly motivated buyer right now, but the online systems don't know you made a purchase yet.

Host 1

Right.

Host 2

So the cross-sell opportunities like offering you a discount on hiking socks or a backpack are entirely missed. Or worse, the online ad network wastes money showing you advertisements for the boots you literally already own.

Host 1

Yes. I hate when that happens. It's so annoying. So to solve this, Flink essentially acts as the central nervous system for an omnichannel architecture.

Host 2

Exactly.

Host 1

It ingests a stream of online click streams, every product you view on the app. It uses CDC to stream offline transactions from the physical point of sale databases. It streams updates from the customer support call center. It basically pulls all these disparate feeds together and joins them on a single key, the customer ID.

Host 2

And the result is a continuously updated materialized profile of the customer held entirely in Flink state. The scale of this operation is just difficult to overstate.

Host 1

Give us an example.

Host 2

Well, during Alibaba's Double 11 shopping festival, their Flink clusters process over 470 million events per second.

Host 1

470 million per second.

Host 2

Yes. They are joining offline logistics data with live user browsing to adjust product rankings, update dynamic pricing, and personalize recommendations instantly for hundreds of millions of concurrent shoppers.

Host 1

The mechanics of personalizing those recommendations must rely heavily on how Flink interprets human behavior, though. Particularly through something the sources call session windows.

Host 2

Yeah, session windows are key.

Host 1

Because in a standard time window, you might just group all actions that happen between 1 p.m. and 2 p.m., but human browsing doesn't fit neatly into hourly blocks.

Host 2

Nope. Human behavior is defined by bursts of activity followed by periods of idleness. Session windows in Flink are incredibly powerful because they are defined by gaps.

Host 1

Gaps, how so?

Host 2

You configure the window to stay open as long as the user is active. If the user clicks on hiking boots, reads a review, and adds a tent to their cart, all of those events are grouped into a single session. Flink only closes the window when a specified period of inactivity passes, say the user doesn't click anything for 15 minutes.

Host 1

Okay. And when that window closes, Flink evaluates the state of that specific session.

Host 2

Exactly.

Host 1

Did they abandon a cart full of high margin items? If so, the instant that 15-minute inactivity threshold is breached, Flink triggers a webhook to email them a temporary 10% discount code to pull them back in.

Host 2

Boom.

Host 1

And the action is driven entirely by the natural cadence of the user's behavior.

Host 2

Yeah, the architecture is constantly adapting to the present reality of the data.

Host 1

Which is really the core takeaway for anyone navigating this space. Whether you are building an intrusion detection system or attempting to unify global retail channels, the reliance on batch processing forces a business to live in the past.

Host 2

It really does.

Host 1

Apache Flink and the broader shift toward event-driven streaming provides the infrastructure to act with context in the exact moment an event matters most.

Host 2

It removes latency not just from data pipelines, but from business decisions entirely.

Host 1

And if we take that idea of autonomous millisecond decision making and attach it to where the tech industry is heading right now, the implications are just staggering. Oh, absolutely. We have Flink evaluating massive amounts of behavioral context, joining it with historical models, and triggering automated actions. Consider what happens when we wire this nervous system into a gentic AI.

Host 2

Wow. You're talking about removing the predefined CEP rules completely.

Host 1

Exactly. Instead of a human engineer writing a rule that says five failed logins means a brute force attack, an autonomous AI agent sits at the end of the Flink pipeline.

Host 2

Right.

Host 1

Flink feeds the AI an instantly unified, stateful view of the entire global network. The AI analyzes the context, identifies a novel threat signature no human has ever seen, and autonomously deploys network-wide firewall rules to neutralize it all in a fraction of a second, without a single human in the loop.

Host 2

That is wow. We are moving towards systems where the infrastructure doesn't just transport the data, it actively reasons about it in real time.

Host 1

It is an exhilarating and maybe slightly terrifying, entirely new frontier for data engineering. Thank you for joining us on this deep dive. The next time your card approves in the blink of an eye or a website seems to anticipate your next move before you even make it, you know the architectural heavy lifting making it possible. Catch you next time.

Save to playlist

Manage playlists →

Share