What is Event Driven Architecture (EDA)?

Summary

In this informative talk, Whitney from IBM's Cloud team delves into event-driven architectures, highlighting their role in enabling real-time interactions, scalable microservices, and extensible data analytics. She explains the principles of the reactive manifesto, emphasizing asynchronous messaging, scalability, and resilience in system design. Whitney illustrates how events, as immutable statements of past occurrences, facilitate targeted and conversational messaging between services, leading to a responsive and maintainable system. She further explores the benefits of event logs for triggering actions, optimizing data persistence, and supporting stream processing, all of which contribute to a robust, pluggable, and history-aware system.

Takeaways

• 🌟 Event-driven architectures are designed to enable real-time user interactions, pluggable microservices, and extensible streaming and data analytics.
• 📜 The Reactive Manifesto provides a set of guidelines for building responsive, resilient, and scalable systems through asynchronous messaging.
• 🔄 Scalability in system design means the ability to expand or contract hardware usage in response to workload changes.
• 🛡️ Resilience implies the absence of single points of failure and the capacity to handle system failures gracefully.
• 📢 Asynchronous messaging is central to the Reactive Manifesto, allowing for non-blocking communication between system components.
• 📝 An event in this context is an immutable statement of fact about something that happened in the past, used for communication between services.
• 🔄 Event-driven systems allow for highly scalable architectures, as the event producer is not concerned with the number of consumers.
• 🗃️ Event logs provide a replayable history of events, enabling persistent data storage and the ability to process historical data.
• 🤖 The event backbone is the central system through which all events flow, enabling various applications to produce and consume events.
• 🚀 Event logs can trigger actions, such as serverless functions, which can be modular pieces of code associated with specific events.
• 🔧 Event-driven architectures support stream processing, allowing for real-time calculations, business rule implementation, and analytics.
• 🔌 The publish-subscribe (pub-sub) communication model is integral to event-driven systems, facilitating plug-and-play microservices that can easily integrate into the system.

Q & A

What is an event-driven architecture?

An event-driven architecture is a system design approach that relies on the production, detection, consumption of, and reaction to events. It enables real-time interactions and is highly scalable and resilient, as services react to events without direct coupling.

What does the reactive manifesto aim to provide in system design?

The reactive manifesto provides a set of community-driven guidelines intended to offer a cohesive approach to system design, emphasizing principles such as message-driven architecture, scalability, resilience, and responsiveness.

What are the core values of the reactive manifesto?

The core values of the reactive manifesto include building message-driven, asynchronous systems that are scalable, resilient, and responsive, ensuring the system can handle varying workloads and avoid single points of failure.

How does event-driven architecture enable real-time user interactions?

Event-driven architecture enables real-time user interactions by allowing systems to react instantly to events as they occur. This is achieved through the use of asynchronous messaging and the ability to trigger immediate actions in response to events.

What is the difference between messaging and eventing in system communication?

Messaging involves direct communication between services, where the sender has knowledge of the receiver. Eventing, on the other hand, involves the production of events without concern for which service is consuming them, promoting a decoupled and scalable system design.

Why is data persistence in event-driven architectures considered to be more permanent compared to messaging?

In event-driven architectures, data persistence can be more permanent because events are logged and can be stored indefinitely, depending on the system setup. This creates a replayable event log and a history of all events, unlike messaging where data persistence is often transient.

What is the role of the event backbone in an event-driven architecture?

The event backbone is the central component of an event-driven architecture through which all events flow. It facilitates the distribution of events from producers to consumers, enabling a scalable and responsive system.

How can event logs be utilized to optimize and customize data persistence?

Event logs can be used to trigger actions, update local data stores, and produce new streams of data that reflect the most current state. This allows for optimized data persistence tailored to the needs of specific services within the system.

What is the significance of the pub-sub communication model in event-driven architectures?

The pub-sub (publish-subscribe) communication model is significant in event-driven architectures as it allows for a highly pluggable system. Services can easily produce and consume events without direct coupling, promoting extensibility and scalability.

How does event-driven architecture support the implementation of business rules and real-time analytics?

Event-driven architecture supports the implementation of business rules and real-time analytics by providing a stream of events that can be processed in real-time. This enables systems to make immediate decisions based on current data and trends.

What is the relationship between event-driven architecture and serverless functions?

Event-driven architecture can trigger serverless functions, which are modular pieces of code that execute in response to events. Once the function completes its task, it can send another event back into the system, creating a reactive and efficient process flow.

Outlines

00:00

🌟 Introduction to Event-Driven Architectures

Whitney from IBM's Cloud team introduces the concept of event-driven architectures, emphasizing their role in enabling real-time interactions, pluggable microservices, and scalable data analytics. She begins with an overview of system design principles from the reactive manifesto, which advocates for asynchronous messaging, scalability, and resilience in system design. The manifesto promotes a design that is responsive, maintainable, and extensible. Whitney then explains the difference between messaging and eventing, using a retail application as an example to illustrate how eventing facilitates communication between various services without a direct dependency, leading to a highly scalable system with potential for persistent data storage and event replayability.

05:02

📡 The Event Backbone and Its Capabilities

This paragraph delves into the specifics of event producers in an event-driven architecture, such as web and mobile applications, as well as IoT devices. It introduces the 'event backbone' as the central system through which all events flow. The capabilities enabled by this architecture include the triggering of actions based on events, optimization of data persistence, and the maintenance of an up-to-date inventory through local data stores. The paragraph also touches on the use of event logs for data-heavy applications like AI and the importance of a data lake for storing and accessing raw and transformed data. Additionally, it highlights the potential for stream processing using the Apache Kafka Streams API, which can facilitate real-time recommendations and analytics in applications like music streaming services.

10:08

🔌 Pub-Sub Communication and Microservices Integration

The final paragraph focuses on the pub-sub communication layer of event-driven architectures, which allows for the easy integration of microservices. It discusses the plug-and-play nature of these architectures, where new microservices can be added without disrupting existing applications. The paragraph also addresses the ability of new services to consume data from the beginning of the application's history if stream history has been preserved. This feature, along with the overall benefits of event-driven architectures, is positioned as an enhancement to the reactive manifesto's principles for system design, with eventing at its core.

Keywords

💡Event-driven architecture

Event-driven architecture is a programming paradigm where the flow of the program is determined by events such as user actions, sensor outputs, or messages from other programs/threads. In the context of the video, it is the core of the system design that enables real-time interactions and scalability. The script mentions how this architecture can be used to build a responsive and maintainable system that can handle various services like checkout, inventory, shipping, and contact in a retail application.

💡Reactive manifesto

The reactive manifesto is a set of principles that guide the design of reactive systems, aiming to ensure they are responsive, resilient, elastic, and message-driven. It is referenced in the script as the foundation for building systems that can adapt to varying workloads and are capable of handling failures gracefully. The video discusses how these principles are applied to create systems that are both scalable and resilient, like using Kubernetes for distributed systems.

💡Asynchronous messaging

Asynchronous messaging is a communication method where the sender of a message does not wait for a response before continuing its execution. It is a key aspect of the reactive manifesto and event-driven architecture. The script explains that this approach allows for a system design that can scale and is resilient, as it does not rely on immediate responses from other services, which is crucial for building responsive systems.

💡Scalability

Scalability refers to the ability of a system to handle a growing amount of work by adding resources or by improving the efficiency of the system. The video emphasizes the importance of scalability in system design, where the hardware can expand or contract based on the workload. This is particularly important in cloud environments and is a core principle of the reactive manifesto.

💡Resilience

Resilience in system design means the capacity of the system to recover gracefully from failures and continue to function without any single point of failure. The script discusses how this is achieved through distributed systems and Kubernetes, ensuring that if one part of the system fails, the rest can continue to operate effectively.

💡Event

In the context of the video, an event is defined as an immutable statement of fact about something that happened in the past. It is a fundamental concept in event-driven architectures, where events are used to trigger actions or reactions in the system. For example, a checkout event in a retail application would trigger updates in inventory, shipping, and contact services.

💡Event log

An event log is a record of all events that have occurred within a system. The video script mentions how event logs can be replayable, providing a history of events that can be used for various purposes, such as debugging, auditing, or restoring system state. This is a key feature of event-driven systems, allowing for persistent data storage and historical analysis.

💡Data persistence

Data persistence refers to the storage of data in a way that it can be retrieved and used even after the original process that created it has completed. In the script, it is contrasted with transient data, where data is only available for a short time. The video explains how event-driven architectures can provide permanent or possibly permanent data persistence, which is essential for system reliability and historical data analysis.

💡Pub-sub communication

Publish-subscribe communication, or pub-sub, is a messaging pattern where the sender (publisher) of information sends data without knowing who, if anyone, will receive it (subscriber). The video script uses this concept to explain how microservices can be loosely coupled in an event-driven architecture, allowing new services to be easily plugged in and consume or produce data from streams without affecting existing services.

💡Serverless function as a service

Serverless function as a service (FaaS) is a type of cloud computing service that allows users to deploy and run small pieces of code without the need to manage the underlying infrastructure. The script gives an example of how an IoT device, like a robot vacuum cleaner, can trigger a serverless function to perform a task, demonstrating the flexibility and scalability of event-driven architectures.

💡Stream processing

Stream processing is the act of performing real-time processing on data streams. The video script explains how stream processing, built on top of the Apache Kafka Streams API, can be used for real-time analytics and decision-making in applications like music streaming services or ride-sharing apps. It allows for immediate reactions to data as it is received, enhancing user experience and system efficiency.

Highlights

Event-driven architectures enable real-time user interactions, pluggable microservices, and extensible streaming and data analytics.

The reactive manifesto provides a cohesive approach to system design with guidelines for message-driven, scalable, and resilient systems.

Asynchronous messaging is a core principle for creating scalable and resilient system designs.

Distributed systems or Kubernetes are implied for scalable and resilient system design to avoid single points of failure.

Responsive systems are valued for their ability to maintain and update easily, as well as their extensibility.

An event is defined as an immutable statement of fact about something that happened in the past.

Messaging models in retail applications allow for direct communication between services like checkout, inventory, shipping, and contact.

Eventing allows for a system that is highly scalable, with the ability to expand to numerous services without affecting the producer.

Event logs provide a replayable stream history, offering a permanent or possibly permanent data persistence.

The event backbone is the central system through which all events run, enabling various applications to produce and consume events.

Event logs can trigger actions, such as spinning up a serverless function in response to an IoT device event.

Stream processing allows for real-time calculations and the implementation of business rules and policies.

Data lakes save all data that comes through the system, powering data-heavy applications like AI.

Pub-sub communication allows for pluggable microservices that can easily integrate into the system without disrupting other applications.

New microservices can consume from the beginning of the application's stream history, providing a complete context.

Event-driven architectures build on the reactive manifesto ideas, offering a responsive and message-driven system design.

Transcripts

What are event-driven architectures? And how can  they enable real-time user interactions, pluggable  

microservices, and extensible streaming and data  analytics? My name is Whitney, I'm on the Cloud  

team here at IBM. Before I dig in on event driven  architecture, specifically I'd like to talk about  

system design, more generally and specifically  the reactive manifesto. The reactive manifesto  

is a set of community-driven guidelines that are  intended to give a cohesive approach to systems  

design. So, at the core of the reactant manifesto  we have, we want our system to be message driven.

And so, specifically asynchronous messaging.

And then also, we want our our  system design to be scalable,

and we want it to be resilient.

So this implies distributed systems or Kubernetes.  So by scalable we mean that we want the  

hardware used to expand as the workload  expands and contract as the workload contracts  

and by resilient we mean we we don't  want any single point of failure  

and if the system does fail we want it to be able  to tolerate tolerate that elegantly so with this  

foundation in place we should be able to build a  system that is responsive responsive is the value

on top of that we can expect  a system that is maintainable

so it means it's easy to change easy to fix a bug  or do an update and we want one that is extensible

so we should be able to expand the system easily  

so now that we have the core of what we  want our systems designed to look like  

now let's talk about let's level set  here and talk about like what is an event  

so an event is a statement of fact an immutable  statement of fact about something that happened  

in the past so let's give that some context  we'll talk about a retail application  

so a retail application has a checkout service  

and that checkout service is going to want  to communicate with an inventory service

a shipping service and a contact service

so with the messaging model if the inventory wants  to know what the checkout is doing the checkout  

will send a message directly to inventory  to let the inventory know a checkout happen  

and another one directly to shipping and directly  to contact so with messaging we have targeted

delivery um not only that but inventory  can maybe send a message back to  

checkout it can be a conversational  delivery conversational messaging  

and then finally let's talk about where that  message lives before it's sent it's it's  

lives on the host service or on the producing  service when it's received it lives on the um  

it lives on the host service until it's received  by the consuming service or in the case of a  

message broker it might sit on a machine in  between but regardless the the data persistence is

transient

now let's compare that to eventing so if  our checkout service is producing events  

it's producing an event every time a checkout  happens without any concern to which service  

is consuming that and so with that we have a  system that's highly scalable so we have um  

three services here interested in the checkout  but even if we expand it to a hundred services  

listening to check out that wouldn't affect  what the checkout service is doing in any way  

not only that when we have all the checkouts  in a row we come up with an event log  

and it can be replayable  so we have a stream history  

and a history of everything that that  every checkout event that has happened

and then our data persistence depending on how the  cluster is set up it can be set up to be immutable  

it can be there forever as long as you have the  physical hardware to back it up so we have um  

instead of having transient data persistent  so we have uh permanent or possibly permanent

not immutable but permanent data persistence e and  t um so this is the foundation of our event driven  

architecture so we have a resilient a reactive  responsive system design and then that message  

driven element is is event messaging specifically  so when we go into our proper event design the  

first thing we're going to consider are our event  producers so we might have a web application  

that produces events and we  might have a a mobile application

and let's say we also have an edge device that's  producing events so uh internet of things device

and all of these are producing  events into our system so

so this piece of the system here that  has all the events running through it  

this is called the event backbone

so what does this system enable  

well first of all we have our event logs being  produced by all the producing applications

so the event logs can do a few things first it  can trigger an action so if we have an internet  

of things device let's say our internet of things  device is a robot vacuum cleaner and when we turn  

on that robot vacuum cleaner it spins up a pod  on the infrastructure and that pod is a function  

as a service so function as a service is a modular  piece of code often associated with an edge device  

that doesn't exist on a server on the server until  it is turned on and that's when it is spun up so  

that's called serverless function as a service  is sits on top of serverless infrastructure  

but the point is that event triggers an action  of the with the robot vacuum cleaner to spin up  

bounce all around the house make everything  squeaky clean and then when the function is  

finished running it powers down and perhaps  that'll cause our function as a service to send  

another event back into the system to let  it know that the the function is complete  

another cool thing about the event log is it  can be used to help optimize and customize data  

persistence so if our web application is our maybe  it's our retail application that we talked about  

before and it's doing a stream of checkout events  well our inventory service can be reading from  

that and it's keeping track of a local data  store that's keeping track of the inventory  

so our inventory service will consume from  that data stream from the web application it'll  

modify the local data and then it'll produce  again into the event backbone and so this new  

stream is giving the most current inventory  to any other application in the system that  

wants to consume from it and so while we're  talking about data another thing that happens  

we can save all of the data that comes through the  system so all the raw data or all of the transform  

data here at the end once it goes through the  system we'll save it all in a huge data lake

and this is what's going to power  really data-heavy applications  

like ai for example and the data lake is  accessible from anywhere in the system  

that any service that you let consume from it

so another thing that our event-driven  architectures can enable are a stream processing  

so this is built on top of  the apache kafka streams api  

so let's say that our mobile app is producing  let's say it's a music streaming service and  

every time let's say up until your in your whole  life up until this point you've been listening to  

only miley cyrus and britney spears but today you  decide you know what i want to see what jay-z is  

about and you turn jay-z on well immediately  your streaming app is showing you kanye west  

and lauren hill and giving you ideas about what  you might want to listen to next so that stream  

processing is a really powerful and what gives  the gives real-time calculations like if it's a  

ride share app you um that you know exactly how  far away your ride is but it can also be used to  

implement business rules and business policies  and it can be used for real-time analytics

and then let's also talk about with  our system the communications layer

so specifically it's called pub sub communication  

for publish subscribe so we could have any  microservice a containerized microservice

and just like we talked about before with it being  extensible we can very easily plug it into we can  

write a new microservice and plug it into our  system and immediately have it start consuming  

from any stream and producing to a new stream or  even producing to a stream that already exists  

and none of the other applications need  to be concerned with it at all so it's  

highly plugable not only that but because  of the stream history we talked about before  

when you plug it into the system you don't have  to have it start consuming from right this moment  

it can be consuming from as long as from the  beginning of your application if you've been  

saving your stream history for that long so  as long as that stream history persists when  

you plug in a new system it can consume from  either the old data or can be consumed from  

the present moment so with that we talked about  the major benefits of event driven architecture  

that sit on top of the reactive manifesto ideas  for system design with eventing at its core