The Pattern You MUST Learn in .NET

Summary

The video introduces the transactional outbox pattern, an important design pattern in messaging and microservices. It allows atomic operations between a database and a message queue by using a database transaction to make changes and write events to an outbox table. A background process then pushes those events to a message queue. The video demonstrates implementing this pattern simply in .NET with MassTransit to achieve reliable messaging. It also discusses alternatives like distributed transactions and the presenter's preferred approach of using serverless architectures.

Takeaways

• 💬 The video introduces the transactional outbox pattern in messaging within .NET applications, focusing on its importance for ensuring reliable message delivery in distributed systems.
• 🔥 Demonstrates the use of Amazon SQS (Simple Queue Service) and MassTransit for implementing the outbox pattern, but emphasizes that the approach is not limited to SQS and can work with other messaging services like RabbitMQ or Azure Service Bus.
• 📦 Explains that the outbox pattern helps solve the problem of ensuring that a database operation and a message publication to a queue are treated as a single atomic operation, preventing data inconsistencies.
• 🛠️ Outlines the steps to implement the outbox pattern in .NET, including the addition of specific MassTransit and Entity Framework Core packages and configuration changes in the application.
• 📈 Uses a practical example of a customer API to show how the outbox pattern can be applied to a real-world scenario, including creating, updating, and deleting customer records and publishing messages related to these events.
• 💻 Highlights the importance of idempotence in message processing to handle potential duplicate messages that could arise due to the nature of distributed systems and the outbox pattern.
• ✅ Provides code snippets and explanations for setting up the transactional outbox pattern, including database migrations for outbox-related tables and configuring MassTransit to use these tables.
• 📚 Mentions the availability of a complete code example in the video description, offering viewers a resource to explore the implementation details further.
• 📲 Discusses alternative approaches to solving the problem addressed by the outbox pattern, hinting at the presenter's preference for a different method based on his experience with cloud and serverless architectures.
• ❓ Encourages viewers to share their own experiences and approaches to implementing reliable messaging patterns in their .NET applications, fostering a community dialogue.

Q & A

What is the main topic of the video introduced by Nick?

The main topic of the video is the transactional outbox pattern in messaging systems, particularly its application in .NET environments.

Why is the outbox pattern considered important in messaging systems?

The outbox pattern is considered important because it ensures reliable messaging and transactional consistency between database operations and message queuing, which is crucial for distributed systems.

What messaging service is used as an example in the video?

Amazon SQS (Simple Queue Service) is used as the queuing mechanism example in the video.

How does the transactional outbox pattern help in messaging systems?

The transactional outbox pattern helps by storing messages in a database outbox as part of the same transaction as the business operation. This ensures that messages are reliably published to the message queue once the transaction commits, avoiding data inconsistencies.

What is MassTransit and how is it used in the context of the video?

MassTransit is an abstraction layer over queueing mechanisms or pub/sub systems that simplifies messaging in .NET applications. In the video, it's used to demonstrate the implementation of the transactional outbox pattern with various transport layers, like Amazon SQS.

Why is the outbox pattern necessary, according to the video?

The outbox pattern is necessary to ensure atomic operations between database changes and message publishing, solving the problem of ensuring both occur successfully or neither, to maintain data integrity across distributed systems.

What problem does the transactional outbox pattern solve regarding message publishing?

It solves the problem of ensuring reliable message delivery even when an application is scaled out, by atomically storing messages in a database outbox before publishing them to the queue, thus avoiding lost messages or inconsistencies.

What alternative approaches to the transactional outbox pattern are mentioned?

While specific alternatives are not detailed, the video mentions that there are other options if one does not want to use the outbox pattern, implying the existence of different messaging and transaction consistency strategies.

How does MassTransit support the transactional outbox pattern?

MassTransit supports the transactional outbox pattern through integration with Entity Framework Core, allowing for the configuration of outbox and inbox state entities to manage message transactions and reliability.

What is idempotency, and why is it important in the context of the outbox pattern?

Idempotency ensures that even if a message is delivered or processed multiple times, the effect on the system is the same as if it were processed once. It's important in the outbox pattern to handle duplicate messages without causing unintended effects.

Outlines

00:00

📢 Introduction to Transactional Outbox Patterns

Nick introduces the concept of the transactional outbox pattern in messaging within .NET applications, emphasizing its significance for ensuring reliable message delivery in distributed systems. He explains the rationale behind using the pattern, its operational dynamics, and alternatives for those opting not to use it. The video showcases the use of Amazon SQS for queueing mechanisms, supported by MassTransit for abstraction across different messaging systems. Nick also highlights a free AWS-sponsored course for developers interested in AWS services, alongside a demonstration involving a customer API utilizing a PostgreSQL database, illustrating basic CRUD operations and the initial setup for implementing the outbox pattern.

05:01

🔍 Implementing the Transactional Outbox Pattern

This section delves into the technical implementation of the transactional outbox pattern. Nick outlines the process of integrating messaging functionality into an application using MassTransit and Amazon SQS. He explains the challenges associated with ensuring atomic operations between a database and a message queue, proposing the transactional outbox as a solution for achieving atomicity within the database's scope. By introducing an additional outbox table and leveraging a background service to process these entries, the application can maintain data consistency and reliability across distributed components, addressing the common issues of message duplication and ensuring idempotency in message processing.

10:02

🛠️ Advanced Configuration and Usage of the Outbox Pattern

Nick further explores the transactional outbox pattern by adding advanced configurations using Entity Framework Core with MassTransit. He guides through the setup process, including creating necessary tables and configuring the application to support the outbox pattern effectively. This involves adjustments in the application's database context and MassTransit setup to ensure messages are correctly captured in the outbox and processed reliably. Nick demonstrates the practical application of these configurations with a code walkthrough, emphasizing the seamless integration of the outbox pattern into existing application workflows without significant code alterations.

15:02

🧪 Demonstrating the Outbox Pattern in Action

In this segment, Nick provides a live demonstration of the outbox pattern, showing how messages are handled transactionally within a database before being published to a message queue. He walks through the entire flow, from database modification to message consumption, illustrating the practical benefits and solving common messaging challenges. This part emphasizes the importance of atomicity, reliability, and idempotency in distributed messaging systems. Nick's demonstration includes debugging sessions and detailed explanations of how the outbox pattern ensures that messages are only published after successful database transactions, enhancing application reliability.

20:03

🔄 Conclusion and Alternative Approaches

Nick concludes by summarizing the benefits of the transactional outbox pattern and offering insights into alternative approaches for managing distributed messaging challenges. He invites viewers to share their own experiences and preferences for messaging patterns, fostering a community discussion on best practices. Despite the focus on the transactional outbox pattern, Nick hints at his preferred methods for handling messaging in cloud and serverless architectures, suggesting the potential for future content exploring these alternatives. The video wraps up with an encouragement for viewer engagement and continued learning in the domain of distributed messaging systems.

Keywords

💡Outbox pattern

A pattern for reliably publishing messages or events after a database transaction succeeds. It inserts outgoing messages into an outbox table that is processed asynchronously. This guarantees reliable delivery. It is used in the video when adding a customer to ensure downstream processes are notified.

💡Transactional outbox

A type of outbox pattern that utilizes database transactions to atomically update state and enqueue messages for downstream consumers. This ensures consistency between databases and message queues.

💡Message queue

An asynchronous messaging system used to decouple and distribute processing across microservices. The video uses SQS but the outbox pattern works for any message queue like RabbitMQ.

💡Idempotency

The ability to handle duplicate messages such that processing a message multiple times yields the same result as processing it once. The video recommends building this into message consumers to account for potential redelivery.

💡MassTransit

A .NET framework for distributed application coordination over message queues. It is used in the video to implement the transactional outbox pattern by publishing messages after database commits.

💡Entity Framework Core

An object-relational mapper used in .NET to interface with databases. The video leverages EF Core to manage the outbox database tables required for the transactional outbox pattern.

💡Publisher

The component that constructs an outgoing message and adds it to the outbox after mutating application state. This occurs when creating a customer in the video.

💡Consumer

The component that processes messages delivered from the outbox to the message queue. The video has a worker service with defined consumers.

💡Atomicity

The transactional guarantee that a series of database operations either completely succeeds or fails. Utilizing the outbox pattern allows atomic writes to the outbox alongside business data.

💡Decoupling

The separation of components so changes in one component do not directly impact others. The video decouples the customer API from downstream message processing.

Highlights

The transactional outbox pattern is one of the most important patterns you need to understand and know how to use in messaging

The outbox pattern allows database changes and message publishing to happen atomically in a transaction

The outbox pattern writes database changes and a message to publish to an outbox table transactionally

A separate background process reads the outbox and publishes the messages asynchronously

The outbox pattern avoids race conditions between database commits and message publishing

MassTransit natively supports the transactional outbox pattern with Entity Framework Core

The consumer service needs the outbox dependencies to process messages published from the outbox

The publish call does not actually publish a message, it just marks the outbox message entity to publish

The outbox message is published when SaveChanges is called, enclosing the database commit and publish in one transaction

The outbox background process can be disabled to process messages offline in batches

The outbox pattern can work with any messaging provider like RabbitMQ by changing the MassTransit configuration

The code looks like separate database and messaging calls but the outbox makes it a single transaction

The presenter has a different preferred approach to the problem using serverless technologies

Leave comments if you want to see a video explaining the presenter's serverless approach to the problem

Let the presenter know in the comments if you use a different approach to implement the outbox pattern

Transcripts

hello everybody I'm Nick and in this

video I'm going to introduce you to the

extremely highly requested topic of

outbox patterns in messaging Inn net

most specifically in this video I'm

going to show you the transactional

outbox pattern which is one of the most

important patterns you need to

understand and know how to use in

messaging in this video I'm going to

explain what it is how it works why we

need it and I'm also going to talk about

some Alternatives if you don't want to

use that pattern what other options do

you have in this video I'm going to be

using Amazon sqs which which is the

queuing mechanism in Amazon and this

video is sponsored by AWS so massive

thank you to them however this is not

only applicable to sqs because we're

going to be using mass transit to show

all this any transport or any backing

service we'll use whether that's rabid

mq or Azo service bus or anything else

will work exactly as you see it now AWS

has also sponsored a dome train course

so if you want to get started with AWS

Services as a cop developer click the

link in the description it's absolutely

free and yours to keep forever if if you

like of content and you want to see more

make sure you subscribe for more

training check out my courses on doet

tr.com okay so let me show what I have

here I have this customer API over here

it has one controller we can create get

get all update or delete a customer and

behind the scenes this is using a

database which I am running here in

Docker and this is a post Christ

database to store all the users I'm

creating so if I go over here and I say

run this API let me show you what I have

and here I can see my database and as

you can see I have a single user so if I

go to insomnia I can say get all

customers and I'm going to get this

customer back over here so I'm going to

get it I can get a customer by ID so I

can copy this ID and I can go here and

paste it at the end of the URL and I'm

going to get it if it is wrong I'm going

to get a 404 and I can also create a

customer as you can see over here so

let's create Nick again and this is all

created and you can see the created

users now I'm going to go quickly and

delete these customers now here's what

happens in most most applications when

you want to create a resource or update

or delete or basically mutate some state

which it's mutation might be important

for some other aspect of your business

for example when you create a customer

you also might want to quue an email for

email verification for example or

anything else any aspect of the business

and any team of the business might need

to know that the customer was created

updated deleted and so on so you can

subsequently for example delete more

information about them for gdpr purposes

or anonymize it well you might need to

do many things now I could just go to my

service over here and say okay here's

where I create my customer as I'm adding

them into the database and saving them I

can also say send email I can also say I

don't know call some API I can do many

many things but the problem I have here

is that this method now is doing too

much and I get very slow and the only

thing that's important for the customer

is that it was added to the database and

that is it they get the response they go

their merry way and hopefully my

application behind the scenes can do all

the other things now people have tried

naive ways for years to do this for

example they're spinning up a background

thread and they're putting this

functionality in the thread but what

happens if your application is scaled

out what happens if this is a micros

there's so many moving parts so a great

way to deal with this is to use a queing

mechanism and use asynchronous messaging

and either publish an event where people

can subscribe and listen to that or you

can just send it to a specific queue and

that queue can do something later for

example queue that email now let's see

how we would Implement something like

that now before I move on I'd like to

let you know we just launched a brand

new course on D train called getting

started with modular monoliths in net

and this authored by the legend Steve

adalis Smith I'm sure Steve has taught

many of you already with courses on

other platforms like prite but now he

authored his first of many courses on

dome train and it's all about how to get

started with modular monoliths not only

will he teach teach you the theory

behind the concept and how it compares

to microservices or traditional

monoliths but he will also build a whole

system in that course hands on with code

and diagrams and examples you can follow

along it is an amazing course and it is

the best way to get started with modular

monol hands down in.net now to celebrate

the launch I'd like to offer the first

500 of you a 20% discount so either use

a link in the description or apply code

modular at checkout to claim that 20%

off it's a great opportunity to get

started with a concept and I can't

stress enough how much of an amazing

author Steve is now back to the video

I'm going to go to the program.cs and as

you can see over here I've already added

and you get package called mass transit

now I've already covered mass transit in

net in a previous video so I'm going to

link that below if you want to see how

to get started with mass transit this

assumes you sort of understand what mass

transit is but if you don't all it

really is it's an abstraction layer over

queing mechanisms or Pub Subs such as

rabbit mq or maybe sqs Aus service bus

this sort of thing so you code against a

common abstraction and then behind the

scenes M Transit can Implement all those

other transports specifically so the

only thing that changes is me saying M

transit. Aur service bus for example to

go from sqs to service bus and so on

we've seen all that in the previous

video so now that I've configured M

Transit here and I'm saying hey use this

sqsq over here and configure all the end

points all I need to do to introduce the

messaging functionality is go to my

service and say private read only I

publish endpoint so I'm going to use a

publish endpoint here which is scoped by

default in registration and I'm going to

take it and I'm going to say oh here is

where I want to publish a message and

how convenient I have a few message

contracts over here so this is the

record representing a customer being

created so I'm going to go here and I'm

going to say VAR message equals new

customer created and you can have a

mapper if you want all I'm going to do

is I'm going to manually map this user

over here so ID full name email and date

of birth and then what I'll say is await

the publish endpoint and then publish

that message and that is it and that

then we'll publish it into the queue and

just to see all that working I'm going

to stick a breakpoint here and just say

debug my application so I'm using AWS I

have the toolkit over here to see all my

messages in the sqs you can see the cues

are automatically created because must

Transit will do that and I'm going to go

ahead and say create a customer so send

that request I'm hitting my breakpoint

I'm going here I'm saying add the user

to the database and then I get my

message and over here I am publishing

that message so if I go to customer

created the queue and I view messages

you can see it over here it's already

published and you can see everything

about it over here and then after it's

published I also saved to the database

and thankfully it was saved successfully

and eventually I have as you can see

over here a consumer and if you want to

grab this code you can use a link down

below to get it for free but I can go to

this worker over here which has some

very simple configuration and it has

consumers for every type of message in

this case all it's really doing is it's

logging don't do logging this way by the

way this is not the right way to do

logging I'm just doing it like this

because I want to print something to the

console to show that I consumed it so

I'm going to run the worker and as

you're going to see very quickly we're

going to see that the customer created

message has been consumed if I go ahead

and I create a second customer you will

see immediately that another request has

been filed and it has been processed

through a new message fantastic very

easy messaging in the app now do you see

where the problem is with this you might

have spotted it the problem is that I'm

publishing this message which by the way

it could also be below the creation so

we sort of ensure that something was

created and only if it was created maybe

let's say result is more than uh zero

only then we publish the message which

is a nice safe guard to make sure we

don't publish a message for something

that wasn't done but what guarantees

that yeah okay this was added to the

database but do we know for a fact that

this was published to the queue we don't

we can't yes there are ways to get

around this with distributed

transactions and two-phase commits but

this is a very very hard problem in

general in applications because you

can't have Atomic operations between the

database you're using and the message

bus you're publishing towards and then

what do you do if this failed you roll

back and what if the roll back fail like

this such a big problem and chicken and

neck problem you have to solve here and

there's no obvious easy solution but but

the truth is there actually is it's both

very easy to implement and pretty

obvious you see we cannot have an atomic

operation between a database and a

message CU but we can have an atomic

operation in a database you can have a

transaction that does many things it

adds the customer and then it adds the

instruction to do something about the

added customer so what instead of

writing in a single table the customer's

table over here we also had a second

table called the outbox table and

there's actually more into that behind

the scenes but let's in theory just say

we have the outbox table saying that hey

a customer was created and then a

service picks that up after it's been

transactionally written and it has to

push it into a queue and then that queue

is read by a consumer and that is

published so if I was to use a very

scientific tool to demonstrate this I

can go here and I can say that instead

of writing into a single place over here

which in this case this is the customers

table let's also in the same transaction

because now we are on a database like I

said write something to the outbox then

have something and we will see what that

something is listen to that outbox and

see hey do you have any messages for me

to process and then that thing will not

only read those messages but also it is

going to publish them as we're going to

see here to AQ so take that message and

publish it now there is a bit of an

issue here where you can have the

situation where if this is trying to

read from the outbox and then get the

message and publish it what if the

publishing fails well you can update the

state of the outbox and say that this

message hasn't been processed yet and

only say that it has after you guarantee

that it is been pushed into the message

box the usual problem with all this is

that in some implementations you cannot

guarantee exactly one delivery with this

process so you need to have IDM potency

baked into your message processing

meaning that if a message is pushed

twice into the queue or consumed twice

your application only makes one action

not two having worked in finance dealing

with payments this was the bread and

butter for everything we did you don't

want to have a message or an event

handled twice because if you do someone

might pay twice

and people don't like paying twice if

they have only paid once now you could

definitely go and manually Implement all

this but you don't have to because mass

transit natively supports the outbox

pattern the transactional outbox pattern

on top of an inmemory one which I'm not

going to cover so how will we introduce

the pattern here well first what I'm

going to do is I'm going to move this

message publishing between the action

I'm taking on the database the addition

and before the save change is a sync and

I'm going to paste it here and I'm still

saying publish so the code I have here

will publish into the queue nothing

about the database as it is now I'm

going to go to my API new packages and

I'm going to say Mass transit. entity

framwork core and this is natively

supported with ENT framework you can

just add ENT framework just for this you

don't have to use it for your whole

application but to simplify management

of this whole process M Transit chose to

use EF core which by the way is fine

it's performance it's fast it is great

it's net 8 we're good and once I do that

I can go to my app DB context and I have

to specify in the model that there are

some tables that need to be created for

example I'm going to say override on

model creating and I'm going to add

three lines first I'll say model

builder. add inbox State entity then

model builder. add outbox State entity

and I'm also going to say modelbuilder

do add outbox message entity those three

things is what I need and once I specify

that I need a bit more configuration on

the admass transit method so what I'm

going to say in there is x. add Entity

framework outbox and I'm going to

specify my DB context so add DB context

some configuration over here and I'm

going to say o do use postgress and also

use bus outbox and I'm going to leave

the defaults for now I will also specify

here in this service o do query delay

and I'm going to set that to 1 second

I'm going to explain what that is in a

second I'm doing this for demo purposes

so we can have all the actions happen

quickly once I do that I want to create

migrations for this new feature because

there will be tables needed and these

tables are configured through this so

I'll say net EF migrations add outbox

and let's get that created here we go

migrations created so if I go here I can

see all the migrations and all the the

code needed for those new tables and

then I can say netf database update and

once I do that we're going to get all

those tables created if I refresh the

tables you're going to see inbox State

outbox message and then outbox state so

that will now handle our outbox pattern

and now that's it I'm not going to touch

anything else about my code meaning that

this publishing the dot publish that's

supposed to push into a que won't be

touched now let me quickly show you what

you need to do on the consumer side of

things first we need to add a couple of

packages so first we need the must

Transit entry framework core package we

added in the main API project and then

we'll also add the postgress um an

framework core then we're going to

create an app DB context it doesn't have

to be the same as the main application

this is just here for the outbox this

means that there's no other DB context

needed in here or DB set all you need is

a traditional model creation that is it

now back to the program.cs in the add

mass transit call we're going to say add

anti framework outbox appb context as in

the other application and here all we're

going to say is we're going to have a

duplicate detection window to detect

duplicate messages from the inbox so I'm

going to have that as time span Dot from

seconds 30 and then I'm going to say hey

use postgress but nothing about the bus

outbox just use postgress and that is it

now for all this to work I also need to

wire up of course the app DB context so

I'm going to say add DB context as I

would normally do app DB context and

then configure that to use postris and

that is it all I'm going to do now is

I'm going to run my worker which is

supposed to be listening to the cues and

as you can see it is listening to all

the cues and then I'm going to exactly

as it is I want to remind you I did not

touch the publishing message I'm going

to do is I'm going to stick a breakpoint

and run the API and let me show you what

happens in fact I'm going to remove the

breako for now and first I'm going to

publish a message so I'm going to go

straight over here into insomnia and I'm

going to say create a customer so

customer created a few things are

happening here we don't really care

about that but if I go to the worker as

you can see we did process the worker

because now we have the customer created

consumer message and all the details

that's it so nothing seems to have

changed because it still seem like I

just published into a que so what

problem did I actually solve well let me

show you I'm going to go to the customer

service and I'm going to slap a

breakpoint on the add a sync method

which I want to remind you in anyti

framework core doesn't take any action

it just marks The Entity to be added

when you call Save changes which again

is transactional so let's go ahead and

say create another customer and let's

hit that breakpoint we marking that

entity and then normally on this publish

call we would publish something into the

queue directly going to step over that I

haven't called save changes a sync yet

but I want to show you we have nothing

in the queue let me just save you

messages the queue is empty and you

might expect that because the worker is

running so you're assuming I publish

something and it's been consumed by the

worker the truth is it hasn't there is

no message consumption because nothing

has been published yet the moment

however I say save changes sync then and

only then something will happen and what

will happen is in the database if I go

to the outbox State you're going to see

the state of an outbox message and all

its details and then in the outbox

message you're going to see a new entry

over here which just was added with this

save async call alongside my new

customer which contains in the body the

whole message serialized so now the

moment I say run freely don't stick to

that break point and don't stay there to

my service my service behind the scenes

will use a bus running in the background

the hostage service running in the

background and push those messages and

as you can see it's not here anymore and

push that into the queue and my worker

then will pick it up and as you can see

it has been processed that's the whole

point of the outbox we are doing our

action transactionally and we're adding

an entry into the outbox and then later

service picks it up and it pushes it

into the message so both things happen

automically it is implemented in a great

way in mass transit is very simple to

work you don't need to change your code

this all still looks like two

independent calls but behind the scenes

this is all transactional now you could

stop that worker service that picks the

outbox and publishes it if you go over

here and just to remind you this query

delay is a delay that the service that

will read the messages in that outbox

and take them and push them will have to

perform so we don't just Spam the

database with many many requests per

second basically so in this whole thing

I could also say that disable that

delivery service that background service

logic because you just want to use the

adbox and you might want to have offline

processing so there's no que there's no

bus working you might also want to go

further and say disable the inbox

cleanup service you have full control

and you can even run this service that

looks into the database and pushes

messages into the queue to run out of

this process so you split that

responsibility to another service and

then you have the worker that still

consumes straight from the queue but it

acknowledges that the Out book exists so

it can also mutate the state as needed

and like I said if you wanted to use any

other provider with any of this you

could easily just go here and say mass

transit. rabbitmq for example remove

Amazon sqs why that up and the only call

that needs to change is the using

whatever you want to use r mq or any

other provider you want in this case sqs

is what I've used in production

extensively so that's the one I'm using

and that's it your code now just

supports the transactional outbox

pattern again you're going to find all

this code in the description down below

it's all for free and if you have any

questions please leave them in the

comments down below now something I want

to point out is that this is not my

favorite way to deal with this problem I

have a different approach having worked

in the cloud and in serverless for an

extensive period of time we actually

deal with this problem differently and

if you want to know how we deal with

that problem leave a comment down below

and I will make a video on that but now

when from you what do you think about

this and are you using something else to

implement it maybe you're using your own

approach leave a comment down below and

let me know well that's all I had for

you for video thank you very much for

watching and as always keep coding