The Pattern You MUST Learn in .NET
Summary
TLDRThe video introduces the transactional outbox pattern, an important design pattern in messaging and microservices. It allows atomic operations between a database and a message queue by using a database transaction to make changes and write events to an outbox table. A background process then pushes those events to a message queue. The video demonstrates implementing this pattern simply in .NET with MassTransit to achieve reliable messaging. It also discusses alternatives like distributed transactions and the presenter's preferred approach of using serverless architectures.
Takeaways
- 💬 The video introduces the transactional outbox pattern in messaging within .NET applications, focusing on its importance for ensuring reliable message delivery in distributed systems.
- 🔥 Demonstrates the use of Amazon SQS (Simple Queue Service) and MassTransit for implementing the outbox pattern, but emphasizes that the approach is not limited to SQS and can work with other messaging services like RabbitMQ or Azure Service Bus.
- 📦 Explains that the outbox pattern helps solve the problem of ensuring that a database operation and a message publication to a queue are treated as a single atomic operation, preventing data inconsistencies.
- 🛠️ Outlines the steps to implement the outbox pattern in .NET, including the addition of specific MassTransit and Entity Framework Core packages and configuration changes in the application.
- 📈 Uses a practical example of a customer API to show how the outbox pattern can be applied to a real-world scenario, including creating, updating, and deleting customer records and publishing messages related to these events.
- 💻 Highlights the importance of idempotence in message processing to handle potential duplicate messages that could arise due to the nature of distributed systems and the outbox pattern.
- ✅ Provides code snippets and explanations for setting up the transactional outbox pattern, including database migrations for outbox-related tables and configuring MassTransit to use these tables.
- 📚 Mentions the availability of a complete code example in the video description, offering viewers a resource to explore the implementation details further.
- 📲 Discusses alternative approaches to solving the problem addressed by the outbox pattern, hinting at the presenter's preference for a different method based on his experience with cloud and serverless architectures.
- ❓ Encourages viewers to share their own experiences and approaches to implementing reliable messaging patterns in their .NET applications, fostering a community dialogue.
Q & A
What is the main topic of the video introduced by Nick?
-The main topic of the video is the transactional outbox pattern in messaging systems, particularly its application in .NET environments.
Why is the outbox pattern considered important in messaging systems?
-The outbox pattern is considered important because it ensures reliable messaging and transactional consistency between database operations and message queuing, which is crucial for distributed systems.
What messaging service is used as an example in the video?
-Amazon SQS (Simple Queue Service) is used as the queuing mechanism example in the video.
How does the transactional outbox pattern help in messaging systems?
-The transactional outbox pattern helps by storing messages in a database outbox as part of the same transaction as the business operation. This ensures that messages are reliably published to the message queue once the transaction commits, avoiding data inconsistencies.
What is MassTransit and how is it used in the context of the video?
-MassTransit is an abstraction layer over queueing mechanisms or pub/sub systems that simplifies messaging in .NET applications. In the video, it's used to demonstrate the implementation of the transactional outbox pattern with various transport layers, like Amazon SQS.
Why is the outbox pattern necessary, according to the video?
-The outbox pattern is necessary to ensure atomic operations between database changes and message publishing, solving the problem of ensuring both occur successfully or neither, to maintain data integrity across distributed systems.
What problem does the transactional outbox pattern solve regarding message publishing?
-It solves the problem of ensuring reliable message delivery even when an application is scaled out, by atomically storing messages in a database outbox before publishing them to the queue, thus avoiding lost messages or inconsistencies.
What alternative approaches to the transactional outbox pattern are mentioned?
-While specific alternatives are not detailed, the video mentions that there are other options if one does not want to use the outbox pattern, implying the existence of different messaging and transaction consistency strategies.
How does MassTransit support the transactional outbox pattern?
-MassTransit supports the transactional outbox pattern through integration with Entity Framework Core, allowing for the configuration of outbox and inbox state entities to manage message transactions and reliability.
What is idempotency, and why is it important in the context of the outbox pattern?
-Idempotency ensures that even if a message is delivered or processed multiple times, the effect on the system is the same as if it were processed once. It's important in the outbox pattern to handle duplicate messages without causing unintended effects.
Outlines
📢 Introduction to Transactional Outbox Patterns
Nick introduces the concept of the transactional outbox pattern in messaging within .NET applications, emphasizing its significance for ensuring reliable message delivery in distributed systems. He explains the rationale behind using the pattern, its operational dynamics, and alternatives for those opting not to use it. The video showcases the use of Amazon SQS for queueing mechanisms, supported by MassTransit for abstraction across different messaging systems. Nick also highlights a free AWS-sponsored course for developers interested in AWS services, alongside a demonstration involving a customer API utilizing a PostgreSQL database, illustrating basic CRUD operations and the initial setup for implementing the outbox pattern.
🔍 Implementing the Transactional Outbox Pattern
This section delves into the technical implementation of the transactional outbox pattern. Nick outlines the process of integrating messaging functionality into an application using MassTransit and Amazon SQS. He explains the challenges associated with ensuring atomic operations between a database and a message queue, proposing the transactional outbox as a solution for achieving atomicity within the database's scope. By introducing an additional outbox table and leveraging a background service to process these entries, the application can maintain data consistency and reliability across distributed components, addressing the common issues of message duplication and ensuring idempotency in message processing.
🛠️ Advanced Configuration and Usage of the Outbox Pattern
Nick further explores the transactional outbox pattern by adding advanced configurations using Entity Framework Core with MassTransit. He guides through the setup process, including creating necessary tables and configuring the application to support the outbox pattern effectively. This involves adjustments in the application's database context and MassTransit setup to ensure messages are correctly captured in the outbox and processed reliably. Nick demonstrates the practical application of these configurations with a code walkthrough, emphasizing the seamless integration of the outbox pattern into existing application workflows without significant code alterations.
🧪 Demonstrating the Outbox Pattern in Action
In this segment, Nick provides a live demonstration of the outbox pattern, showing how messages are handled transactionally within a database before being published to a message queue. He walks through the entire flow, from database modification to message consumption, illustrating the practical benefits and solving common messaging challenges. This part emphasizes the importance of atomicity, reliability, and idempotency in distributed messaging systems. Nick's demonstration includes debugging sessions and detailed explanations of how the outbox pattern ensures that messages are only published after successful database transactions, enhancing application reliability.
🔄 Conclusion and Alternative Approaches
Nick concludes by summarizing the benefits of the transactional outbox pattern and offering insights into alternative approaches for managing distributed messaging challenges. He invites viewers to share their own experiences and preferences for messaging patterns, fostering a community discussion on best practices. Despite the focus on the transactional outbox pattern, Nick hints at his preferred methods for handling messaging in cloud and serverless architectures, suggesting the potential for future content exploring these alternatives. The video wraps up with an encouragement for viewer engagement and continued learning in the domain of distributed messaging systems.
Mindmap
Keywords
💡Outbox pattern
💡Transactional outbox
💡Message queue
💡Idempotency
💡MassTransit
💡Entity Framework Core
💡Publisher
💡Consumer
💡Atomicity
💡Decoupling
Highlights
The transactional outbox pattern is one of the most important patterns you need to understand and know how to use in messaging
The outbox pattern allows database changes and message publishing to happen atomically in a transaction
The outbox pattern writes database changes and a message to publish to an outbox table transactionally
A separate background process reads the outbox and publishes the messages asynchronously
The outbox pattern avoids race conditions between database commits and message publishing
MassTransit natively supports the transactional outbox pattern with Entity Framework Core
The consumer service needs the outbox dependencies to process messages published from the outbox
The publish call does not actually publish a message, it just marks the outbox message entity to publish
The outbox message is published when SaveChanges is called, enclosing the database commit and publish in one transaction
The outbox background process can be disabled to process messages offline in batches
The outbox pattern can work with any messaging provider like RabbitMQ by changing the MassTransit configuration
The code looks like separate database and messaging calls but the outbox makes it a single transaction
The presenter has a different preferred approach to the problem using serverless technologies
Leave comments if you want to see a video explaining the presenter's serverless approach to the problem
Let the presenter know in the comments if you use a different approach to implement the outbox pattern
Transcripts
hello everybody I'm Nick and in this
video I'm going to introduce you to the
extremely highly requested topic of
outbox patterns in messaging Inn net
most specifically in this video I'm
going to show you the transactional
outbox pattern which is one of the most
important patterns you need to
understand and know how to use in
messaging in this video I'm going to
explain what it is how it works why we
need it and I'm also going to talk about
some Alternatives if you don't want to
use that pattern what other options do
you have in this video I'm going to be
using Amazon sqs which which is the
queuing mechanism in Amazon and this
video is sponsored by AWS so massive
thank you to them however this is not
only applicable to sqs because we're
going to be using mass transit to show
all this any transport or any backing
service we'll use whether that's rabid
mq or Azo service bus or anything else
will work exactly as you see it now AWS
has also sponsored a dome train course
so if you want to get started with AWS
Services as a cop developer click the
link in the description it's absolutely
free and yours to keep forever if if you
like of content and you want to see more
make sure you subscribe for more
training check out my courses on doet
tr.com okay so let me show what I have
here I have this customer API over here
it has one controller we can create get
get all update or delete a customer and
behind the scenes this is using a
database which I am running here in
Docker and this is a post Christ
database to store all the users I'm
creating so if I go over here and I say
run this API let me show you what I have
and here I can see my database and as
you can see I have a single user so if I
go to insomnia I can say get all
customers and I'm going to get this
customer back over here so I'm going to
get it I can get a customer by ID so I
can copy this ID and I can go here and
paste it at the end of the URL and I'm
going to get it if it is wrong I'm going
to get a 404 and I can also create a
customer as you can see over here so
let's create Nick again and this is all
created and you can see the created
users now I'm going to go quickly and
delete these customers now here's what
happens in most most applications when
you want to create a resource or update
or delete or basically mutate some state
which it's mutation might be important
for some other aspect of your business
for example when you create a customer
you also might want to quue an email for
email verification for example or
anything else any aspect of the business
and any team of the business might need
to know that the customer was created
updated deleted and so on so you can
subsequently for example delete more
information about them for gdpr purposes
or anonymize it well you might need to
do many things now I could just go to my
service over here and say okay here's
where I create my customer as I'm adding
them into the database and saving them I
can also say send email I can also say I
don't know call some API I can do many
many things but the problem I have here
is that this method now is doing too
much and I get very slow and the only
thing that's important for the customer
is that it was added to the database and
that is it they get the response they go
their merry way and hopefully my
application behind the scenes can do all
the other things now people have tried
naive ways for years to do this for
example they're spinning up a background
thread and they're putting this
functionality in the thread but what
happens if your application is scaled
out what happens if this is a micros
there's so many moving parts so a great
way to deal with this is to use a queing
mechanism and use asynchronous messaging
and either publish an event where people
can subscribe and listen to that or you
can just send it to a specific queue and
that queue can do something later for
example queue that email now let's see
how we would Implement something like
that now before I move on I'd like to
let you know we just launched a brand
new course on D train called getting
started with modular monoliths in net
and this authored by the legend Steve
adalis Smith I'm sure Steve has taught
many of you already with courses on
other platforms like prite but now he
authored his first of many courses on
dome train and it's all about how to get
started with modular monoliths not only
will he teach teach you the theory
behind the concept and how it compares
to microservices or traditional
monoliths but he will also build a whole
system in that course hands on with code
and diagrams and examples you can follow
along it is an amazing course and it is
the best way to get started with modular
monol hands down in.net now to celebrate
the launch I'd like to offer the first
500 of you a 20% discount so either use
a link in the description or apply code
modular at checkout to claim that 20%
off it's a great opportunity to get
started with a concept and I can't
stress enough how much of an amazing
author Steve is now back to the video
I'm going to go to the program.cs and as
you can see over here I've already added
and you get package called mass transit
now I've already covered mass transit in
net in a previous video so I'm going to
link that below if you want to see how
to get started with mass transit this
assumes you sort of understand what mass
transit is but if you don't all it
really is it's an abstraction layer over
queing mechanisms or Pub Subs such as
rabbit mq or maybe sqs Aus service bus
this sort of thing so you code against a
common abstraction and then behind the
scenes M Transit can Implement all those
other transports specifically so the
only thing that changes is me saying M
transit. Aur service bus for example to
go from sqs to service bus and so on
we've seen all that in the previous
video so now that I've configured M
Transit here and I'm saying hey use this
sqsq over here and configure all the end
points all I need to do to introduce the
messaging functionality is go to my
service and say private read only I
publish endpoint so I'm going to use a
publish endpoint here which is scoped by
default in registration and I'm going to
take it and I'm going to say oh here is
where I want to publish a message and
how convenient I have a few message
contracts over here so this is the
record representing a customer being
created so I'm going to go here and I'm
going to say VAR message equals new
customer created and you can have a
mapper if you want all I'm going to do
is I'm going to manually map this user
over here so ID full name email and date
of birth and then what I'll say is await
the publish endpoint and then publish
that message and that is it and that
then we'll publish it into the queue and
just to see all that working I'm going
to stick a breakpoint here and just say
debug my application so I'm using AWS I
have the toolkit over here to see all my
messages in the sqs you can see the cues
are automatically created because must
Transit will do that and I'm going to go
ahead and say create a customer so send
that request I'm hitting my breakpoint
I'm going here I'm saying add the user
to the database and then I get my
message and over here I am publishing
that message so if I go to customer
created the queue and I view messages
you can see it over here it's already
published and you can see everything
about it over here and then after it's
published I also saved to the database
and thankfully it was saved successfully
and eventually I have as you can see
over here a consumer and if you want to
grab this code you can use a link down
below to get it for free but I can go to
this worker over here which has some
very simple configuration and it has
consumers for every type of message in
this case all it's really doing is it's
logging don't do logging this way by the
way this is not the right way to do
logging I'm just doing it like this
because I want to print something to the
console to show that I consumed it so
I'm going to run the worker and as
you're going to see very quickly we're
going to see that the customer created
message has been consumed if I go ahead
and I create a second customer you will
see immediately that another request has
been filed and it has been processed
through a new message fantastic very
easy messaging in the app now do you see
where the problem is with this you might
have spotted it the problem is that I'm
publishing this message which by the way
it could also be below the creation so
we sort of ensure that something was
created and only if it was created maybe
let's say result is more than uh zero
only then we publish the message which
is a nice safe guard to make sure we
don't publish a message for something
that wasn't done but what guarantees
that yeah okay this was added to the
database but do we know for a fact that
this was published to the queue we don't
we can't yes there are ways to get
around this with distributed
transactions and two-phase commits but
this is a very very hard problem in
general in applications because you
can't have Atomic operations between the
database you're using and the message
bus you're publishing towards and then
what do you do if this failed you roll
back and what if the roll back fail like
this such a big problem and chicken and
neck problem you have to solve here and
there's no obvious easy solution but but
the truth is there actually is it's both
very easy to implement and pretty
obvious you see we cannot have an atomic
operation between a database and a
message CU but we can have an atomic
operation in a database you can have a
transaction that does many things it
adds the customer and then it adds the
instruction to do something about the
added customer so what instead of
writing in a single table the customer's
table over here we also had a second
table called the outbox table and
there's actually more into that behind
the scenes but let's in theory just say
we have the outbox table saying that hey
a customer was created and then a
service picks that up after it's been
transactionally written and it has to
push it into a queue and then that queue
is read by a consumer and that is
published so if I was to use a very
scientific tool to demonstrate this I
can go here and I can say that instead
of writing into a single place over here
which in this case this is the customers
table let's also in the same transaction
because now we are on a database like I
said write something to the outbox then
have something and we will see what that
something is listen to that outbox and
see hey do you have any messages for me
to process and then that thing will not
only read those messages but also it is
going to publish them as we're going to
see here to AQ so take that message and
publish it now there is a bit of an
issue here where you can have the
situation where if this is trying to
read from the outbox and then get the
message and publish it what if the
publishing fails well you can update the
state of the outbox and say that this
message hasn't been processed yet and
only say that it has after you guarantee
that it is been pushed into the message
box the usual problem with all this is
that in some implementations you cannot
guarantee exactly one delivery with this
process so you need to have IDM potency
baked into your message processing
meaning that if a message is pushed
twice into the queue or consumed twice
your application only makes one action
not two having worked in finance dealing
with payments this was the bread and
butter for everything we did you don't
want to have a message or an event
handled twice because if you do someone
might pay twice
and people don't like paying twice if
they have only paid once now you could
definitely go and manually Implement all
this but you don't have to because mass
transit natively supports the outbox
pattern the transactional outbox pattern
on top of an inmemory one which I'm not
going to cover so how will we introduce
the pattern here well first what I'm
going to do is I'm going to move this
message publishing between the action
I'm taking on the database the addition
and before the save change is a sync and
I'm going to paste it here and I'm still
saying publish so the code I have here
will publish into the queue nothing
about the database as it is now I'm
going to go to my API new packages and
I'm going to say Mass transit. entity
framwork core and this is natively
supported with ENT framework you can
just add ENT framework just for this you
don't have to use it for your whole
application but to simplify management
of this whole process M Transit chose to
use EF core which by the way is fine
it's performance it's fast it is great
it's net 8 we're good and once I do that
I can go to my app DB context and I have
to specify in the model that there are
some tables that need to be created for
example I'm going to say override on
model creating and I'm going to add
three lines first I'll say model
builder. add inbox State entity then
model builder. add outbox State entity
and I'm also going to say modelbuilder
do add outbox message entity those three
things is what I need and once I specify
that I need a bit more configuration on
the admass transit method so what I'm
going to say in there is x. add Entity
framework outbox and I'm going to
specify my DB context so add DB context
some configuration over here and I'm
going to say o do use postgress and also
use bus outbox and I'm going to leave
the defaults for now I will also specify
here in this service o do query delay
and I'm going to set that to 1 second
I'm going to explain what that is in a
second I'm doing this for demo purposes
so we can have all the actions happen
quickly once I do that I want to create
migrations for this new feature because
there will be tables needed and these
tables are configured through this so
I'll say net EF migrations add outbox
and let's get that created here we go
migrations created so if I go here I can
see all the migrations and all the the
code needed for those new tables and
then I can say netf database update and
once I do that we're going to get all
those tables created if I refresh the
tables you're going to see inbox State
outbox message and then outbox state so
that will now handle our outbox pattern
and now that's it I'm not going to touch
anything else about my code meaning that
this publishing the dot publish that's
supposed to push into a que won't be
touched now let me quickly show you what
you need to do on the consumer side of
things first we need to add a couple of
packages so first we need the must
Transit entry framework core package we
added in the main API project and then
we'll also add the postgress um an
framework core then we're going to
create an app DB context it doesn't have
to be the same as the main application
this is just here for the outbox this
means that there's no other DB context
needed in here or DB set all you need is
a traditional model creation that is it
now back to the program.cs in the add
mass transit call we're going to say add
anti framework outbox appb context as in
the other application and here all we're
going to say is we're going to have a
duplicate detection window to detect
duplicate messages from the inbox so I'm
going to have that as time span Dot from
seconds 30 and then I'm going to say hey
use postgress but nothing about the bus
outbox just use postgress and that is it
now for all this to work I also need to
wire up of course the app DB context so
I'm going to say add DB context as I
would normally do app DB context and
then configure that to use postris and
that is it all I'm going to do now is
I'm going to run my worker which is
supposed to be listening to the cues and
as you can see it is listening to all
the cues and then I'm going to exactly
as it is I want to remind you I did not
touch the publishing message I'm going
to do is I'm going to stick a breakpoint
and run the API and let me show you what
happens in fact I'm going to remove the
breako for now and first I'm going to
publish a message so I'm going to go
straight over here into insomnia and I'm
going to say create a customer so
customer created a few things are
happening here we don't really care
about that but if I go to the worker as
you can see we did process the worker
because now we have the customer created
consumer message and all the details
that's it so nothing seems to have
changed because it still seem like I
just published into a que so what
problem did I actually solve well let me
show you I'm going to go to the customer
service and I'm going to slap a
breakpoint on the add a sync method
which I want to remind you in anyti
framework core doesn't take any action
it just marks The Entity to be added
when you call Save changes which again
is transactional so let's go ahead and
say create another customer and let's
hit that breakpoint we marking that
entity and then normally on this publish
call we would publish something into the
queue directly going to step over that I
haven't called save changes a sync yet
but I want to show you we have nothing
in the queue let me just save you
messages the queue is empty and you
might expect that because the worker is
running so you're assuming I publish
something and it's been consumed by the
worker the truth is it hasn't there is
no message consumption because nothing
has been published yet the moment
however I say save changes sync then and
only then something will happen and what
will happen is in the database if I go
to the outbox State you're going to see
the state of an outbox message and all
its details and then in the outbox
message you're going to see a new entry
over here which just was added with this
save async call alongside my new
customer which contains in the body the
whole message serialized so now the
moment I say run freely don't stick to
that break point and don't stay there to
my service my service behind the scenes
will use a bus running in the background
the hostage service running in the
background and push those messages and
as you can see it's not here anymore and
push that into the queue and my worker
then will pick it up and as you can see
it has been processed that's the whole
point of the outbox we are doing our
action transactionally and we're adding
an entry into the outbox and then later
service picks it up and it pushes it
into the message so both things happen
automically it is implemented in a great
way in mass transit is very simple to
work you don't need to change your code
this all still looks like two
independent calls but behind the scenes
this is all transactional now you could
stop that worker service that picks the
outbox and publishes it if you go over
here and just to remind you this query
delay is a delay that the service that
will read the messages in that outbox
and take them and push them will have to
perform so we don't just Spam the
database with many many requests per
second basically so in this whole thing
I could also say that disable that
delivery service that background service
logic because you just want to use the
adbox and you might want to have offline
processing so there's no que there's no
bus working you might also want to go
further and say disable the inbox
cleanup service you have full control
and you can even run this service that
looks into the database and pushes
messages into the queue to run out of
this process so you split that
responsibility to another service and
then you have the worker that still
consumes straight from the queue but it
acknowledges that the Out book exists so
it can also mutate the state as needed
and like I said if you wanted to use any
other provider with any of this you
could easily just go here and say mass
transit. rabbitmq for example remove
Amazon sqs why that up and the only call
that needs to change is the using
whatever you want to use r mq or any
other provider you want in this case sqs
is what I've used in production
extensively so that's the one I'm using
and that's it your code now just
supports the transactional outbox
pattern again you're going to find all
this code in the description down below
it's all for free and if you have any
questions please leave them in the
comments down below now something I want
to point out is that this is not my
favorite way to deal with this problem I
have a different approach having worked
in the cloud and in serverless for an
extensive period of time we actually
deal with this problem differently and
if you want to know how we deal with
that problem leave a comment down below
and I will make a video on that but now
when from you what do you think about
this and are you using something else to
implement it maybe you're using your own
approach leave a comment down below and
let me know well that's all I had for
you for video thank you very much for
watching and as always keep coding
تصفح المزيد من مقاطع الفيديو ذات الصلة
Dapr Community Call - May 29th 2024 (#104)
Kafka vs. RabbitMQ vs. Messaging Middleware vs. Pulsar
Testing Entity Framework Core Correctly in .NET
What is a MESSAGE QUEUE and Where is it used?
Introdução a bancos de dados SQLite com Python e módulo sqlite3
Microservices with Databases can be challenging...
5.0 / 5 (0 votes)