The Pattern You MUST Learn in .NET
Summary
TLDRDans cette vidéo, Nick explique le concept de 'outbox pattern' pour la messagerie asynchrone dans les applications .NET. Il montre comment implémenter ce modèle transactionnel avec MassTransit pour assurer la cohérence des données lors de la publication de messages vers des files d'attente comme AWS SQS. Il explique les avantages par rapport à d'autres approches et indique qu'il préfère une autre solution serveless qu'il détaillera dans une prochaine vidéo.
Takeaways
- 😀 Le sujet de la vidéo est le motif de la boîte de sortie (Outbox Pattern) dans les systèmes de messagerie, un sujet très demandé.
- 📝 Nick explique le fonctionnement, l'importance et les alternatives du motif transactionnel de la boîte de sortie dans les applications de messagerie.
- 🛠 La vidéo utilise Amazon SQS pour les démonstrations, mais les concepts sont applicables à d'autres services de messagerie comme RabbitMQ ou Azure Service Bus.
- 🔍 Nick présente un cas d'utilisation concret avec une API client et une base de données PostgreSQL gérée dans Docker.
- 🚀 Le motif de la boîte de sortie est présenté comme solution aux problèmes de cohérence des données dans les opérations de base de données et de messagerie.
- 📦 Mass Transit, une couche d'abstraction pour les systèmes de messagerie, est utilisée pour illustrer l'implémentation du motif.
- ⚙️ La mise en œuvre du motif de la boîte de sortie inclut l'utilisation de transactions pour s'assurer que les messages sont envoyés de manière fiable et cohérente.
- 📈 L'importance de l'idempotence dans le traitement des messages est soulignée pour éviter les actions en double dans les systèmes financiers.
- 🎓 La vidéo mentionne également un nouveau cours sur les monolithes modulaires en .NET, soulignant l'importance de la formation continue.
- 💡 Nick encourage les retours et les questions pour approfondir la discussion sur le motif de la boîte de sortie et son utilisation dans différentes architectures.
Q & A
Qu'est-ce que le modèle de boîte de sortie transactionnelle dans le contexte de la messagerie .NET?
-Le modèle de boîte de sortie transactionnelle est un modèle de conception utilisé dans les systèmes de messagerie .NET pour garantir que les opérations de base de données et les messages envoyés à la file d'attente sont traités de manière atomique. Cela signifie que si l'opération de base de données réussit, le message correspondant est également envoyé à la file d'attente de manière fiable.
Pourquoi est-il important de comprendre et d'utiliser le modèle de boîte de sortie transactionnelle?
-Comprendre et utiliser le modèle de boîte de sortie transactionnelle est crucial car il aide à maintenir la cohérence des données et l'intégrité des opérations dans des systèmes distribués. Cela garantit que les événements importants ne sont ni perdus ni dupliqués en cas de défaillance partielle du système.
Quel est le rôle d'Amazon SQS dans le contexte de ce modèle?
-Amazon SQS (Simple Queue Service) est utilisé comme mécanisme de file d'attente dans ce modèle pour gérer les messages de manière asynchrone. Il permet de stocker les messages qui doivent être traités, ce qui garantit leur livraison et leur traitement ultérieur, même en cas de défaillance temporaire du système.
Comment Mass Transit est-il utilisé avec le modèle de boîte de sortie transactionnelle?
-Mass Transit est utilisé comme une couche d'abstraction au-dessus des mécanismes de file d'attente comme Amazon SQS. Il facilite l'implémentation du modèle de boîte de sortie transactionnelle en fournissant un cadre pour publier et consommer des messages de manière fiable et cohérente entre les différents services et la base de données.
Quelles sont les alternatives au modèle de boîte de sortie transactionnelle?
-Les alternatives au modèle de boîte de sortie transactionnelle incluent l'utilisation de transactions distribuées, de mécanismes de confirmation manuelle, ou de différentes stratégies de gestion des échecs comme les tentatives de réessai ou le traitement manuel des erreurs. Chaque alternative a ses propres avantages et inconvénients selon le contexte spécifique de l'application.
Quel est le défi principal abordé par le modèle de boîte de sortie transactionnelle?
-Le défi principal abordé par le modèle de boîte de sortie transactionnelle est de garantir que les modifications de la base de données et l'envoi de messages à une file d'attente se produisent de manière atomique, évitant ainsi les problèmes de données manquantes ou en double en cas de défaillance du système.
Comment fonctionne la détection de messages en double dans ce modèle?
-La détection de messages en double dans ce modèle est gérée en utilisant une fenêtre de détection et en marquant les messages traités. Cela empêche le traitement répété du même message si celui-ci est envoyé plusieurs fois à la file d'attente, un aspect crucial pour maintenir l'idempotence des opérations.
Pourquoi est-il important d'avoir des opérations idempotentes dans les systèmes de messagerie?
-Les opérations idempotentes sont essentielles dans les systèmes de messagerie pour garantir que même si un message est traité plusieurs fois, il ne produit qu'un seul effet significatif. Cela est particulièrement important dans les systèmes financiers ou d'autres applications critiques où les actions en double peuvent avoir des conséquences graves.
En quoi consiste l'interaction entre la base de données et la file d'attente dans le modèle de boîte de sortie transactionnelle?
-Dans le modèle de boîte de sortie transactionnelle, les modifications de la base de données et les instructions pour envoyer des messages à la file d'attente sont enregistrées dans une même transaction. Ainsi, si la transaction échoue, ni les modifications de la base de données ni l'envoi de messages ne sont effectués, garantissant la cohérence des opérations.
Quelle est la principale préoccupation lors de l'implémentation de ce modèle en termes de performances?
-La principale préoccupation lors de l'implémentation du modèle de boîte de sortie transactionnelle est l'impact sur les performances, car l'ajout d'une couche de gestion de transactions peut augmenter la latence et la complexité du système. Il est crucial d'équilibrer les besoins de fiabilité et de performance selon les exigences spécifiques de l'application.
Outlines
😀 Introduction au modèle de boîte de sortie dans la messagerie .NET
Nick présente le modèle de boîte de sortie (outbox pattern), un concept crucial pour la messagerie dans les applications .NET, utilisant Amazon SQS comme exemple, mais applicable à d'autres services de messagerie grâce à MassTransit. Il explique l'importance de comprendre ce modèle pour gérer efficacement les transactions et les messages entre services. Le sponsoring d'AWS et un cours gratuit sur AWS pour les développeurs .NET sont également mentionnés, ainsi qu'une démonstration de l'utilisation d'une API client et d'une base de données PostgreSQL pour illustrer la création et la gestion des clients.
🚀 Implémentation et avantages de la boîte de sortie
Cette section détaille la mise en œuvre pratique du modèle de boîte de sortie dans une application .NET utilisant MassTransit avec Amazon SQS. Nick illustre comment publier des messages dans une file d'attente à partir d'un service de gestion des clients, en soulignant les défis liés à la garantie de la consistance des transactions entre la base de données et la file d'attente des messages. Il explique ensuite l'introduction d'une 'outbox' pour assurer des opérations atomiques dans la base de données, permettant une gestion fiable et cohérente des messages sans compromettre la transactionnalité.
🔧 Configuration technique et avantages de la boîte de sortie
Nick approfondit la configuration technique nécessaire pour implémenter le modèle de boîte de sortie, en utilisant Entity Framework Core avec MassTransit. Il guide à travers l'ajout de packages nécessaires, la configuration du DbContext, et la création de migrations pour les tables nécessaires dans la base de données. Cette approche permet d'assurer l'intégrité des transactions et la fiabilité de la messagerie en enregistrant d'abord les messages dans une table 'outbox' au sein de la même transaction que les opérations de base de données, avant de les publier dans la file d'attente, abordant ainsi les défis de cohérence des données.
📦 Démonstration pratique et résolution de problèmes avec la boîte de sortie
Nick démontre l'utilisation de la boîte de sortie dans une application en temps réel, soulignant comment les messages sont traités de manière transactionnelle et sécurisée. Il explique le fonctionnement interne, y compris la gestion des états des messages dans la base de données jusqu'à leur publication effective dans la file d'attente. La section aborde également les stratégies pour gérer les échecs de publication et assurer la livraison unique des messages, soulignant l'importance de la conception idempotente des consommateurs de messages.
🎓 Conclusion et meilleures pratiques pour l'utilisation de la boîte de sortie
En conclusion, Nick résume les avantages de l'utilisation du modèle de boîte de sortie pour gérer les transactions et la messagerie dans les applications .NET, tout en offrant un aperçu des alternatives basées sur son expérience avec le cloud et les architectures sans serveur. Il invite les spectateurs à partager leurs propres expériences et solutions alternatives, clôturant sur l'invitation à explorer plus de contenu éducatif et des cours sur le développement .NET.
Mindmap
Keywords
💡modèle
💡asynchrone
💡transactionnel
💡boîte de sortie
💡message
💡file d'attente
💡transport
💡consommateur
💡cohérence
💡transaction
Highlights
Introduces transactional outbox pattern for managing side effects when mutating state
Outbox pattern queues side effects after database transaction commits
Avoids race conditions between database commits and queued messages
Implemented using EF Core to manage outbox tables transactionally
Message publishing moved after database save to ensure atomicity
Background hosted service reads outbox and publishes messages
Consumer service processes published messages
Outbox entries tracked to handle publishing failures
Idempotent message processing required in case of duplicates
Can disable delivery/cleanup services for offline processing
Implementation abstracted from transport with MassTransit
Works with any message transport like RabbitMQ or Azure Service Bus
Alternative approaches for cloud and serverless architectures
Encourages feedback on other outbox implementation options
Source code available for free implementation reference
Transcripts
hello everybody I'm Nick and in this
video I'm going to introduce you to the
extremely highly requested topic of
outbox patterns in messaging Inn net
most specifically in this video I'm
going to show you the transactional
outbox pattern which is one of the most
important patterns you need to
understand and know how to use in
messaging in this video I'm going to
explain what it is how it works why we
need it and I'm also going to talk about
some Alternatives if you don't want to
use that pattern what other options do
you have in this video I'm going to be
using Amazon sqs which which is the
queuing mechanism in Amazon and this
video is sponsored by AWS so massive
thank you to them however this is not
only applicable to sqs because we're
going to be using mass transit to show
all this any transport or any backing
service we'll use whether that's rabid
mq or Azo service bus or anything else
will work exactly as you see it now AWS
has also sponsored a dome train course
so if you want to get started with AWS
Services as a cop developer click the
link in the description it's absolutely
free and yours to keep forever if if you
like of content and you want to see more
make sure you subscribe for more
training check out my courses on doet
tr.com okay so let me show what I have
here I have this customer API over here
it has one controller we can create get
get all update or delete a customer and
behind the scenes this is using a
database which I am running here in
Docker and this is a post Christ
database to store all the users I'm
creating so if I go over here and I say
run this API let me show you what I have
and here I can see my database and as
you can see I have a single user so if I
go to insomnia I can say get all
customers and I'm going to get this
customer back over here so I'm going to
get it I can get a customer by ID so I
can copy this ID and I can go here and
paste it at the end of the URL and I'm
going to get it if it is wrong I'm going
to get a 404 and I can also create a
customer as you can see over here so
let's create Nick again and this is all
created and you can see the created
users now I'm going to go quickly and
delete these customers now here's what
happens in most most applications when
you want to create a resource or update
or delete or basically mutate some state
which it's mutation might be important
for some other aspect of your business
for example when you create a customer
you also might want to quue an email for
email verification for example or
anything else any aspect of the business
and any team of the business might need
to know that the customer was created
updated deleted and so on so you can
subsequently for example delete more
information about them for gdpr purposes
or anonymize it well you might need to
do many things now I could just go to my
service over here and say okay here's
where I create my customer as I'm adding
them into the database and saving them I
can also say send email I can also say I
don't know call some API I can do many
many things but the problem I have here
is that this method now is doing too
much and I get very slow and the only
thing that's important for the customer
is that it was added to the database and
that is it they get the response they go
their merry way and hopefully my
application behind the scenes can do all
the other things now people have tried
naive ways for years to do this for
example they're spinning up a background
thread and they're putting this
functionality in the thread but what
happens if your application is scaled
out what happens if this is a micros
there's so many moving parts so a great
way to deal with this is to use a queing
mechanism and use asynchronous messaging
and either publish an event where people
can subscribe and listen to that or you
can just send it to a specific queue and
that queue can do something later for
example queue that email now let's see
how we would Implement something like
that now before I move on I'd like to
let you know we just launched a brand
new course on D train called getting
started with modular monoliths in net
and this authored by the legend Steve
adalis Smith I'm sure Steve has taught
many of you already with courses on
other platforms like prite but now he
authored his first of many courses on
dome train and it's all about how to get
started with modular monoliths not only
will he teach teach you the theory
behind the concept and how it compares
to microservices or traditional
monoliths but he will also build a whole
system in that course hands on with code
and diagrams and examples you can follow
along it is an amazing course and it is
the best way to get started with modular
monol hands down in.net now to celebrate
the launch I'd like to offer the first
500 of you a 20% discount so either use
a link in the description or apply code
modular at checkout to claim that 20%
off it's a great opportunity to get
started with a concept and I can't
stress enough how much of an amazing
author Steve is now back to the video
I'm going to go to the program.cs and as
you can see over here I've already added
and you get package called mass transit
now I've already covered mass transit in
net in a previous video so I'm going to
link that below if you want to see how
to get started with mass transit this
assumes you sort of understand what mass
transit is but if you don't all it
really is it's an abstraction layer over
queing mechanisms or Pub Subs such as
rabbit mq or maybe sqs Aus service bus
this sort of thing so you code against a
common abstraction and then behind the
scenes M Transit can Implement all those
other transports specifically so the
only thing that changes is me saying M
transit. Aur service bus for example to
go from sqs to service bus and so on
we've seen all that in the previous
video so now that I've configured M
Transit here and I'm saying hey use this
sqsq over here and configure all the end
points all I need to do to introduce the
messaging functionality is go to my
service and say private read only I
publish endpoint so I'm going to use a
publish endpoint here which is scoped by
default in registration and I'm going to
take it and I'm going to say oh here is
where I want to publish a message and
how convenient I have a few message
contracts over here so this is the
record representing a customer being
created so I'm going to go here and I'm
going to say VAR message equals new
customer created and you can have a
mapper if you want all I'm going to do
is I'm going to manually map this user
over here so ID full name email and date
of birth and then what I'll say is await
the publish endpoint and then publish
that message and that is it and that
then we'll publish it into the queue and
just to see all that working I'm going
to stick a breakpoint here and just say
debug my application so I'm using AWS I
have the toolkit over here to see all my
messages in the sqs you can see the cues
are automatically created because must
Transit will do that and I'm going to go
ahead and say create a customer so send
that request I'm hitting my breakpoint
I'm going here I'm saying add the user
to the database and then I get my
message and over here I am publishing
that message so if I go to customer
created the queue and I view messages
you can see it over here it's already
published and you can see everything
about it over here and then after it's
published I also saved to the database
and thankfully it was saved successfully
and eventually I have as you can see
over here a consumer and if you want to
grab this code you can use a link down
below to get it for free but I can go to
this worker over here which has some
very simple configuration and it has
consumers for every type of message in
this case all it's really doing is it's
logging don't do logging this way by the
way this is not the right way to do
logging I'm just doing it like this
because I want to print something to the
console to show that I consumed it so
I'm going to run the worker and as
you're going to see very quickly we're
going to see that the customer created
message has been consumed if I go ahead
and I create a second customer you will
see immediately that another request has
been filed and it has been processed
through a new message fantastic very
easy messaging in the app now do you see
where the problem is with this you might
have spotted it the problem is that I'm
publishing this message which by the way
it could also be below the creation so
we sort of ensure that something was
created and only if it was created maybe
let's say result is more than uh zero
only then we publish the message which
is a nice safe guard to make sure we
don't publish a message for something
that wasn't done but what guarantees
that yeah okay this was added to the
database but do we know for a fact that
this was published to the queue we don't
we can't yes there are ways to get
around this with distributed
transactions and two-phase commits but
this is a very very hard problem in
general in applications because you
can't have Atomic operations between the
database you're using and the message
bus you're publishing towards and then
what do you do if this failed you roll
back and what if the roll back fail like
this such a big problem and chicken and
neck problem you have to solve here and
there's no obvious easy solution but but
the truth is there actually is it's both
very easy to implement and pretty
obvious you see we cannot have an atomic
operation between a database and a
message CU but we can have an atomic
operation in a database you can have a
transaction that does many things it
adds the customer and then it adds the
instruction to do something about the
added customer so what instead of
writing in a single table the customer's
table over here we also had a second
table called the outbox table and
there's actually more into that behind
the scenes but let's in theory just say
we have the outbox table saying that hey
a customer was created and then a
service picks that up after it's been
transactionally written and it has to
push it into a queue and then that queue
is read by a consumer and that is
published so if I was to use a very
scientific tool to demonstrate this I
can go here and I can say that instead
of writing into a single place over here
which in this case this is the customers
table let's also in the same transaction
because now we are on a database like I
said write something to the outbox then
have something and we will see what that
something is listen to that outbox and
see hey do you have any messages for me
to process and then that thing will not
only read those messages but also it is
going to publish them as we're going to
see here to AQ so take that message and
publish it now there is a bit of an
issue here where you can have the
situation where if this is trying to
read from the outbox and then get the
message and publish it what if the
publishing fails well you can update the
state of the outbox and say that this
message hasn't been processed yet and
only say that it has after you guarantee
that it is been pushed into the message
box the usual problem with all this is
that in some implementations you cannot
guarantee exactly one delivery with this
process so you need to have IDM potency
baked into your message processing
meaning that if a message is pushed
twice into the queue or consumed twice
your application only makes one action
not two having worked in finance dealing
with payments this was the bread and
butter for everything we did you don't
want to have a message or an event
handled twice because if you do someone
might pay twice
and people don't like paying twice if
they have only paid once now you could
definitely go and manually Implement all
this but you don't have to because mass
transit natively supports the outbox
pattern the transactional outbox pattern
on top of an inmemory one which I'm not
going to cover so how will we introduce
the pattern here well first what I'm
going to do is I'm going to move this
message publishing between the action
I'm taking on the database the addition
and before the save change is a sync and
I'm going to paste it here and I'm still
saying publish so the code I have here
will publish into the queue nothing
about the database as it is now I'm
going to go to my API new packages and
I'm going to say Mass transit. entity
framwork core and this is natively
supported with ENT framework you can
just add ENT framework just for this you
don't have to use it for your whole
application but to simplify management
of this whole process M Transit chose to
use EF core which by the way is fine
it's performance it's fast it is great
it's net 8 we're good and once I do that
I can go to my app DB context and I have
to specify in the model that there are
some tables that need to be created for
example I'm going to say override on
model creating and I'm going to add
three lines first I'll say model
builder. add inbox State entity then
model builder. add outbox State entity
and I'm also going to say modelbuilder
do add outbox message entity those three
things is what I need and once I specify
that I need a bit more configuration on
the admass transit method so what I'm
going to say in there is x. add Entity
framework outbox and I'm going to
specify my DB context so add DB context
some configuration over here and I'm
going to say o do use postgress and also
use bus outbox and I'm going to leave
the defaults for now I will also specify
here in this service o do query delay
and I'm going to set that to 1 second
I'm going to explain what that is in a
second I'm doing this for demo purposes
so we can have all the actions happen
quickly once I do that I want to create
migrations for this new feature because
there will be tables needed and these
tables are configured through this so
I'll say net EF migrations add outbox
and let's get that created here we go
migrations created so if I go here I can
see all the migrations and all the the
code needed for those new tables and
then I can say netf database update and
once I do that we're going to get all
those tables created if I refresh the
tables you're going to see inbox State
outbox message and then outbox state so
that will now handle our outbox pattern
and now that's it I'm not going to touch
anything else about my code meaning that
this publishing the dot publish that's
supposed to push into a que won't be
touched now let me quickly show you what
you need to do on the consumer side of
things first we need to add a couple of
packages so first we need the must
Transit entry framework core package we
added in the main API project and then
we'll also add the postgress um an
framework core then we're going to
create an app DB context it doesn't have
to be the same as the main application
this is just here for the outbox this
means that there's no other DB context
needed in here or DB set all you need is
a traditional model creation that is it
now back to the program.cs in the add
mass transit call we're going to say add
anti framework outbox appb context as in
the other application and here all we're
going to say is we're going to have a
duplicate detection window to detect
duplicate messages from the inbox so I'm
going to have that as time span Dot from
seconds 30 and then I'm going to say hey
use postgress but nothing about the bus
outbox just use postgress and that is it
now for all this to work I also need to
wire up of course the app DB context so
I'm going to say add DB context as I
would normally do app DB context and
then configure that to use postris and
that is it all I'm going to do now is
I'm going to run my worker which is
supposed to be listening to the cues and
as you can see it is listening to all
the cues and then I'm going to exactly
as it is I want to remind you I did not
touch the publishing message I'm going
to do is I'm going to stick a breakpoint
and run the API and let me show you what
happens in fact I'm going to remove the
breako for now and first I'm going to
publish a message so I'm going to go
straight over here into insomnia and I'm
going to say create a customer so
customer created a few things are
happening here we don't really care
about that but if I go to the worker as
you can see we did process the worker
because now we have the customer created
consumer message and all the details
that's it so nothing seems to have
changed because it still seem like I
just published into a que so what
problem did I actually solve well let me
show you I'm going to go to the customer
service and I'm going to slap a
breakpoint on the add a sync method
which I want to remind you in anyti
framework core doesn't take any action
it just marks The Entity to be added
when you call Save changes which again
is transactional so let's go ahead and
say create another customer and let's
hit that breakpoint we marking that
entity and then normally on this publish
call we would publish something into the
queue directly going to step over that I
haven't called save changes a sync yet
but I want to show you we have nothing
in the queue let me just save you
messages the queue is empty and you
might expect that because the worker is
running so you're assuming I publish
something and it's been consumed by the
worker the truth is it hasn't there is
no message consumption because nothing
has been published yet the moment
however I say save changes sync then and
only then something will happen and what
will happen is in the database if I go
to the outbox State you're going to see
the state of an outbox message and all
its details and then in the outbox
message you're going to see a new entry
over here which just was added with this
save async call alongside my new
customer which contains in the body the
whole message serialized so now the
moment I say run freely don't stick to
that break point and don't stay there to
my service my service behind the scenes
will use a bus running in the background
the hostage service running in the
background and push those messages and
as you can see it's not here anymore and
push that into the queue and my worker
then will pick it up and as you can see
it has been processed that's the whole
point of the outbox we are doing our
action transactionally and we're adding
an entry into the outbox and then later
service picks it up and it pushes it
into the message so both things happen
automically it is implemented in a great
way in mass transit is very simple to
work you don't need to change your code
this all still looks like two
independent calls but behind the scenes
this is all transactional now you could
stop that worker service that picks the
outbox and publishes it if you go over
here and just to remind you this query
delay is a delay that the service that
will read the messages in that outbox
and take them and push them will have to
perform so we don't just Spam the
database with many many requests per
second basically so in this whole thing
I could also say that disable that
delivery service that background service
logic because you just want to use the
adbox and you might want to have offline
processing so there's no que there's no
bus working you might also want to go
further and say disable the inbox
cleanup service you have full control
and you can even run this service that
looks into the database and pushes
messages into the queue to run out of
this process so you split that
responsibility to another service and
then you have the worker that still
consumes straight from the queue but it
acknowledges that the Out book exists so
it can also mutate the state as needed
and like I said if you wanted to use any
other provider with any of this you
could easily just go here and say mass
transit. rabbitmq for example remove
Amazon sqs why that up and the only call
that needs to change is the using
whatever you want to use r mq or any
other provider you want in this case sqs
is what I've used in production
extensively so that's the one I'm using
and that's it your code now just
supports the transactional outbox
pattern again you're going to find all
this code in the description down below
it's all for free and if you have any
questions please leave them in the
comments down below now something I want
to point out is that this is not my
favorite way to deal with this problem I
have a different approach having worked
in the cloud and in serverless for an
extensive period of time we actually
deal with this problem differently and
if you want to know how we deal with
that problem leave a comment down below
and I will make a video on that but now
when from you what do you think about
this and are you using something else to
implement it maybe you're using your own
approach leave a comment down below and
let me know well that's all I had for
you for video thank you very much for
watching and as always keep coding
Voir Plus de Vidéos Connexes
Arbitrage sur Amazon : Comment commander chez un grossiste ?
Simuler la connexion entre ordinateurs avec Filius
Éviter les silences gênants - Guide Gratuit
Notion tutoriel : Organiser ses vidéos Youtube et retenir l'essentiel grâce à Notion ▶️🏷️
Build a Simple Notion Second Brain from Scratch (in ONLY 10 Minutes)
Your Credit Card is at Risk because of this hacking device!
5.0 / 5 (0 votes)