What is a Message Queue?

Summary

In this informative video, Jamil Spain from IBM Cloud explores the concept of message queuing, an architectural technique that facilitates asynchronous communication between applications. He explains the importance of decoupling applications for scalability and flexibility, using email as a relatable example of how messages can be queued and processed independently. Jamil also discusses different messaging patterns like point-to-point, publish-subscribe, and request-reply, emphasizing their roles in modern cloud-native architectures. The video is a valuable resource for developers looking to enhance their application design with message queuing.

Takeaways

• 😀 Message queuing is an architectural technique that facilitates asynchronous communication between applications.
• 🔄 It allows for the decoupling of applications, enabling them to function independently without immediate processing dependencies.
• 📧 The concept of a message queue can be likened to an email inbox, where messages are held until they are ready to be processed.
• 💌 Messages can vary in type, including data payloads, files, or metadata that triggers actions in other systems.
• 👤 Key roles in message queuing are the producer, which sends messages, and the consumer, which processes them.
• 📦 The point-to-point messaging pattern involves a direct message exchange between a single producer and a single consumer.
• 📢 Publish-subscribe (PubSub) is a pattern where a single producer sends messages to multiple consumers interested in the message topic.
• 🔁 Request-reply is a messaging pattern where a producer sends a request and waits for a response on a separate queue.
• 🔄 Decoupling through message queuing leads to more manageable, scalable, and maintainable applications.
• 🌐 Message queuing is particularly beneficial in cloud-native architectures where services are distributed and require loose coupling.

Q & A

What is the primary expectation in traditional application development regarding processing time?

In traditional application development, there is an expectation that applications would process everything immediately, with instant confirmation that something is happening.

How does message queuing differ from traditional synchronous communication?

Message queuing allows for asynchronous communication, meaning that applications can continue to function without waiting for an immediate response from other parts or systems, unlike synchronous communication which requires an immediate response.

What does the term 'decoupling' mean in the context of message queuing?

Decoupling refers to the ability to break applications apart into separate components that can operate independently, allowing for greater flexibility and scalability in application architecture.

Can you provide an example of asynchronous communication mentioned in the script?

An example of asynchronous communication given in the script is email, where a message can be sent and the sender can continue with other tasks without waiting for a response.

What are the two main roles or profiles in a message queuing system?

The two main roles in a message queuing system are the producer, which sends messages to the queue, and the consumer, which retrieves and processes the messages.

What is the difference between point-to-point and publish-subscribe messaging patterns?

In a point-to-point messaging pattern, messages are sent from a single producer to a single consumer. In contrast, the publish-subscribe (PubSub) pattern allows multiple consumers to subscribe to messages from a single producer.

How does message queuing support scalability in applications?

Message queuing supports scalability by allowing different parts of an application to be scaled independently based on demand, without affecting the overall system, as each component can be developed, deployed, and scaled separately.

What is the significance of the 'queue' in message queuing?

The 'queue' in message queuing is a data structure that holds messages in the order they are received and delivers them sequentially to the consumer, ensuring that messages are processed in a controlled and organized manner.

How can message queuing help in handling high traffic on web forms?

Message queuing can help in handling high traffic on web forms by allowing the form data to be sent to a queue and processed asynchronously, which prevents the system from being overwhelmed and ensures that the user interface remains responsive.

What are the benefits of using message queuing in a microservices architecture?

Using message queuing in a microservices architecture allows for better decoupling of services, improved scalability, and the ability to handle failures more gracefully, as each service can operate independently and communicate through messages.

How does message queuing enable the use of different programming languages for different components of an application?

Message queuing enables the use of different programming languages for different components by allowing each service or application part to operate independently, focusing on its single responsibility without being tied to a specific technology stack.

Outlines

00:00

📚 Introduction to Message Queuing

Jamil Spain from IBM Cloud introduces the concept of message queuing, an architectural technique that facilitates asynchronous communication between applications. He discusses the traditional expectation of immediate processing in applications and how message queuing allows for a more flexible, decoupled approach. Jamil explains the term 'message queuing' by breaking it down into 'message', which refers to data that needs to be transmitted, and 'queue', which is a sequential list that holds messages in order for processing. He uses email as an analogy to explain the asynchronous nature of message queuing, where messages are sent and received independently of each other's processing time.

05:04

🔄 Message Queuing Patterns

The script continues with Jamil explaining different message queuing patterns, starting with the point-to-point model where a producer sends a message directly to a consumer. He uses a visual aid to depict a queue and illustrates the roles of producers and consumers in this process. Jamil then moves on to the publish-subscribe (PubSub) model, where multiple consumers can listen to a channel for messages. He gives examples such as sending notifications to various departments upon a new order in an e-commerce setting. Lastly, he touches on the request-reply pattern, where a producer sends a message and waits for a response on a separate queue, emphasizing the asynchronous nature of these interactions.

10:05

🔧 Benefits of Message Queuing

Jamil outlines the benefits of using message queuing, focusing on decoupling as the primary advantage. He explains that decoupling allows for separate components of an application to operate independently, which simplifies development and maintenance. It also enables scalability by allowing specific parts of an application to be scaled based on demand without affecting the entire system. Jamil suggests that decoupling can lead to more efficient resource allocation and the ability to use different technologies or languages for different parts of an application, based on their specific needs.

15:06

🌟 Closing Thoughts on Message Queuing

In the final paragraph, Jamil wraps up the discussion by emphasizing the importance of message queuing in modern cloud-native architectures. He encourages developers, both new and experienced, to consider message queuing as a technique to enhance their application's architecture. Jamil invites viewers to share their experiences and questions about implementing message queuing and ends the video with a call to action for likes and subscriptions for more content like this.

Keywords

💡Message Queuing

Message Queuing is an architectural technique that facilitates asynchronous communication between different parts of an application or different applications altogether. It allows for the decoupling of processes, enabling them to operate independently and communicate through messages that are queued for processing. In the video, Jamil Spain explains that message queuing is essential for modern applications that need to handle operations that don't require immediate processing, thereby enhancing the application's ability to scale and manage different workloads efficiently.

💡Asynchronous Communication

Asynchronous communication refers to a communication model where the sender of a message does not need an immediate response from the receiver. This allows for operations to be processed independently of each other, without waiting for a response. In the context of the video, Jamil uses email as an analogy to explain asynchronous communication, where a message is sent and the sender can proceed with other tasks without waiting for the receiver to process or respond to the message.

💡Queue

A queue, in the simplest form, is a line of items that are processed sequentially. In the realm of message queuing, a queue holds messages in the order they are received and delivers them to the intended recipient. Jamil Spain describes a queue as a mechanism that ensures messages are processed in a first-in, first-out (FIFO) manner, which is crucial for maintaining the order and integrity of message processing.

💡Producer

In the context of message queuing, a producer is an entity that generates and sends messages to a queue. The producer is responsible for creating the message and placing it into the queue for later processing. Jamil Spain mentions that in a web form scenario, the form data can be transformed into a message by the producer, which is then placed on a queue for eventual processing, such as data insertion into a database.

💡Consumer

A consumer, in message queuing, is an entity that receives and processes messages from a queue. The consumer is at the other end of the communication channel, waiting to receive and act upon the messages that have been queued. Jamil Spain illustrates this concept by describing how a consumer might process messages from a queue to perform tasks such as updating a database or triggering marketing automation workflows.

💡Decoupling

Decoupling in software architecture refers to the practice of separating the functionality of an application into distinct components that operate independently. This allows for easier maintenance, scalability, and the ability to update or replace parts of the system without affecting others. In the video, Jamil Spain emphasizes the importance of decoupling in microservices architecture, where message queuing plays a key role in allowing different services to communicate without being tightly bound to each other.

💡Scalability

Scalability is the ability of a system to handle increased load or to be expanded to accommodate that growth. In the video, Jamil Spain discusses how message queuing contributes to scalability by allowing different parts of an application to scale independently based on demand. This is particularly useful in cloud-native architectures where different services may need to scale up or down to meet varying workloads.

💡Point-to-Point

Point-to-point messaging is a communication pattern where a message is sent from one producer to one consumer. This is a direct communication model where the message is intended for a specific recipient. Jamil Spain uses the example of a web form submission being turned into a message by the producer and then processed by a specific consumer, such as a service that inserts the data into a database.

💡Publish-Subscribe (PubSub)

Publish-Subscribe is a messaging pattern where a message is sent by a producer to multiple consumers who have expressed interest in the topic of the message. This pattern is useful for broadcasting messages to various parts of an application that need to react to the same event. In the video, Jamil Spain describes how a new customer sign-up or an e-commerce order can trigger multiple actions across different services, such as accounting, shipping, and notifications.

💡Request-Reply

Request-Reply is a messaging pattern where a message is sent from a producer requesting some action or information, and the consumer processes the request and sends a response back to the producer. This pattern is akin to a two-way communication where the producer awaits a reply after making a request. Jamil Spain explains this concept by describing how a producer might send a request for information, the consumer processes it, and then the producer receives a response with the processed data.

Highlights

Introduction to message queuing as an architectural technique for asynchronous communication.

Historical context where applications were expected to process everything immediately.

Expectation of instant response in REST API calls and the challenges it presents.

The concept of breaking applications apart using message queuing for better architecture.

Definition and explanation of asynchronous communication in application development.

The role of message queuing in allowing applications to function independently.

Explanation of the term 'message' in message queuing, referring to data transmission.

Definition of 'queue' and its function in holding messages in order for processing.

Analogy of email as an example of asynchronous communication and message queuing.

Different types of messages and their formats in message queuing systems.

The producer and consumer roles in message queuing.

Point-to-point messaging pattern for direct communication between applications.

Publish-subscribe (PubSub) model for broadcasting messages to multiple consumers.

Request-reply pattern for handling messages that require a response.

Benefits of message queuing including decoupling and scalability.

How decoupling enables teams to work independently and enhances application scalability.

The flexibility of language choice and technology stack in a decoupled application architecture.

Message queuing as a gateway to cloud-native architecture and its relevance for new and existing developers.

Encouragement for developers to consider message queuing in their architectural designs.

Transcripts

Hello, my name is Jamil Spain,

Developer Advocate with IBM Cloud,

and my topic for today is

Message Queuing.

Now, as a technologist, when I first

started out developing applications,

there was an expectation that in my

code that applications would

process everything immediately.

And even when I first started

dealing with REST APIs, there's

really the expectation that I make a

front end, makes a rest call to a

backend through to a REST API call

that you're expecting a response to

come back. Some type of response

code must go back to let me know

404 was not good, 200

it was good.

But again, there's that expectation

that things happen instantly, and I

get instant confirmation that

something is happening.

And often I wrote a lot

of code to

accommodate situations where

that communication could not happen

or did not

function properly there.

But today's topic kind of branches

out through that.

Let's deal with break down what the

term message queuing actually means

here. So overall, from a

thirty thousand foot view,

it's a definitely a architecture

technique that allows you to

break applications apart.

Apps apart.

With what we like to call

asynchronous communication.

I'll abbreviate that there.

So asynchronous communication.

So I want my particular

applications to or particular

subsets to still function

and not worry about the other parts

there versus the

other part where all my code

was in one stack, and

in only in a monolithic application

or when I built an application where

all the functionality exists

that, you know, one

set of functions and one set of code

on my application.

I'm just always handling a

transaction off to another set of

code. But there is an expectation

that things follow through and

process right then.

There's no way to really kind of

delay things to happen later.

And this is what message queuing is

all about. Well, let's break down

the first part here.

First, let's deal with the

word message.

And as it did denotes that

it is really something

that a piece of

information that wants to get

processed from one system to

another, an application

will just say subsystem for me as

well, kind of denoted by

with that particular piece.

So you have data

that needs to get

transmitted. Now this could be

actual data payloads, actual

files, it could

be metadata that that reference

is that something should happen on

the other in.

So many different types

of data

formats can be there.

All right. And then we have the word

queue.

Which queue in the most simplest

form refers to kind

of a line of things that a process

sequentially.

So when you think of a message

queue, you think of the messages

of coming into a queue

as a way that kind of holds the

messages in the order they are

received and delivers them

to the actual party.

So one way to think about this,

I know this kind of explained

message, explained queuing,

explained the overall, but think

about a tool that everyone uses

every day, email.

Email is an excellent example

of asynchronous communication

and how that works.

I can, theoretically or

not, theoretically, actually in

reality, in real life,

I send a message out, but

I can wait for a response later.

It goes into an inbox.

Think of that inbox, your email

inbox as a queue of messages

that are held there.

And at some point the

person is going to receive it

has the opportunity to look at it

whenever they want, if they take a

longer time, there's a larger list

that has accumulated of messages

that you have to read, which is

usually the case when I go through

some of my email email boxes,

or for ones that I check constantly,

like, say, for instance, your work

email, you always constantly

going through the list that you're

there waiting to kind of reply

back. But excellent example

of asynchronous.

I can really fill out an email,

compose it, send it off.

I'm not worried about whether you

received it or when you answer it,

but I can move on to something else.

All right, so perfect example of

asynchronous, and we'll carry that

example through as

well to some of the use cases

here.

So we talk through those,

let's talk about the type of

messages that occur.

So we've mentioned queue, which is a

line

of messages.

So if I were to draw this out

in this perspective here, let's do

it this way, let's

take this green as the queue.

And they also say that each one of

these is going to be an actual

message.

And so at some point you're going to

have the...

They're going to send the kind of

the message here.

And then at the other end,

somebody is going to open it up.

Get a little envelope there.

All right, a little artwork here.

So the person that's putting the

message on is going to be what's

known as the producer.

And then the person consuming the

message is the consumer.

So these are often the two profiles

or roles that you can take.

And of course, you can be the

producer one time and be the

consumer on another example.

So this one pattern here that you're

looking at is more like a

point to point.

I am setting it for one application

to the other.

All right.

It could be a

let's say I have a web form

and you're filling out, and as you

complete the web form, I take the

metadata from that form, JSON

object, I could put that on a queue

to be processed later to be inserted

into a database, to be

inserted to another system for

marketing automation, whatever

it may be the use case.

The point is that I have, I need

messages to guarantee delivery to

get there and not

be dependent upon you, submit

the form, I send it back to that

other system and get a response

back.

I can keep ingesting particular

messages, forms and middles and

keep loading those in to go from

there. So when you have these kind

of use cases, these are excellent

things. They should read out that

hey, instead of me just doing direct

http connection, let me think of a

message queuing pattern that

I could facilitate to go here.

So we'll call this point to point.

The next one is something

like a publish, publish

subscribe.

Also abbreviated as PubSub

that you will hear and this works

from the capability that I have

a producer.

Let's go back to the green, I'll do

another queue, and we'll do

more messages that come in.

So on the PubSub model,

I'm going to have

multiple

parties or applications that

are interested in getting

this message

as it comes through.

All right, so pub, publish

subscribe, so these consumers

are listening for this particular,

I'll call it a channel or interested

in this particular thread of

messages that may come across.

So it could be that new customer

signs up, you may want to send

something to

for a new order, eCommerce order

comes in. You may want to send

something to accounting for an

invoice to go out.

You may want to send something to

the shipping or notifications

for stock or, you know,

any multiple things

that you want to kind of get

through. That could be something you

can shoot out to kind of go.

And the last one I'll work through

is what we like to call

request-reply reply-requests.

It's done from the standard that you

have a producer

that's going to send out

a message.

All right.

Let me redraw this again really

fast here.

I got messages that I'm going to put

in.

OK.

And so this could be

one way, they have could have one

channel.

All right. So there could be one

way where they want to go out,

someone who consume it.

I could put a request out,

someone gets it, does some

processing.

But the response to that that they

want to do, I'm going to also

be listening on

another queue

for a response to that.

So that could be now that all of

a sudden

where they are reading this request

that comes through that they could

put the resulting workload

or whatever computation that they're

doing, whatever algorithm that must

be done.

Say, for instance, maybe you're

some of those patterns where you

want to put your information in and

look at all the offers that are out

there. It could be a actual search

to go out and get all this

information. Compile that back

to maybe a list of object,

that information comes back and

then I'm listening, the producer is

also listening to another queue

to say, "Oh, the response

is ready." And then

it takes that information and then

presents it. So you have

point-to-point.

All right, this is a Pub.

PubSub and then more of a

request and reply.

All right. So a different type of

messaging pattern that can occur.

And again, I love to kind of think

through things from a backwards

perspective.

It's really about as a technology.

Isn't it a solution architect?

Sometimes you want to put your hat

on where you say,

"What is my intent?

What do I need to happen?" And that

really drives what particular piece

of technology can drive

that particular component here.

All right. So three patterns we

identify message queuing.

Let's wrap it up with the benefits

naturally of why you want to

facilitate or go through the effort

to orchestrate something like this.

And I like to bring this down to

two major

particular pieces here.

The first,

decoupling.

All right, it looks like an N, but

it's really an I in an N, all right.

Trust me there.

All right. So decoupling here.

So what this allows me to do,

we mentioned the first part about

breaking applications apart.

It allows me now to have

separate items that can

occur. So I can,

in going along what we like to call

the microservice pattern,

responsibility pattern, I can

dedicate one particular piece

of code application

to just be responsible

for that one piece of work.

All right.

And if I was in a monolithically

kind of code base, I

would have to progress things all at

one time. If I wanted to just work

on my payments module, I

would also have to test everything

to make sure how that affects other

parts here.

Well, and this particular piece,

let's see, I just wanted the web

front end to be the web front end.

I don't want you to handle a lot of

the processing and doing all

the database connections from there.

I'll let other systems do that part,

or they can be very well at just

connecting to my database,

connecting to my services and my

systems of record, anything in that.

So, decoupling allows you to really

separate things up, where

apps

can just be

apps, you know, just

perform their particular job

here as well.

And naturally,

that will

enforce scalability in your

application, because now

you have apps that are decoupled,

you have teams that can work on

applications, I can think about my

application much simpler.

I may have one component that I want

to work on at a time while the other

is guaranteed to function

as is. And also it

allows me to add more resources to

certain parts.

As we mentioned in the other use

case earlier, where I have a web

form that wants to maybe

handle a lot of high traffic of

submitting on my front end may have

a lot, but I may want to wait

before I process a lot

of that on the backend and not have

that immediate.

I may want to put it through more

checks, maybe more detection,

maybe more actual code

to detect for any of these things

or the use cases are really numerous

here. I'm trying to simplify down

to some something simple that we can

digest here, but

this allows me to have some things

that can scale out to meet demand

without having to

scale the whole application up.

So I immediately get that piece

here.

And so from a perspective of

processing of data,

this is great for me.

And I can also,

evolve other sections of my

code to work better, scale faster.

I want to swap out different

components.

All right.

And even in my decoupling.

Whereas before monolithically,

I probably am writing everything in

one language because it's all one

single code base.

Decoupling gives me the ability to

maybe lean on other languages for

a certain expertise.

I may have my front end in a Java,

but my back end may be in a

Node.js or any other language,

Python, I can take advantage of

many different other benefits of the

language you can offer.

Because again, it's all about the

single responsibility of work that

they must achieve here.

So, to summarize

it all, message queuing, it's

definitely for the new developer.

It is something to consider

from an architectural perspective of

what you want to achieve.

For our existing developers, I

started out as a LAMP and a MEAN

developer myself, started

out for monolithically moved in the

HTTP communication between services.

And this is definitely a technique

that I have embraced in my

hybrid architecture to say there are

some circumstances when these

patterns resonate with me.

I'm like, "yes, that would fit

perfectly to work." Definitely,

if you're getting in the cloud

native being that we have

things that are decoupled

or at the part that you need

many different pieces to function

together to connect them together,

message queuing is a great

segue way to get you immediately

there in the cloud native world.

So definitely a great technique

to try here.

Hope this video was useful.

I can't wait to hear how you are

going to implement message queuing

in your architecture.

Thank you for your time.

If you have questions, please drop

us a line below.

And if you want to see more videos

like this in the future, please

like and subscribe.