Transformers - Part 7 - Decoder (2): masked self-attention

Summary

This video script delves into the concept of masked self-attention within the decoder of a neural network, a crucial component for parallelizing calculations during training. It explains how the decoder maintains input shape and uses embedded vectors of the same length as the output sequence. The key focus is on the masked multi-head self-attention layer, which ensures the network doesn't 'cheat' by accessing future words in the sequence, thus learning to predict unseen words effectively. The script illustrates the construction of this layer, detailing the process from computing queries and keys to applying a mask that zeroes out weights for subsequent words, culminating in the computation of new embeddings that depend solely on preceding input vectors.

Takeaways

• 🧠 Masked Self-Attention is a key component of the decoder in parallelizing calculations during training.
• 🔄 The decoder consists of 'N' decoder blocks with the same structure but different parameters, similar to the encoder.
• 📏 The input and output matrices in the decoder maintain the same shape as the embedded output sequence.
• 🔢 The number of vectors in the decoder is generally different from the number processed by the encoder.
• 🚀 The goal is to compute all next word prediction probabilities in parallel, which speeds up training.
• 🚫 The decoder should not 'cheat' by having access to the target word when predicting the next word in the sequence.
• 🔑 Queries, keys, and values are computed for each input token, which are essential for the self-attention mechanism.
• 🎭 Masking is applied to unnormalized weights to ensure that later words do not influence the current word's embedding.
• 📊 The softmax operation is used to normalize weights, but with a mask to prevent future words from influencing the current word.
• 🔍 The masked self-attention layer ensures that the embedding for a word only depends on the preceding words.
• 🔄 Multi-head self-attention combines multiple instances of masked self-attention to create a comprehensive output.

Q & A

What is the main purpose of the masked self-attention mechanism in the decoder?

The main purpose of the masked self-attention mechanism in the decoder is to enable parallelization of calculations during training and to ensure that the prediction of each word in the output sequence does not have access to future words in the sequence.

How does the decoder maintain the shape of the input?

The decoder maintains the shape of the input by ensuring that the matrices passed between the layers have the same shape as the embedded version of the output sequence, with the number of vectors being the same as the number of elements in the output sequence.

What is the significance of the number of vectors in the decoder being generally different from the encoder?

The significance is that it allows the decoder to process the output sequence in a way that is tailored to the specific requirements of generating translations, which may differ from the input sequence processed by the encoder.

Why is it important to feed the entire output sequence into the decoder at once during training?

Feeding the entire output sequence into the decoder at once during training allows for the computation of all next word prediction probabilities in parallel, which speeds up the training process.

How does the decoder prevent the network from 'cheating' during the computation of next word probabilities?

The decoder prevents cheating by designing the network such that when computing the probabilities of a word in the sequence, it does not have access to that word or any subsequent words.

What role does the start of sequence token play in the decoder's computation of word probabilities?

The start of sequence token serves as the initial input for the decoder, and it is used along with the output from the encoder to compute the probabilities for the first word in the output sequence.

Can you explain how the masked multi-head self-attention layer is constructed using an example?

The masked multi-head self-attention layer is constructed by first computing queries, keys, and values for each input token. Then, it masks the unnormalized weights to ensure that words in the sequence do not influence the computation of previous words. After normalization, the new embeddings are computed as a weighted average of the value vectors, ensuring that each word embedding only depends on the preceding input vectors.

Why is it unnecessary to compute z43 and z53 when focusing on computing y3?

It is unnecessary to compute z43 and z53 when focusing on y3 because the masked self-attention mechanism ensures that the embedding for the third word (y3) should only depend on the first three input words (x1, x2, and x3), and not on any subsequent words.

How is the masked self-attention expressed in matrix form?

In matrix form, the masked self-attention first computes queries, keys, values, and z values using weight matrices. It then applies a mask to set the weights for later words to zero when computing the weights for a specific word. After normalization, the new embeddings are computed by taking the product of the value matrix and the weight matrix for the output.

What is the final step in the computation of the masked multi-head self-attention layer?

The final step is to concatenate the different y matrices computed by the different heads and then multiply this tall matrix with a weight matrix (w_o) to obtain the output y, which has the same dimension as the input x.

How does the order of input vectors affect the masked self-attention mechanism?

The order of input vectors is important in masked self-attention because it determines the dependencies between words in the sequence. The mechanism ensures that each word embedding only depends on the preceding input vectors, reflecting the sequential nature of language.

Outlines

00:00

🤖 Masked Self-Attention in Decoders

This paragraph introduces the concept of masked self-attention within the decoder, a crucial component that enables parallelization of calculations during training. The decoder is composed of 'N' identically structured blocks with unique parameters, maintaining the input shape while ensuring that the matrices passed between layers match the shape of the embedded output sequence. The goal is to understand the masked multi-head self-attention layer, which is the first of two self-attention layers in the decoder. The paragraph emphasizes the need for the decoder to compute all next word prediction probabilities in parallel during training, without 'cheating' by accessing future words in the sequence. This is illustrated with an example of how the network should only consider the output from the encoder and preceding words when predicting the next word, ensuring the network learns to predict unseen words for effective translation.

05:01

🔍 Constructing the Masked Multi-Head Self-Attention Layer

The second paragraph delves into the construction of the masked multi-head self-attention layer using an example. It explains the process of computing new embeddings for each word in the translation, ensuring that later words do not influence earlier ones. The paragraph details the computation of queries, keys, values, and z-values, and the application of a mask to prevent future words from affecting the current word's embedding. The normalization of weights to sum to one and the computation of the new embeddings as a weighted average of value vectors are also discussed. The paragraph concludes with the expression of the masked self-attention layer in matrix form, highlighting the use of weight matrices and softmax operations to ensure that each word's embedding depends only on the preceding input vectors, aligning with the desired property of the decoder.

Keywords

💡Decoder

The decoder is a component of a neural network model, specifically in the context of this video, it is part of a transformer model used for tasks such as translation. It is designed to read the output from an encoder and produce an output sequence, typically a translation of the input text. In the video, the decoder is composed of multiple blocks with the same structure but different parameters, maintaining the shape of the input and ensuring that the matrices passed between layers have the same shape as the embedded output sequence.

💡Masked Self-Attention

Masked self-attention is a mechanism that allows the decoder to focus only on the relevant parts of the input sequence when making predictions. This is crucial for parallelizing calculations during training, as it prevents the decoder from 'cheating' by looking ahead in the sequence. The script describes how this is achieved by masking future tokens' influence on the current token's computation, ensuring that each token's embedding only depends on the preceding tokens.

💡Parallelization

Parallelization refers to the process of performing multiple calculations simultaneously, which is a technique used in training neural networks to speed up the process. In the context of the video, the goal is to compute all next word prediction probabilities in parallel by feeding the entire output sequence into the decoder at once during training. This is an important optimization that makes training more efficient.

💡Multi-Head Self-Attention

Multi-head self-attention is an extension of the self-attention mechanism where the input is split into multiple 'heads' that each compute a different representation of the input. This allows the model to jointly attend to information at different positions from different representational spaces. The script explains that the decoder includes a masked multi-head self-attention layer as its first self-attention layer, which is crucial for the model's ability to handle sequences effectively.

💡Embedding

In the context of neural networks, an embedding is a numerical representation of a word or a phrase that captures its semantic meaning. The video script mentions that the decoder maintains the shape of the input and that the embedded vectors have the same length as the output embeddings. The embeddings are the numerical inputs that the decoder uses to generate the output sequence.

💡Softmax

Softmax is a function often used in neural networks to convert a vector of values into a probability distribution. In the script, it is mentioned that the softmax function is used to obtain weights from the z values, which are then used to compute the new embeddings in the self-attention mechanism. The masked softmax is a variant used to ensure that certain weights are zeroed out, preventing future tokens from influencing the current computation.

💡Query, Key, Value

In the self-attention mechanism, queries, keys, and values are derived from the input embeddings. Queries are used to probe the sequence represented by keys, and the resulting scores are used to weight the values. The script explains that these are computed for each input token and are essential for determining how much attention each word in the sequence should receive.

💡Z Values

Z values, as mentioned in the script, are the result of taking the inner product between key and query vectors. They represent the unnormalized attention scores between different tokens in the sequence. These scores are then used to compute the attention weights through a softmax operation, with masking applied to ensure future tokens do not influence the current token's computation.

💡Normalization

Normalization in this context refers to the process of adjusting the attention weights so that each column in the weight matrix sums to 1. This is important for ensuring that the final embeddings are a proper weighted sum of the value vectors, as explained in the script. Normalization is a crucial step in the self-attention mechanism to make sure the model's predictions are well-calibrated.

💡Weighted Average

A weighted average is a mathematical operation that combines different values according to their weights. In the script, it is mentioned that the new embeddings for each word are computed as a weighted average over the value vectors, where the weights are determined by the attention mechanism. This operation ensures that the embeddings reflect the importance of each word in the sequence relative to the current word being processed.

💡Start of Sequence Token

The start of sequence token is a special token used in sequence processing tasks to indicate the beginning of a sequence. In the script, it is mentioned that when computing the probabilities for the second word in the sequence, the network should only have access to the output from the encoder and the start of sequence token, which serves as the initial context for the decoder.

Highlights

The decoder in the video is studied for its use of masked self-attention to parallelize calculations during training.

The decoder contains 'N' decoder blocks with the same structure but different parameters, similar to the encoder.

The decoder maintains the shape of the input and the matrices passed between layers have the same shape as the embedded output sequence.

The number of vectors in the decoder is generally different from the number processed by the encoder.

The goal is to learn about the masked multi-head self-attention layer, the first of two self-attention layers in the decoder.

Computing all next word prediction probabilities in parallel during training is a desired property for speeding up training.

Feeding the entire output sequence into the decoder at once is only possible during training.

The network must not have access to the target word when computing probabilities to avoid 'cheating' and ensure learning to predict unseen words.

Masked multi-head self-attention is constructed to ensure dependencies are only on preceding words, not subsequent ones.

The query, key, and value vectors are computed for each input token in the masked self-attention layer.

Unnormalized weights are masked to prevent influence from future tokens before normalization.

The operation performed corresponds to a softmax with respect to certain z values, with unnecessary computations avoided.

The new embedding for a word is computed as a weighted average over different value vectors, with future words' weights set to zero.

The complete expression for a masked self-attention layer is presented in matrix form, emphasizing the dependency only on preceding input vectors.

A mask is applied to ensure that weights for later words are zero when computing the weights for a given word.

The multi-head attention layer concatenates outputs from different heads and multiplies with a matrix to obtain an output with the same dimension as the input.

Outputs from different self-attention heads are computed using masked self-attention to prevent embeddings from depending on later input words.

Transcripts

in this second video about the decoder

we study

masked self-attention which is what

enables us

to parallelize calculations during

training

here is an illustration of the decoder

which contains

capital n decoder blocks where all have

the same structure

but different parameters similarly to

the encoder

the decoder maintains the shape of the

input

however it's important to note that the

matrices that are passed

between the layers in the decoder all

have the same shape as the embedded

version of the

output sequence that is the number of

vectors

is always the same as the number of

elements in the output sequence

and the embedded vectors all have the

same length

as the output embeddings feeded into

the first self-attention layer we note

specifically

that the number of vectors is generally

different

to the number of vectors processed by

the encoder

the objective in this video is to learn

about the masked

multi-head self-attention layer which is

the first of the two

self-attention layers in the decoder

however

let us first reason about one of the

properties that we would like the

decoder to have

to speed up training it would be good if

we could compute

all the next word prediction

probabilities in parallel

we will then feed the entire output

sequence

into the decoder at once i'm here using

x

to denote the desired translation since

that's used as

input to the decoder we note that

this is only possible during training

since we obviously

don't have access to the target

translation when we want to use the

network

to translate the sentence so what we

describe here

only speeds things up during training

but this is of course also very

important

finally in order to work we need to

design the network

such that it cannot cheat that is when

computing the probabilities of the next

word in the sequence

it obviously should not have access to

that word

for instance when computing the

probabilities of x2

the network should only have access to

the output from the encoder

and x1 which in this case is the start

of sequence token

if we also tell the network that x2 is i

the task would be trivial and the

network would not learn to predict

unseen words which is the ability needed

in order to later produce translations

similarly when computing the

probabilities for

x4 the network should only have access

to the output from the encoder

as well as x1 x2 and x3

and we shouldn't tell the network that

the value of x4

is uh as you can see to keep the

notation simple

i've only written the probability of x4

here

which means that i've omitted the

variables that we condition on which

in this case should be the encoder

output and x1

x2 and x3 we have also omitted

the variables that we condition on in

the other expressions on this

slide let us now illustrate how the

masked

multihead self-attention layer is

constructed

using an example this layer receives

one input vector for each word in the

translation

as illustrated in the figure if we focus

on how we compute

the new embedding for y3 we realized

that it should only depend on

x1 x2 and x3 since the embedding for the

third word in our sequence

will eventually be used to predict x4

as a first step we proceed as usual and

compute queries

keys and values for each input token

we can also compute the z values by

taking the inner products between

keys and queries since we are focusing

on how to compute

y3 it's actually sufficient to compute

the query vector q3

and the z values z13 z23

up until z53 for instance to compute

z13 we take the inner product between

the key vector k1

and the query vector q3 and we then

divide by the square root of d

where d is the length of these query and

key vectors

we would normally feed these values into

a softmax

to obtain the weights but here we want

to ensure that x4

and x5 do not influence y3

and we therefore first mask the

unnormalized weights

by setting the weights for the fourth

and the fifth token

to zero we then simply normalize the

weights to obtain weights that sum to

one

however if you look closely it's easy to

see that the operation that we perform

here

actually corresponds to taking a soft

max with respect to z13

to z 3 3. we can therefore simply write

this as follows

where the first three elements are given

by the softmax

whereas the final two elements are zero

as you can see we never actually use z43

and z53 and it's therefore unnecessary

to compute them

finally we compute y3 by taking a

weighted average

over the different value vectors since

the weights for the fourth and the fifth

value vectors

are zero y3 does not depend on x4 and x5

which is what we wanted to achieve the

complete expression

for a masked self-attention layer is

easy to express

in matrix form we first compute queries

keys values and z values using the

weight matrices

w q w k w v

and the standard expressions expressed

in terms of the unnormalized weights

we then apply a mask that sets many of

the weights to zero

specifically this ensures that when

computing the weights

for word number i the weights for all

later words are zero these weights

then need to be normalized such that

each column in the matrix

sums to 1. the result is a matrix with

normalized weights

in each column we can also express this

in terms of

softmax operations to fit at least a few

terms on the same slide

i've here introduced a shorthand

notation sm

for softmax i'm also using a sub index

to refer to different elements in the

output

from the softmax operation for instance

the first column only has one non-zero

element

and for that to sum to one that element

has to be one

in the second column the first two

elements are non-zero

and given by taking a soft max of z12

and z22 in general column i has

i non-zero elements that we can compute

by taking

a soft max with respect to z1i

to zii finally we compute the new

embeddings by taking

capital v times capital w this implies

that y i is a weighted sum of the value

vectors

v 1 to v i we therefore conclude that

the i

word embedding only depends on the i

first input vectors to this layer

this is clearly the property that we

desired however

we also note that the order of the input

vectors is

important when we use masked

self-attention

and we no longer have a mapping from one

set to another

so far we have learned about masked

self-attention

and the decoder combines h of these into

the masked

multi-head self-attention layers the

overall structure of the multi-head

attention layer is the same as in the

encoder

and we first concatenate the different y

matrices

computed by the different heads before

multiplying this tall matrix with a

matrix

w o to obtain an output y

which has the same dimension as the

input x

the only difference is that the outputs

y i

from the different self-attention heads

are computed using

masked self-attention to ensure that the

embeddings

never depend on later input words