Illustrated Guide to Transformers Neural Network: A step by step explanation

Summary

该视频详细介绍了transformer模型的工作原理。它利用注意力机制,允许模型关注输入序列中每个词与其他词之间的关系。视频通过一个聊天机器人的示例,步步解读了transformer模型的encoder和decoder部分。它展示了注意力机制如何帮助模型生成更好的预测。总的来说,该视频帮助观众深入理解这种基于注意力的模型的威力。

Takeaways

• 😀 转换器通过注意力机制实现了在NLP任务上的突破
• 😊 转换器可以参考很长的上下文信息,不像RNN有短期记忆问题
• 🤔 多头注意力允许模型学习输入中不同单词之间的关联
• 😯 位置编码给输入注入了顺序信息,因为转换器没有循环结构
• 😀 将查询,键和值的注意力机制应用于自注意力中
• 🤔 编码器通过自注意力和前馈网络编码输入序列
• 😊 解码器通过遮挡未来记号只关注过去
• 🧐 前馈网络和残差连接帮助网络更好地训练
• 😀 堆叠编码器和解码器层可提升模型预测能力
• 🥳 转换器架构让NLP取得了前所未有的成果

Q & A

变压器的核心机制是什么?

变压器最核心的机制是“attention机制”,这使得模型可以学习关注输入序列的,不同部分,从而做出更好的预测。

多头注意力机制是什么?

多头注意力机制产生多个attention层或“头”,每个头会独立地学习不同的attention表示,然后把它们合并以产生更丰富的表示。

为什么要使用位置编码?

因为变压器编码器没有RNN所具有的循环结构,所以需要通过位置编码向嵌入向量注入位置信息,让模型能够区分不同位置的词。

编码器的作用是什么?

编码器的作用是将输入序列映射为一个连续的、带有注意力信息的表示,这可以帮助解码器在解码过程中正确地关注输入的相关部分。

解码器的关键特点是什么?

解码器是自回归的,它接收先前输出作为当前输入,同时也接收来自编码器的输出。解码器通过Masked Attention 防止模型看到未来词。

变压器为什么常超越RNN?

因为注意力机制的关系,变压器在理论上具有无限的工作记忆,可以更好地处理长序列。而RNN则受限于短期记忆问题。

变压器主要应用于哪些领域?

变压器被广泛应用于机器翻译、开放域问答、语音识别等多种自然语言处理任务中,取得了显著成效。

BPE是什么?

BPE(Byte Pair Encoding)是一种词元化算法,通过迭代地将频繁共现的字符对合并,可以有效地产生词元,提高模型对词汇的覆盖率。

变压器为何需要大量训练数据?

变压器的参数较多,需要大量标注数据进行有效训练,以防止模型过拟合。数据量不足时,变压器的性能会明显下降。

变压器的主要缺点是什么?

变压器计算复杂,需要大量计算资源,同时也需要大量标注数据进行有效训练。这限制了其应用范围。

Outlines

00:00

🤖 变压器模型的介绍与原理

这一部分主要介绍了变压器模型在自然语言处理（NLP）领域的重要性和应用范围，包括机器语言翻译、会话式聊天机器人和搜索引擎等。变压器模型因其优于循环神经网络（RNN）、长短时记忆网络（LSTM）和门控循环单元（GRU）在序列问题上的表现而受到关注。文章还提及了几种著名的变压器模型，如BERT、GPT-2，并重点介绍了开创性的论文“Attention is All You Need”。通过解释注意力机制的工作原理和优势，展示了变压器模型如何能够在处理长序列时保持对早期输入的参考能力，从而克服了RNN等模型的短期记忆限制。

05:03

🔍 变压器编码器和多头注意力机制

第二段深入探讨了变压器模型的编码器和多头注意力机制的具体工作原理。编码器通过自注意力（Self-Attention）机制，使模型能够识别输入序列中各个词之间的相关性。多头注意力机制进一步增强了这一过程，通过将查询（Query）、键（Key）和值（Value）向量化，并执行一系列操作（如点积、缩放、Softmax函数应用等），实现对不同词之间关系的编码。这种机制使模型能够从多个角度学习词之间的关系，提高了处理和理解语言的能力。通过引入残差连接和层归一化，还提到了如何通过这些技术提高模型的训练效率和稳定性。

10:04

📝 变压器解码器及文本生成

本段介绍了变压器模型的解码器部分及其在文本生成中的应用。解码器结构与编码器相似，但增加了一个额外的多头注意力层，用于处理编码器输出的信息，以及避免在生成未来词汇时提前看到信息的掩码技术。这允许模型在生成文本时考虑之前的上下文，并根据输入序列中的关键信息生成相关响应。通过层叠解码器层，模型能够生成复杂且连贯的文本输出，展现了变压器在自然语言生成任务中的强大能力。此外，还解释了如何通过线性层和Softmax函数将解码器输出转换为最终的词汇概率分布，完成文本生成过程。

Keywords

💡transformer

transformers 是一种新的神经网络架构,基于注意力机制,用来处理序列数据。视频中详细解释了 transformer 的工作原理,它通过编码器和解码器的结构,使用多头注意力机制来学习输入序列中的上下文和语义信息,从而生成输出。transformer 相比循环神经网络有更强的长序列建模能力。

💡attention mechanism

注意力机制是 transformer 模型的核心。它允许模型学习输入序列中每个词与其他词之间的相关性或上下文依赖性。视频详细介绍了自注意力机制的计算过程,以及 masking 方法防止解码器参考后续词汇的技巧。注意力机制使得 transformer 在长序列任务上性能较好。

💡encoder

编码器在 transformer 架构中负责分析输入序列,通过多头自注意力层和前馈全连接层提取输入的特征表示,得到一个连续的向量表示,聚合了整个输入序列的上下文信息。

💡decoder

解码器基于编码器输出,在 transformer 中采用自动回归的方式生成输出序列。关键在于使用遮掩多头自注意力防止解码历程中参考后续词汇。解码器能够关注编码器输出的相关部分。

💡multi-headed attention

多头注意力机制为注意力机制带来了多个并行的子空间投影,每个头负责学习输入序列不同的表示子空间。多头结构增强了模型学习语义信息的多样性。编码器和解码器中都使用了多头自注意力结构。

💡positional encoding

由于 transformer 没有循环或卷积结构,需要通过位置编码的方法为模型输入的词汇嵌入添加位置信息,以表示其在序列中的位置。视频介绍了使用三角函数实现位置编码的技巧。

💡residual connection

残差连接让网络中间层的输出能够直接传到后面的层,保证信息的流动,有利于梯度反向传播和模型的训练。在 transformer 的编码器和解码器中,每个子层后面都有残差连接和层归一化。

💡sequence to sequence learning

序列到序列学习是指用神经网络完成输入序列到输出序列的映射任务,如机器翻译、文本摘要、对话系统等等。transformer 的编码器解码器结构使其成为目前最强的序列到序列学习模型之一。

💡natural language processing

自然语言处理是指使计算机分析、理解和处理人类语言的技术与应用领域。如机器翻译、语音识别、情感分析等等。transformer 强大的语言表示能力使其成为当今自然语言处理領域的重要模型。

💡abstract continuous representation

视频中提到编码器要将输入序列映射为一个抽象的、连续的向量表示,其中包含了输入序列的全部学习到的信息。这种泛化的中间表达保留了语义信息,是序列到序列模型的关键。

Highlights

变压器通过注意力机制打破了许多NLP记录,推动了当前的技术状态

注意力机制允许模型关联输入序列中的每个词语与其他词语

位置编码通过正弦和余弦函数为每个词语的嵌入向量添加位置信息

多头注意力允许每个头学习不同的表示,从而为编码器提供更多的表示能力

残差连接有助于网络训练,允许梯度直接通过网络流动

解码器是自动回归的,它逐步生成文本序列

遮掩被用来防止解码器参考未来令牌

第一个多头注意力层计算解码器输入的注意力分数

第二个多头注意力层将编码器输出与解码器输入相关联

堆叠多个编码器和解码器层可以增强模型的预测能力

解码器预测单词概率分数,选择概率最高的作为预测单词

变压器使用注意力机制产生前所未有的NLP结果

循环神经网络由于短期记忆受限,变压器往往更优

变压器适合对更长序列进行编码或解码

注意力机制使NLP行业能达到前所未有的结果

Transcripts

transformers are taking the natural

language processing world by storm these

incredible models are breaking multiple

NLP records and pushing the

state-of-the-art they are used in many

applications like machine language

translation conversational chat BOTS and

even a power better search engines

transformers are the rage and deep

learning nowadays but how do they work

why are they outperformed a previous

king of sequence problems like recurrent

neural networks gr use and LS tiens

you've probably heard of different

famous transformer models like Burt CBT

and GB t2 in this video we'll focus on

the one paper that started it all

attention is all you need to understand

transformers we first must understand

the attention mechanism to get an

intuitive understanding of the attention

mechanism let's start with a fun text

generation model that's capable of

writing its own sci-fi novel we'll need

to prime in a model with an arbitrary

input and a model will generate the rest

okay

let's make the story interesting as

aliens entered our planet and began to

colonize earth a certain group of

extraterrestrials begin to manipulate

our society through their influence of a

certain number of the elite of the

country to keep an iron grip over the

populace by the way I then just make

this up this was actually generated by

open AI is GPT to transformer model

shout out to hugging face for an awesome

interface to play with I'll provide a

link in description okay so the model is

a little dark but what's interesting is

how it works as a model generate tax

word by word it has the ability to

reference or tend to words that is

relevant to the generated word how the

model knows which were to attend to is

all learned while training with

backpropagation our intends are also

capable of looking at previous inputs

too but the power of the attention

mechanism is that it doesn't suffer from

short-term memory rnns have a shorter

window to reference from so when a story

gets longer rnns can't access word

generated earlier in the sequence

this is still true for gr use and L

STM's although they do have a bigger

capacity to achieve longer term memory

therefore having a longer window to

reference from the attention mechanism

in theory and given enough compute

resources have an infinite window to

reference from therefore being capable

of using the entire context of the story

while generating the text this power was

demonstrated in the paper attention is

all you need when the author's introduce

a new novel neural network called the

Transformers which is an attention based

encoder decoder type architecture on a

high level the encoder Maps an input

sequence into an abstract continuous

representation that holds all the

learned information of that input to

decoder then takes our continuous

representation and step by step

generates a single output while also

being fed to previous output let's walk

through an example

the attention is all you need paper

applied to transformer model on a neuro

machine translation problem our

demonstration of the transformer model

would be a conversational chat bot the

example with taking an input tax hi how

are you and generate the response I am

fine

let's break down the mechanics of the

network step by step the first step is

feeding our input into a word embedded

layer a word embedding layer can be

thought of as a lookup table to grab a

learn factor of representation of each

word neural networks learned through

numbers so each word maps to a vector

with continuous values to represent that

word

next step is to inject positional

information into the embeddings because

a transformer encoder has no recurrence

like recurrent known networks we must

add information about the positions into

the input embeddings

this is done using positional encoding

the authors came up with a clever trick

using sine and cosine functions we won't

go into the mathematical details of the

positional codings in this video but

here are the basics for every odd time

step create a vector using the cosine

function for every even time step create

a vector using the sine function then

add those vectors to their corresponding

embedding vector this successfully gives

the network information on two positions

of each vector the sine and cosine

functions were chosen in tandem because

they have linear properties the model

can easily learn to attend to now we

have the encoder layer the encoder

layers job is to map all input sequence

into an abstract continuous

representation that holds the learned

information for that entire sequence it

contains two sub modules multi-headed

attention followed by a fully connected

network there are also residual

connections around each of the two sub

modules followed by a layer

normalization to break this down let's

look at the multi headed attention

module multi-headed attention Indian

code applies a specific attention

mechanism called self attention self

attention allows a model to associate

each individual word in the input to

other words in the input so in our

example it's possible that our model can

learn to associate the word you with how

M are it's also possible that the model

learns that word structured in this

pattern are typically a question

so respond appropriately to achieve self

attention we feed the input into three

distinct fully connected layers to

create the query key and value vectors

what are these vectors exactly I found a

good explanation on stock-exchange

stating the query key and value concept

comes from the retrieval system for

example when you type a query to search

for some video on YouTube

the search engine will map your query

against a set of keys for example video

title description etc associated with

candidate videos in the database then

present you with the best match video

let's see how this relates to self

attention the queries and keys undergoes

a dot product matrix multiplication to

produce a score matrix the score matrix

determines how much focus should a word

be put on other words so each word will

have a score to correspond to other

words in the time step the higher score

the more the focus this is how queries

are mapped to keys then the scores get

scaled down by getting divided by the

square root of the dimension of the

queries and the keys this is to allow

for more stable gradients as multiplying

values can have exploding effects next

you take the softmax the scaled score to

get the attention weights which gives

you probability values between 0 & 1

by doing the softmax the higher scores

get heightened and the lower scores are

depressed this allows the model to be

more confident on which words to attend

to then you take the attention weights

and multiply it by your value vector to

get an output vector the higher softmax

scores will keep the value of the words

the model learn is more important the

lower scores will drown out their

irrelevant words you feed the output

vector into a linear layer to process to

make this a multi-headed attention

computation you need to split the query

key in value into adding vectors before

applying self attention to split vectors

that goes through the same self

attention process individually each self

attention process is called a head each

head produces an output vector that gets

concatenated into a single vector before

go through in a final linear layer in

theory each head would learn something

different therefore giving the encounter

model more representation power okay so

that's multi-headed attention to sum it

up multi-headed attention is a module in

a transformer network that

you to the attention waits for the input

and produces an output vector with

encoded information on how each word

should attend to all other words in a

sequence

next step the multi-headed attention

output vector is added to the original

input this is called a residual

connection the output of the residual

connection goes through a layer

normalization the normalized residual

output gets fed into a point-wise

feed-forward network for further

processing the point-wise feed-forward

network are a couple of linear layers

with a relict evasion in between the

output of that is again added to the

input of the point-wise feed-forward

network and further normalized the

residual connections helps the network

train by allowing gradients to flow

through the networks directly the layer

normalizations are used to stabilize the

network which results in sustained

producing the training time necessary

and a point-wise feed-forward layer are

used to further process the attention

output potentially giving it a richer

representation

and that wraps up the encoded layer all

these operations is for the purpose of

encoding the input to a continuous

representation with attention

information this will help the decoder

focus on the appropriate words in the

input during the decoding process you

can stack the encoder and times to

further encode the information where

each layer has the opportunity to learn

different attention representations

therefore potentially boosting the

predictive power of the transformer

network now we move on to the decoder

the decoders job is to generate text

sequences the decoder has similar sub

layers as the encoder it has two

multi-headed attention layers a

point-wise feed-forward layer with

residual connections and layer

normalization after each sub layer these

sub layers behave similarly to layers in

the encoder but each multi-headed

attention layer has a different job it's

capped off with a linear layer that acts

like a classifier and a soft Max to get

the word probabilities the decoder is

auto regressive it takes in the list of

previous outputs as inputs as well as

the encoder outputs that contains the

attention information from the input the

decoder stops decoding when it generates

an end token as an output let's walk

through the decoding steps the input

goes through an embedding layer in a

position on coding layer to get

positional embeddings the positional

embeddings gets fed into the first

multi-headed attention layer which

computes the attention score for the

decoders input this multi-headed

attention layer operates slightly

different since the decoders

autoregressive and generates the

sequence word-by-word you need to

prevent it from condition into future

tokens for example when computing

attention scores on the word am you

should not have access to the word fine

because our word is a future word that

was generated after the word am should

only have access to itself and the words

before this is true for all other words

where they can only attend to previous

words we need a method to prevent

computing attention scores for future

words this method is called masking

to prevent the decoder from looking at

future tokens you apply a look-ahead

mask the mask is added before

calculating the softmax and after

scaling the scores let's take a look at

how this works the mask is a matrix

that's the same size as the attention

scores filled with values of materials

and negative infinity x' when you add

the mask to the scale attention scores

you get a matrix of scores with the top

right triangle filled with negative

infinity x' the reason for this is once

you take the softmax of the mask scores

the negative infinity is get zeroed out

leaving a zero attention score for

future tokens as you can see the

attention scores for M have values for

itself and all other words before it but

zero for the word fine this essentially

tells the model to put no focus on those

words this masking is the only

difference on how the attention scores

are calculated in the first multi-headed

attention layer this layers still have

multiple heads that the masks are being

applied to before getting concatenated

and fed through a linear layer for

further processing the output of the

first multi-headed attention is a mask

output vector with information on how

the model should attend on the decoders

inputs

now on to the second multi-headed

attention layer for this layer the

encoders output are the queries in the

keys in the first multi-headed attention

layer outputs are the values this

process matches the encoders input to

the decoders input allowing the decoder

to decide which encoder input is

relevant to put focus on the output of

the second multi-headed attention goes

through a point wise feed-forward layer

for further processing the output of the

final point wise feed-forward layer goes

through a final linear layer that access

a classifier the classifier is as

biggest number of classes you have for

example if you have 10,000 classes for

10,000 words the output of that

classifier will be of size 10,000 the

output of the classifier again gets fed

into a soft max layer the soft max layer

produced probability scores between 0

and 1 for each class we take the index

of the highest probability score and

that equals our predicted word the

decoder didn't taste the output and adds

it to the list of decoder inputs and

continue decoding again until end token

is predicted for our case the highest

probability prediction is the final

class which is assigned to the end token

this is how the decoder generates the

output the decoder can be stacked n

layers high each layer taking in inputs

from the encoder and the layers before

it by stacking layers the model can

learn to extract and focus on different

combinations of attention from its

attention heads potentially boosting its

predictive power and that's it that's

the mechanics of the transformers

transformers leverage the power of the

attention mechanism to make better

predictions recur known networks trying

to achieve similar things but because

they suffer from short term memory

transformers are usually better

especially if you want to encode or

generate longer sequences because of the

transformer architecture the natural

language processing industry can now

achieve unprecedented results if you

found this helpful

hit that like and subscribe button also

let me know in comments what you'd like

to see next and until next time thanks

for watching