Demystifying how GPT works: From Architecture to...Excel!?! 🚀

Summary

このビデオシリーズでは、スプレッドシートを使ってGPT-2、ChatGPTの初期の祖先である大規模言語モデルを実装する方法を紹介します。GPT-2 smallを例に、テキストをトークンに分割し、それぞれのトークンを数値のリストにマッピングするプロセスから、マルチヘッドアテンションやマルチレイヤーパーセプトロンを含むモデルの構造までを、基本的なスプレッドシート機能を使用して解説します。このアプローチにより、現代のAI技術がどのように機能するかについて、より深い理解を得ることができます。今後のビデオでは、これらの各ステップについて詳しく説明していきます。

Takeaways

• 📊 このシリーズでは、基本的なスプレッドシート機能だけで大きな言語モデルGPT-2を実装していく。
• 🔍 テキストはトークンに分割され、これらは事前定義された辞書に基づいている。
• 🧮 トークンはバイトペア符号化というアルゴリズムを使用してトークンIDにマッピングされる。
• 📈 各トークンは、意味と位置をキャプチャする768の数字のリストにマッピングされる。
• 🔄 トークンからテキストへの埋め込みは、トークンの意味とプロンプト内の位置を反映している。
• 💡 マルチヘッドアテンションとマルチレイヤーパーセプトロン（ニューラルネットワークの一種）を通じて、トークン間の関係が解析される。
• 🔗 各ブロックの出力は次のブロックの入力として使用され、GPT-2は12の異なるレイヤーを通じてこのプロセスを繰り返す。
• 🎯 アテンションメカニズムは、文中の重要な単語やそれらの関係を識別する。
• 🤖 マルチレイヤーパーセプトロンは、与えられた文脈での単語の最も可能性の高い意味を決定する。
• 📝 最終的な言語ヘッドは、最も可能性の高い次のトークンを選択し、それを文に追加する。

Q & A

GPT-2のスプレッドシート実装では、どのようにテキストが処理されますか？

テキストはまずトークンに分割されます。各単語は事前定義された辞書に基づいてトークンに変換され、スプレッドシートの「プロンプトからトークンへ」タブでバイトペア符号化アルゴリズムにより最終的なトークンIDにマップされます。

埋め込み（embedding）とは何ですか、そしてGPT-2でどのように使用されますか？

埋め込みは、各トークンを数値のリストにマッピングするプロセスです。GPT-2スモールでは、各トークンは768の数値のリストにマップされ、これはトークンの意味と位置を捉えます。

位置埋め込みの目的は何ですか？

位置埋め込みは、トークンのプロンプト内の位置に応じて埋め込み値をわずかに変更することで、トークンの位置情報を捉えます。これにより、モデルは同じ単語でも異なる文脈での意味を区別できます。

多頭注意機構（multi-headed attention）の役割は何ですか？

多頭注意機構は、文中の単語がどのように関連しているかを理解し、重要な単語を特定することで、文脈を把握します。例えば、「he」が「Mike」を指すことを認識するなどです。

多層パーセプトロンの機能とは何ですか？

多層パーセプトロンは、単語の複数の意味を区別し、文脈に基づいて最も適切な意味を選択する役割を果たします。これにより、モデルは続く単語やトークンをより正確に予測できます。

言語ヘッド（language head）の役割は何ですか？

言語ヘッドは、最終ブロックの出力を確率セットに変換し、辞書内の既知のトークンから最も可能性の高いトークンを選択して文を完成させます。

GPT-2のスプレッドシート実装で、どのようにして次のトークンが選択されますか？

スプレッドシートでは、最終ブロックの出力から生成された確率に基づいて、最も可能性の高いトークンが選択されます。このデモでは、最も高い確率を持つトークンが選択されています。

GPT-2モデルの繰り返しプロセスにおける各ブロックの役割は何ですか？

GPT-2の各ブロックは、注意機構とパーセプトロンを含み、入力を受け取り、それを処理して次のブロックへの出力を生成します。このプロセスは、12の異なるレイヤーまたはブロックを通じて繰り返されます。

トークンがどのようにして埋め込みにマップされるかの例を教えてください。

例えば、'Mike' という単語は、トークンIDにマップされ、その後、768の数値からなるリストに変換されます。これにより、単語の意味とその位置が表現されます。

温度（temperature）ゼロとは何を意味しますか？

温度ゼロとは、モデルが最も可能性の高い1つのトークンのみを選択する状態を指します。これは一貫性のある出力を提供しますが、より多くのトークンから選択することで多様性を持たせることもできます。

Outlines

00:00

📝スプレッドシートでGPT2の概要

このパラグラフは、スプレッドシートを使用してGPT2の構造と処理の流れを実装していることを説明しています。入力テキストのトークン化、Embeddingsの生成、AttentionとMLPを使用したブロックの反復処理などの概要が述べられています。

05:03

😕日本語要約は難しい

2番目のパラグラフの内容は技術的で難解です。平易な日本語を使用して要約することをおすすめします。

Keywords

💡Tokenization

The process of splitting text into tokens. This is the first step that the model performs on the input text in order to convert it into a format that can be processed by the later stages of the model.

💡Embeddings

The numeric representations of tokens that capture semantic meaning as well as position. Each token is mapped to a 768-dimensional vector.

Highlights

The transcript walks through implementing GPT-2 in a spreadsheet using basic functions

The spreadsheet implements a smaller version called GPT-2 small but has the same architecture

Input text is split into tokens using byte-pair encoding

Tokens are mapped to lists of numbers called embeddings that capture meaning and position

There are 12 blocks with attention and multi-layer perceptron layers to refine predictions

Attention figures out which words are most relevant to refine the predictions

The final step predicts the most likely next token to complete the prompt

The spreadsheet picks the token with the highest probability for simplicity

The input text is parsed into tokens that map to IDs

Embeddings capture position as well as meaning of tokens

Attention identifies which words have the most influence on predictions

The blocks implement attention and neural network layers iteratively

Attention helps disambiguate meanings of words for the neural network

The final output predicts and selects the most likely next token

The spreadsheet uses the token with maximum probability for consistency

Transcripts

welcome to spreadsheets are all you need

how GPT Works where if you can read a

spreadsheet you can understand modern AI

That's because in this series we're

walking through a spreadsheet that

implements a large language model

entirely in basic spreadsheet functions

and not just any large language model

we're implementing gpt2 an early

ancestor of chat GPT now because it is a

spreadsheet it can only support a

smaller context link and it does

implement the smallest form of gpt2

known as gpt2 small but architecturally

for all intents and purposes it's the

same model that was breaking headlines

just a few short years ago let's take a

look under the hood how it works now in

subsequent videos we're going to go

through each of these stages step by

step but for now I'm going to touch on

each one lightly as a kind of table of

contents for future videos

in addition I've added a final column

here on the right that indicates what

tab in the spreadsheet corresponds to

what action inside

gpt2 let's start at the beginning after

you input your text it is split into a

series of tokens so for example let's

take Mike is quick he moves this would

be split into tokens per a predefined

dictionary now you'll note that every

single word here corresponds to a single

token but that is not always the case in

fact it's not uncommon for a single word

to be split into two three or even more

tokens let's take a look at the

spreadsheet so here's where you input

your prompt and because of the way the

parsing works you have to put each word

a separate line you can have to add the

spaces as well as the punctuation it

then gets taken to this sheet which is

or tab called prompt to tokens where it

goes through an algorithm called bite

pair encoding to map it to a final list

of known own token IDs you see right

here now that we have the tokens we need

to map them to a series of numbers

called an embedding every token is

mapped to a long list of numbers in the

case of gpt2 small it's a list of

768 numbers these capture both the

meaning as well as the position of each

token in the prompt let's see how this

works inside the

spreadsheet

okay so here we are in the spreadsheet

that implements this it's tokens to text

embeddings Tab and there's two parts to

it at the top you'll see our prompt

tokens Mike is quick he moves and these

are those prompt IDs we saw from the

earlier stat and then from columns three

onwards are the list the 768 numbers

that represent the semantic meaning of

the word Mike let's go look at column

770 and we can see where this list

ends right here you can see the list

ending let's go back to the

beginning and you'll notice there's

another list here the job of this list

is to actually change the tokens from

the list above to reflect their

different positions in the prompt let me

explain and demonstrate that here by

changing this word moves to the word

Mike which is the first

word in our prompt we'll go through

here we'll recalculate our

tokens we'll see we get Mike again then

we back to our tokens to text embeddings

we'll calculate the sheet and you'll

notice that Mike here has the same ID

and has the exact same embedding values

as it did does up here right row two and

row seven are totally identical that's

because the only job of this first set

of rows is to capture the semantic

meaning but when we take a look here at

this part where we have the position

embeddings you'll notice that the values

of the embedding for Mike at position

one are different than the values for

Mike at position six we've effectively

altered the values of the embeddings for

Mike slightly to reflect its different

position in the

prompt okay now that we've captured both

the meaning and the position the tokens

in the prompt they pass on to a series

of layers or blocks the first is

multi-headed attention and then the

second is what's known as a multi-layer

perceptron that's another name for a

neural network let's consider our

sentence again Mike is quick he moves

where we want the Transformer or GPT to

fill in the last word the attention

mechanism the first phase tries to

figure out what are the most important

words words in the sentence and how they

relate so for example the word he it

might recognize as referring to M

earlier in the prompt or it might

realize that the word moves and quick

probably relate this information is

important for the next layer the

multi-layer

perceptron so take for example this word

quick it has multiple meanings in

English it can mean moving fast it can

mean bright as in quick of wit it can

mean a body part as in the quick of your

fingernail and in Shakespearean English

it can even mean alive as opposed to

dead as in the phrase the quick and the

dead the information from the attention

layer that the word moves is there with

the word quick helps the multi-layer

perceptron disambiguate which of these

four meanings is most likely in this

sentence and that it's most likely the

first one moving in physical space and

it would use that to figure out what the

most likely next word to complete the

prompt is like the word quickly or the

word fast or the word around all of

which are about fast movement in

physical

space it's also important to note that

this attention then perceptron attention

then perceptron process happens

iteratively in gpt2 small it happens

across 12 different layers as it

iteratively refines its prediction of

what the next most likely word or token

should be

let's see how this is implemented in the

spreadsheet so you'll notice in the

spreadsheet there are these tabs block

zero block one block two all the way to

block 11 these are our 12 blocks and the

output of block zero becomes the input

of block one and the output of block one

becomes the input of block two so

they're all chained together all the way

through let's look inside one of these

blocks so here's the first block and

each block has about 16 steps in this

implementation steps one all the way to

around step 10 are basically your

attention mechanism and from Step 10 all

the way to the remaining 16 is the

multi-layer perceptron we're going to go

through this in a lot more detail in

future videos but I want to give you a

sneak peek of something so

here right at step seven is the heart of

the attention mechanism it tells us

where it's paying the most attention to

amongst the words so let's look at the

word he you'll notice the large lest

value here

0.48 is highest right here so it's

taking the word he and it's realizing

that most likely is referring to the

word Mike 0.48 is larger than any of the

other values so it's going to influence

the values it passes to the multi-layer

perceptron more than any of the other

words the other other words are getting

a much smaller influence on the output

it passes along let's take the word

moves again you'll notice that it's

looking most at the word mik and then

the next other word it's looking most at

is quick so it's going to use the

information from those two words again

that it passes to the next layer to try

and interpret the value or meaning of

the word

moves okay we're almost at the end the

last step is the language head which

figures out what the actual next likely

token is what it does is it takes the

output of the final block and converts

it into a set of probabilities across

all the known tokens and its dictionary

and then it picks from amongst the most

likely tokens randomly one of those

tokens to complete the

sentence in this case it's picked simply

the highest probability token which was

quickly and fills that in let's take a

look at the spreadsh

sheeet now in the spreadsheet you'll see

this is broken across three tabs layer

Norm which is a process we'll talk about

in a future video generating logits and

a softmax again Concepts we talk about

later to find finally get our predicted

token now in a true large language model

that you've probably played with it

actually picks from amongst a set of the

most likely tokens but in order to

simplify this sheet we just simply pick

from the very most likely token which

gives a very consistent output that's

why we've got a Max function it's just

simply taking the most likely output

this is what's known as having

temperature zero when you go outside of

temperature zero it starts picking from

more than just the top token and it

starts looking at the top 10 or 20 or 30

or more tokens and it picks from them

according to an

algorithm okay that's gpt2 at a glance

we'll be going through each of these

steps in future videos but for now I

hope that gives you a starting point as

to what's going on under the hood and

where you can see it happening live for

yourself inside the spreadsheet thank

you