But what is a neural network? | Chapter 1, Deep learning

Summary

The video script delves into the fascinating world of neural networks, explaining their structure and function in an accessible manner. It starts by marveling at the human brain's ability to effortlessly recognize handwritten digits, even in low-resolution images. The script then introduces the concept of neural networks as a mathematical model inspired by the brain, consisting of layers of interconnected 'neurons' that hold numeric values. It explores the network's architecture, weights, biases, and activation functions, illustrating how they work together to process input data and recognize patterns. The script promises to cover the learning process of neural networks in a subsequent video, aiming to provide a comprehensive understanding of this powerful technology.

Takeaways

• 😃 Neural networks are inspired by the human brain and are structured in layers of interconnected neurons (nodes that hold values).
• 🤖 The input layer neurons represent the pixel values of an image, the hidden layers perform computations to detect patterns, and the output layer neurons indicate the predicted digit.
• ⚙️ Each neuron connection has an associated weight and bias that determine how activations from one layer influence the next layer.
• 🧮 The activations are computed through weighted sums and activation functions like the sigmoid, allowing the network to learn complex representations.
• 🔢 For a 28x28 pixel input image and 10 output digits, the example network has around 13,000 weights and biases to be learned.
• 🧩 The goal is for the hidden layers to learn to detect relevant features like edges, patterns, and components that can be combined to recognize digits.
• 📚 Linear algebra concepts like matrices and matrix-vector multiplication provide a compact way to represent and compute the neural network operations.
• 📈 Early neural networks used sigmoid activation functions, but modern networks often use ReLU (rectified linear unit) activations, which are easier to train.
• 💡 Neural networks are just very complicated functions that map input data (like images) to output predictions (like digit classifications).
• 🔄 The next video will cover how these networks can 'learn' the appropriate weights and biases from training data to perform the desired task.

Q & A

What is the purpose of this video series?

The purpose of the video series is to provide an introduction to neural networks, explaining their structure and how they learn, using the example of a neural network that can recognize handwritten digits.

What is a neuron in the context of neural networks?

In the context of neural networks, a neuron is a simple computational unit that holds a number between 0 and 1, representing its activation level.

How is the activation of a neuron in one layer determined by the activations of the previous layer?

The activation of a neuron is determined by taking the weighted sum of the activations from the previous layer, where each connection has a weight associated with it. This weighted sum is then passed through an activation function, such as the sigmoid function, to squish the result between 0 and 1.

What is the purpose of the weights and biases in a neural network?

The weights determine the strength of the connections between neurons in adjacent layers and control how the activations from one layer influence the next layer. The biases act as an additional parameter that shifts the activation function, allowing the neuron to be active or inactive based on a specific threshold.

How many total weights and biases are there in the neural network discussed in the video?

The neural network discussed in the video has a total of almost 13,000 weights and biases.

What is the role of the hidden layers in a neural network?

The hidden layers in a neural network are responsible for detecting and combining low-level features (such as edges) into more complex patterns (such as loops or digits), enabling the network to learn hierarchical representations of the input data.

What is the motivation behind the layered structure of neural networks?

The layered structure of neural networks is motivated by the idea that intelligent tasks, such as image recognition or speech parsing, can be broken down into layers of abstraction, where each layer builds upon the representations learned in the previous layer.

What is the purpose of the sigmoid function in neural networks?

The sigmoid function is used to squish the weighted sum of activations from the previous layer into the range between 0 and 1, ensuring that the activations of the neurons in the current layer fall within the desired range.

What is the advantage of representing neural network computations using matrix notation?

Representing neural network computations using matrix notation allows for more compact and efficient calculations, as many libraries optimize matrix multiplication operations. This notation also makes it easier to communicate and understand the transformations happening between layers.

What is mentioned about the ReLU (Rectified Linear Unit) activation function?

The video mentions that relatively few modern neural networks use the sigmoid activation function anymore, and instead, the ReLU activation function is commonly used as it is easier to train, especially for deep neural networks.

Outlines

00:00

🧠 Neural Networks: The Basics

This section introduces the concept of recognizing handwritten digits, emphasizing the challenge and intricacy involved in programming a computer to identify images represented by a 28x28 pixel grid. It sets the stage for discussing the relevance of machine learning and neural networks, presenting neural networks not as buzzwords but as mathematical constructs that can learn and recognize patterns. The focus is on building a basic understanding of neural networks, highlighting their ability to learn from data and recognize handwritten digits, a classic example in machine learning. This foundation is critical for understanding more complex and variant forms of neural networks explored in contemporary research.

05:02

🔍 Neural Network Layers and Recognition Process

Paragraph 2 dives into the layered structure of neural networks, explaining how each layer's activations influence the subsequent layer, culminating in the network's ability to recognize digits based on the brightness of input pixels. It outlines the theoretical hope that neural networks can intelligently identify components of digits (like loops and lines) through layers of abstraction. This section further explains the conceptual hope for how neural networks might break down complex patterns into simpler, recognizable components, paving the way for understanding the importance of each layer in processing and recognizing patterns in data.

10:02

📐 Mechanics of Neural Network Functionality

This paragraph elucidates the technical workings of neural networks, particularly how activations within a network are determined by weighted sums and biases, which are adjusted through the sigmoid function to keep values between 0 and 1. It outlines the complexity involved in configuring these weights and biases across layers to facilitate pattern recognition, such as edges forming loops. The network's architecture, with its thousands of parameters, demonstrates the intricate process of 'learning' through adjusting these parameters to recognize patterns in data, a concept that underpins machine learning and neural network functionality.

15:05

👁️ Understanding Neural Networks as Functions

Paragraph 4 conceptualizes the entire neural network as a complex function that transforms 784 input numbers (pixels) into 10 output numbers (digits), facilitated by layers of matrix vector products and the sigmoid function. It emphasizes the network's capacity to adapt its thousands of parameters through learning to achieve accurate digit recognition. The discussion includes an evolution in neural network design, shifting from sigmoid functions to more effective activation functions like ReLU for training deeper networks. This transition underscores the ongoing developments and optimizations in neural network methodologies for improved performance in tasks like image recognition.

Keywords

💡Neural Networks

Neural networks are computing systems vaguely inspired by the biological neural networks that constitute animal brains. The video discusses neural networks in the context of machine learning, emphasizing their ability to recognize handwritten digits by processing input through layers of artificial neurons. These networks are capable of learning and making intelligent decisions based on data, illustrating the power and complexity of even the simplest form of neural networks in understanding and classifying images.

💡Machine Learning

Machine learning is a subset of artificial intelligence that focuses on the development of algorithms and statistical models that enable computers to perform specific tasks without explicit instructions, relying instead on patterns and inference. The script highlights the relevance and importance of machine learning in modern technology, with neural networks being a prime example of a machine learning model that learns from data to recognize patterns, such as handwritten digits.

💡Activation

In the context of neural networks, an activation refers to the value that a neuron holds, which is usually a function of its input. The video explains that neurons in the input layer of a neural network hold values representing the grayscale intensity of pixels in an image. Activations are crucial for the network's operation, as they propagate through the layers and determine the final output, showcasing the neural network's decision-making process.

💡Weights and Biases

Weights and biases are fundamental components of neural networks that adjust during the learning process. Weights control the strength of the connection between neurons in different layers, and biases are added to the weighted sum before it is passed through an activation function. The video emphasizes their role in determining how activations of one layer influence those of the next, effectively shaping the network's ability to recognize complex patterns like handwritten digits.

💡Sigmoid Function

The sigmoid function is a type of activation function used in neural networks to convert values into a range between 0 and 1, mimicking the way neurons are activated in the brain. The video describes its application in neural networks to ensure that neuron activations stay within this bounded range, facilitating the modeling of probabilities and decisions.

💡Learning

Learning in neural networks refers to the process of adjusting the weights and biases based on the data provided during training. The goal is to minimize the difference between the network's predictions and the actual target outcomes. The video plans to explore this concept in depth in a follow-up, highlighting how networks evolve and improve their ability to recognize handwritten digits through exposure to more data.

💡Handwritten Digit Recognition

Handwritten digit recognition is a task that involves identifying the numbers from 0 to 9 written by hand, serving as a classic example of a pattern recognition problem tackled by neural networks. The video uses this application to illustrate the process and challenges of designing neural networks capable of learning from pixel data to accurately classify these digits.

💡Layers

Layers in neural networks refer to the tiers of neurons structured in a sequence from input to output, including hidden layers in between. Each layer processes the activations from the previous layer and passes its output to the next. The video explains how these layers collectively work to abstract and interpret complex patterns in data, like identifying the features of handwritten digits.

💡ReLU

ReLU, or Rectified Linear Unit, is mentioned in the video as a modern alternative to the sigmoid function for activation in neural networks. It is simpler and often more effective in practice, being defined as the maximum of zero and the input. ReLU is highlighted for its role in facilitating the training of deep neural networks by providing a more efficient and less computationally expensive model of neuron activation.

💡Pattern Recognition

Pattern recognition is the ability to detect arrangements of characteristics or data that signal a relationship to specific conditions or attributes. The video discusses neural networks' capacity for pattern recognition, particularly in the context of identifying handwritten digits. By learning to recognize patterns in pixel values that correspond to specific numbers, neural networks demonstrate their utility in various applications that require the classification or interpretation of complex data.

Highlights

Brains can effortlessly recognize handwritten digits, even at low resolutions and with variations, which is an astonishingly complex task for computers.

Neural networks are inspired by the brain, with neurons represented as numbers between 0 and 1, and their activations are determined by the activations of neurons in the previous layer.

The hope is that neurons in hidden layers can learn to detect edges, patterns, and subcomponents that make up digits, with higher layers combining these features to recognize full digits.

Each neuron connection has a weight and bias, which are parameters that can be adjusted during training to capture relevant patterns.

The network has thousands of weights and biases that need to be tuned to make it perform the desired task, akin to tweaking various knobs and dials.

The transition of activations from one layer to the next can be represented compactly using matrix-vector notation and operations like matrix multiplication.

The entire network is essentially a function that takes in input data and produces output, albeit a highly complex function with many parameters.

The sigmoid function is used to squish the weighted sum of activations into the range between 0 and 1, mimicking the biological analogy of neurons being inactive or active.

Modern networks often use the ReLU (Rectified Linear Unit) activation function instead of sigmoid, as it is easier to train and perform well with deep networks.

Understanding the meaning of weights and biases can provide insights into how the network is solving the problem and guide further improvements.

The structure of the neural network, with its layers and connections, is motivated by the goal of breaking down the problem into simpler subproblems and combining solutions hierarchically.

The number of layers and neurons in each layer is a design choice that can be experimented with, as there is no definitive rule for the optimal structure.

Training the network involves finding the right settings for all the weights and biases to solve the problem, which is a complex optimization task.

Linear algebra concepts, such as matrices, vectors, and matrix-vector multiplication, are fundamental to understanding and working with neural networks.

While the network structure may seem complicated, its complexity is necessary to tackle challenging problems like digit recognition.

Transcripts

This is a 3.

It's sloppily written and rendered at an extremely low resolution of 28x28 pixels,

but your brain has no trouble recognizing it as a 3.

And I want you to take a moment to appreciate how

crazy it is that brains can do this so effortlessly.

I mean, this, this and this are also recognizable as 3s,

even though the specific values of each pixel is very different from one

image to the next.

The particular light-sensitive cells in your eye that are firing when

you see this 3 are very different from the ones firing when you see this 3.

But something in that crazy-smart visual cortex of yours resolves these as representing

the same idea, while at the same time recognizing other images as their own distinct

ideas.

But if I told you, hey, sit down and write for me a program that takes in a grid of

28x28 pixels like this and outputs a single number between 0 and 10,

telling you what it thinks the digit is, well the task goes from comically trivial to

dauntingly difficult.

Unless you've been living under a rock, I think I hardly need to motivate the relevance

and importance of machine learning and neural networks to the present and to the future.

But what I want to do here is show you what a neural network actually is,

assuming no background, and to help visualize what it's doing,

not as a buzzword but as a piece of math.

My hope is that you come away feeling like the structure itself is motivated,

and to feel like you know what it means when you read,

or you hear about a neural network quote-unquote learning.

This video is just going to be devoted to the structure component of that,

and the following one is going to tackle learning.

What we're going to do is put together a neural

network that can learn to recognize handwritten digits.

This is a somewhat classic example for introducing the topic,

and I'm happy to stick with the status quo here,

because at the end of the two videos I want to point you to a couple good

resources where you can learn more, and where you can download the code that

does this and play with it on your own computer.

There are many many variants of neural networks,

and in recent years there's been sort of a boom in research towards these variants,

but in these two introductory videos you and I are just going to look at the simplest

plain vanilla form with no added frills.

This is kind of a necessary prerequisite for understanding any of the more powerful

modern variants, and trust me it still has plenty of complexity for us to wrap our minds

around.

But even in this simplest form it can learn to recognize handwritten digits,

which is a pretty cool thing for a computer to be able to do.

And at the same time you'll see how it does fall

short of a couple hopes that we might have for it.

As the name suggests neural networks are inspired by the brain, but let's break that down.

What are the neurons, and in what sense are they linked together?

Right now when I say neuron all I want you to think about is a thing that holds a number,

specifically a number between 0 and 1.

It's really not more than that.

For example the network starts with a bunch of neurons corresponding to

each of the 28x28 pixels of the input image, which is 784 neurons in total.

Each one of these holds a number that represents the grayscale value of the

corresponding pixel, ranging from 0 for black pixels up to 1 for white pixels.

This number inside the neuron is called its activation,

and the image you might have in mind here is that each neuron is lit up when its

activation is a high number.

So all of these 784 neurons make up the first layer of our network.

Now jumping over to the last layer, this has 10 neurons,

each representing one of the digits.

The activation in these neurons, again some number that's between 0 and 1,

represents how much the system thinks that a given image corresponds with a given digit.

There's also a couple layers in between called the hidden layers,

which for the time being should just be a giant question mark for

how on earth this process of recognizing digits is going to be handled.

In this network I chose two hidden layers, each one with 16 neurons,

and admittedly that's kind of an arbitrary choice.

To be honest I chose two layers based on how I want to motivate the structure

in just a moment, and 16, well that was just a nice number to fit on the screen.

In practice there is a lot of room for experiment with a specific structure here.

The way the network operates, activations in one

layer determine the activations of the next layer.

And of course the heart of the network as an information processing mechanism comes down

to exactly how those activations from one layer bring about activations in the next layer.

It's meant to be loosely analogous to how in biological networks of neurons,

some groups of neurons firing cause certain others to fire.

Now the network I'm showing here has already been trained to recognize digits,

and let me show you what I mean by that.

It means if you feed in an image, lighting up all 784 neurons of the input layer

according to the brightness of each pixel in the image,

that pattern of activations causes some very specific pattern in the next layer

which causes some pattern in the one after it,

which finally gives some pattern in the output layer.

And the brightest neuron of that output layer is the network's choice,

so to speak, for what digit this image represents.

And before jumping into the math for how one layer influences the next,

or how training works, let's just talk about why it's even reasonable

to expect a layered structure like this to behave intelligently.

What are we expecting here?

What is the best hope for what those middle layers might be doing?

Well, when you or I recognize digits, we piece together various components.

A 9 has a loop up top and a line on the right.

An 8 also has a loop up top, but it's paired with another loop down low.

A 4 basically breaks down into three specific lines, and things like that.

Now in a perfect world, we might hope that each neuron in the second

to last layer corresponds with one of these subcomponents,

that anytime you feed in an image with, say, a loop up top,

like a 9 or an 8, there's some specific neuron whose activation is going to be close to 1.

And I don't mean this specific loop of pixels,

the hope would be that any generally loopy pattern towards the top sets off this neuron.

That way, going from the third layer to the last one just requires

learning which combination of subcomponents corresponds to which digits.

Of course, that just kicks the problem down the road,

because how would you recognize these subcomponents,

or even learn what the right subcomponents should be?

And I still haven't even talked about how one layer influences the next,

but run with me on this one for a moment.

Recognizing a loop can also break down into subproblems.

One reasonable way to do this would be to first

recognize the various little edges that make it up.

Similarly, a long line, like the kind you might see in the digits 1 or 4 or 7,

is really just a long edge, or maybe you think of it as a certain pattern of several

smaller edges.

So maybe our hope is that each neuron in the second layer of

the network corresponds with the various relevant little edges.

Maybe when an image like this one comes in, it lights up all of the

neurons associated with around 8 to 10 specific little edges,

which in turn lights up the neurons associated with the upper loop

and a long vertical line, and those light up the neuron associated with a 9.

Whether or not this is what our final network actually does is another question,

one that I'll come back to once we see how to train the network,

but this is a hope that we might have, a sort of goal with the layered structure

like this.

Moreover, you can imagine how being able to detect edges and patterns

like this would be really useful for other image recognition tasks.

And even beyond image recognition, there are all sorts of intelligent

things you might want to do that break down into layers of abstraction.

Parsing speech, for example, involves taking raw audio and picking out distinct sounds,

which combine to make certain syllables, which combine to form words,

which combine to make up phrases and more abstract thoughts, etc.

But getting back to how any of this actually works,

picture yourself right now designing how exactly the activations in one layer

might determine the next.

The goal is to have some mechanism that could conceivably combine pixels into edges,

or edges into patterns, or patterns into digits.

And to zoom in on one very specific example, let's say the

hope is for one particular neuron in the second layer to pick

up on whether or not the image has an edge in this region here.

The question at hand is what parameters should the network have?

What dials and knobs should you be able to tweak so that it's expressive

enough to potentially capture this pattern, or any other pixel pattern,

or the pattern that several edges can make a loop, and other such things?

Well, what we'll do is assign a weight to each one of the

connections between our neuron and the neurons from the first layer.

These weights are just numbers.

Then take all of those activations from the first layer

and compute their weighted sum according to these weights.

I find it helpful to think of these weights as being organized into a

little grid of their own, and I'm going to use green pixels to indicate positive weights,

and red pixels to indicate negative weights, where the brightness of

that pixel is some loose depiction of the weight's value.

Now if we made the weights associated with almost all of the pixels zero

except for some positive weights in this region that we care about,

then taking the weighted sum of all the pixel values really just amounts

to adding up the values of the pixel just in the region that we care about.

And if you really wanted to pick up on whether there's an edge here,

what you might do is have some negative weights associated with the surrounding pixels.

Then the sum is largest when those middle pixels

are bright but the surrounding pixels are darker.

When you compute a weighted sum like this, you might come out with any number,

but for this network what we want is for activations to be some value between 0 and 1.

So a common thing to do is to pump this weighted sum into some function

that squishes the real number line into the range between 0 and 1.

And a common function that does this is called the sigmoid function,

also known as a logistic curve.

Basically very negative inputs end up close to 0,

positive inputs end up close to 1, and it just steadily increases around the input 0.

So the activation of the neuron here is basically a

measure of how positive the relevant weighted sum is.

But maybe it's not that you want the neuron to

light up when the weighted sum is bigger than 0.

Maybe you only want it to be active when the sum is bigger than say 10.

That is, you want some bias for it to be inactive.

What we'll do then is just add in some other number like negative 10 to this

weighted sum before plugging it through the sigmoid squishification function.

That additional number is called the bias.

So the weights tell you what pixel pattern this neuron in the second

layer is picking up on, and the bias tells you how high the weighted

sum needs to be before the neuron starts getting meaningfully active.

And that is just one neuron.

Every other neuron in this layer is going to be connected to

all 784 pixel neurons from the first layer, and each one of

those 784 connections has its own weight associated with it.

Also, each one has some bias, some other number that you add

on to the weighted sum before squishing it with the sigmoid.

And that's a lot to think about!

With this hidden layer of 16 neurons, that's a total of 784 times 16 weights,

along with 16 biases.

And all of that is just the connections from the first layer to the second.

The connections between the other layers also have

a bunch of weights and biases associated with them.

All said and done, this network has almost exactly 13,000 total weights and biases.

13,000 knobs and dials that can be tweaked and

turned to make this network behave in different ways.

So when we talk about learning, what that's referring to is

getting the computer to find a valid setting for all of these

many many numbers so that it'll actually solve the problem at hand.

One thought experiment that is at once fun and kind of horrifying is to imagine sitting

down and setting all of these weights and biases by hand,

purposefully tweaking the numbers so that the second layer picks up on edges,

the third layer picks up on patterns, etc.

I personally find this satisfying rather than just treating the network as a total

black box, because when the network doesn't perform the way you anticipate,

if you've built up a little bit of a relationship with what those weights and biases

actually mean, you have a starting place for experimenting with how to change the

structure to improve.

Or when the network does work but not for the reasons you might expect,

digging into what the weights and biases are doing is a good way to challenge

your assumptions and really expose the full space of possible solutions.

By the way, the actual function here is a little cumbersome to write down,

don't you think?

So let me show you a more notationally compact way that these connections are represented.

This is how you'd see it if you choose to read up more about neural networks. 214 00:13:41,380 --> 00:13:40,520 Organize all of the activations from one layer into a column as a vector.

Then organize all of the weights as a matrix, where each row of that matrix corresponds

to the connections between one layer and a particular neuron in the next layer.

What that means is that taking the weighted sum of the activations in

the first layer according to these weights corresponds to one of the

terms in the matrix vector product of everything we have on the left here.

By the way, so much of machine learning just comes down to having a

good grasp of linear algebra, so for any of you who want a nice visual

understanding for matrices and what matrix vector multiplication means,

take a look at the series I did on linear algebra, especially chapter 3.

Back to our expression, instead of talking about adding the bias to each one of

these values independently, we represent it by organizing all those biases into a vector,

and adding the entire vector to the previous matrix vector product.

Then as a final step, I'll wrap a sigmoid around the outside here,

and what that's supposed to represent is that you're going to apply the

sigmoid function to each specific component of the resulting vector inside.

So once you write down this weight matrix and these vectors as their own symbols,

you can communicate the full transition of activations from one layer to the next in an

extremely tight and neat little expression, and this makes the relevant code both a lot

simpler and a lot faster, since many libraries optimize the heck out of matrix

multiplication.

Remember how earlier I said these neurons are simply things that hold numbers?

Well of course the specific numbers that they hold depends on the image you feed in,

so it's actually more accurate to think of each neuron as a function,

one that takes in the outputs of all the neurons in the previous layer and spits out a

number between 0 and 1.

Really the entire network is just a function, one that takes in

784 numbers as an input and spits out 10 numbers as an output.

It's an absurdly complicated function, one that involves 13,000 parameters

in the forms of these weights and biases that pick up on certain patterns,

and which involves iterating many matrix vector products and the sigmoid

squishification function, but it's just a function nonetheless.

And in a way it's kind of reassuring that it looks complicated.

I mean if it were any simpler, what hope would we have

that it could take on the challenge of recognizing digits?

And how does it take on that challenge?

How does this network learn the appropriate weights and biases just by looking at data?

Well that's what I'll show in the next video, and I'll also dig a little

more into what this particular network we're seeing is really doing.

Now is the point I suppose I should say subscribe to stay notified

about when that video or any new videos come out,

but realistically most of you don't actually receive notifications from YouTube, do you?

Maybe more honestly I should say subscribe so that the neural networks

that underlie YouTube's recommendation algorithm are primed to believe

that you want to see content from this channel get recommended to you.

Anyway, stay posted for more.

Thank you very much to everyone supporting these videos on Patreon.

I've been a little slow to progress in the probability series this summer,

but I'm jumping back into it after this project,

so patrons you can look out for updates there.

To close things off here I have with me Lisha Li who did her PhD work on the

theoretical side of deep learning and who currently works at a venture capital

firm called Amplify Partners who kindly provided some of the funding for this video.

So Lisha one thing I think we should quickly bring up is this sigmoid function.

As I understand it early networks use this to squish the relevant weighted

sum into that interval between zero and one, you know kind of motivated

by this biological analogy of neurons either being inactive or active.

Exactly.

But relatively few modern networks actually use sigmoid anymore.

Yeah.

It's kind of old school right?

Yeah or rather ReLU seems to be much easier to train.

And ReLU, ReLU stands for rectified linear unit?

Yes it's this kind of function where you're just taking a max of zero

and a where a is given by what you were explaining in the video and

what this was sort of motivated from I think was a partially by a

biological analogy with how neurons would either be activated or not.

And so if it passes a certain threshold it would be the identity function but if it did

not then it would just not be activated so it'd be zero so it's kind of a simplification.

Using sigmoids didn't help training or it was very difficult

to train at some point and people just tried ReLU and it happened

to work very well for these incredibly deep neural networks.

All right thank you Lisha.