Few-Shot Learning (1/3): Basic Concepts

Summary

In this lecture, Assistant Professor Shusen Wang from Stevens Institute of Technology introduces few-shot learning, a method for classification or regression with minimal samples. He uses an animal identification game to illustrate how humans can quickly learn to distinguish between new entities from a few examples. The lecture contrasts few-shot learning with traditional supervised learning, emphasizing the 'learn to learn' approach. It discusses the importance of the support set for making predictions on new classes and introduces key concepts like 'K-way N-shot' learning. The talk also touches on meta-learning and its application in few-shot learning, using relatable examples and mentioning datasets like Omniglot and Mini-ImageNet for practical research.

Takeaways

• 📚 Few-shot learning is a method of classification or regression using a very limited number of samples.
• 👨‍🏫 Dr. Shusen Wang introduces few-shot learning with an example of distinguishing between Armadillos and Pangolins using just four images.
• 🧠 The challenge for computers in few-shot learning is to make accurate predictions with minimal training data, which is easier for humans.
• 🔑 Key terms introduced are 'support set' and 'query', where the support set is a small set of labeled samples used for making predictions at test time.
• 🤖 The goal of few-shot learning is not just to recognize objects but to 'learn to learn', understanding similarities and differences between objects.
• 🐇 Even if the model hasn't seen certain classes during training, like squirrels, it can identify similarities between new, unseen objects.
• 🦊 The model can make predictions about new, unseen classes by comparing a query image to a support set provided at test time.
• 📈 The accuracy of few-shot learning depends on the number of 'ways' (classes in the support set) and 'shots' (samples per class).
• 📊 As the number of ways increases, accuracy typically decreases, while increasing the number of shots improves accuracy.
• 🔍 The basic idea is to train a model to predict similarity, which can then be applied to unseen queries with the help of a support set.
• 📊 Two commonly used datasets for evaluating few-shot learning models are Omniglot, a hand-written character dataset, and Mini-ImageNet, a smaller version of ImageNet.

Q & A

What is few-shot learning?

Few-shot learning is a machine learning approach where a model is trained to make classifications or predictions based on a very small number of samples.

How does the game with Armadillos and Pangolins illustrate few-shot learning?

The game demonstrates the concept of few-shot learning by showing how humans can distinguish between two animals using only four images, highlighting the ability to learn from minimal examples.

What is the challenge for computers in few-shot learning?

The challenge for computers in few-shot learning is that the small number of samples is insufficient for training a deep neural network, which typically requires large datasets.

What are the key terms 'support set' and 'query' in the context of few-shot learning?

The 'support set' refers to a small set of labeled samples used for making predictions at test time, while a 'query' is an unseen sample that the model must classify using the information from the support set.

How is few-shot learning different from standard supervised learning?

Few-shot learning differs from standard supervised learning in that it aims to train the model to understand similarities and differences between objects rather than recognizing specific objects from the training set.

What is meant by 'learning to learn' in meta-learning?

'Learning to learn' in meta-learning refers to the ability of a model to adapt and learn from new tasks with minimal data by leveraging the knowledge acquired from previous learning experiences.

Why is recognizing a squirrel difficult for a model trained without that class?

A model trained without the squirrel class will struggle to recognize it because it has not learned the characteristics of squirrels and lacks the necessary features to distinguish them from other objects.

What is the significance of the terms 'K-way' and 'N-shot' in few-shot learning?

In few-shot learning, 'K-way' refers to the number of classes in the support set, while 'N-shot' indicates the number of samples per class. These terms help define the complexity of the learning task.

How does the prediction accuracy in few-shot learning relate to the number of ways and shots?

The prediction accuracy in few-shot learning generally decreases as the number of ways (classes in the support set) increases and improves as the number of shots (samples per class) increases.

What is the role of a similarity function in few-shot learning?

A similarity function in few-shot learning measures the similarity between two samples. It is used to compare the query image with samples in the support set to determine the most similar class for prediction.

Which datasets are commonly used for evaluating few-shot learning models?

Omniglot and Mini-ImageNet are two widely used datasets for evaluating few-shot learning models. Omniglot contains hand-written characters from various alphabets, while Mini-ImageNet consists of natural images from 100 classes.

Outlines

00:00

🐾 Introduction to Few-Shot Learning

Professor Shusen Wang introduces few-shot learning, a machine learning approach that enables classification or regression with minimal training samples. He uses an interactive game involving Armadillos and Pangolins to illustrate how humans can distinguish between two unknown animals with just a few examples. This concept is then extended to machine learning, where the challenge is to make accurate predictions with limited data. The lecture emphasizes the importance of the 'support set' and 'query' in few-shot learning, where the support set is a small collection of examples too small to train a model but sufficient for making predictions. The goal is not just to recognize objects but to learn to learn, i.e., to understand similarities and differences between objects to generalize to new, unseen examples.

05:01

🧠 Meta-Learning and Its Principles

The concept of meta-learning is explored, which is a type of few-shot learning where the model learns to learn from a small number of examples. The analogy of a child learning to identify an otter by comparing it to cards with different animals is used to explain how meta-learning works. The child, like a meta-learning model, uses prior knowledge of similarities and differences to identify new objects. The lecture distinguishes between traditional supervised learning, which generalizes from a large dataset, and few-shot learning, which makes predictions based on a small 'support set'. The key difference is that few-shot learning deals with unknown classes not seen during training, requiring the model to adapt and learn quickly from limited information.

10:01

📊 Few-Shot Learning Terminology and Accuracy Factors

The lecture delves into the specific terminologies of few-shot learning, such as 'K-way' and 'N-shot', which describe the composition of the support set in terms of the number of classes (K) and the number of samples per class (N). The impact of these factors on prediction accuracy is discussed, with examples illustrating how increasing the number of ways (K) decreases accuracy due to the complexity of distinguishing between more classes, while increasing the number of shots (N) improves accuracy by providing more information per class. The importance of training a model to predict similarity, rather than specific classes, is highlighted as the fundamental approach to few-shot learning.

15:01

🔍 Practical Implementation and Datasets for Few-Shot Learning

The practical application of few-shot learning is discussed, focusing on the use of a similarity function to make predictions. The process involves comparing a query image to a support set and identifying the most similar sample to make a prediction. The lecture introduces two datasets commonly used for few-shot learning research: Omniglot, a hand-written character dataset, and Mini-ImageNet, a diverse image dataset. These datasets are crucial for evaluating few-shot learning models. The summary also mentions the Siamese network, a type of neural network that can be trained to serve as a similarity function, which will be covered in more detail in subsequent lectures.

Keywords

💡Few-shot learning

Few-shot learning is a subfield of machine learning that focuses on training models to make accurate predictions based on a very limited number of examples. This is particularly relevant in scenarios where large datasets are not available. In the context of the video, few-shot learning is exemplified by the challenge of distinguishing between armadillos and pangolins with only four images as a reference. The video emphasizes that while humans can learn to make such distinctions with minimal examples, it is a significant challenge for computers, especially when the number of samples is too small to train a deep neural network effectively.

💡Support set

In few-shot learning, the 'support set' refers to a small collection of labeled examples that provide additional context or information to aid in making predictions about new, unseen data. The video uses the analogy of showing a child a few cards with different animals to help them identify an unknown animal. In the script, the support set is crucial for the model to learn from the limited examples and make predictions about new classes that were not part of the original training data.

💡Query

A 'query' in the context of few-shot learning is an instance or sample for which the model is asked to make a prediction or classification. The video script describes a scenario where a query image is shown to the model, and the model must determine what the image represents based on the information provided by the support set. The term is used to highlight the model's ability to generalize and apply learned patterns to new, unseen data.

💡Meta-learning

Meta-learning, also known as 'learning to learn,' is a concept where a model learns to improve its learning process by recognizing patterns and relationships in data. In the video, meta-learning is illustrated through the analogy of a child learning to identify animals by observing their similarities and differences. The child's ability to apply this knowledge to identify an otter, despite never having seen one before, exemplifies the goal of meta-learning in AI: to enable models to adapt and learn from new data with minimal examples.

💡One-shot learning

One-shot learning is a specific case of few-shot learning where the model is expected to make accurate predictions based on a single example per class. The video script mentions one-shot learning when describing how a child can identify an otter with only one card as a reference. This concept is central to the video's theme, as it demonstrates the potential for AI models to learn and generalize from minimal data.

💡K-way N-shot

In few-shot learning, 'K-way N-shot' refers to a classification task where the support set is composed of K classes, each represented by N examples. This terminology is used to describe the structure of the support set and how it affects the model's ability to learn and make accurate predictions. The video script uses this term to explain how the complexity of the learning task increases with the number of classes (K) and how the accuracy of predictions can improve with more examples per class (N).

💡Similarity function

A 'similarity function' in the context of the video is a mathematical function that measures how similar two samples are. This function is crucial for few-shot learning, as it allows the model to compare a query image with images in the support set and determine the closest match. The video describes how an ideal similarity function would output a high value for identical items and a low value for dissimilar ones, enabling the model to make predictions based on similarity scores.

💡Siamese network

A Siamese network is a type of neural network architecture designed for learning to measure the similarity between two inputs. The video script mentions that in the next lecture, the Siamese network, which can serve as a similarity function, will be studied. This network is trained on a large-scale dataset and can be used to make predictions for unseen queries by comparing the query with samples in the support set.

💡Omniglot dataset

The Omniglot dataset is a collection of hand-written characters from various alphabets, used for few-shot learning research. The video script describes it as a small dataset with over a thousand classes, each having only 20 samples. This dataset is used to illustrate the challenges of few-shot learning, where the model must classify new characters with very few examples. The Omniglot dataset is significant in the video as it represents a real-world application of the few-shot learning concepts discussed.

💡Mini-ImageNet

Mini-ImageNet is another dataset mentioned in the video script, consisting of 100 classes with 600 samples each, totaling 60,000 samples. This dataset is used for few-shot learning research and provides a more complex and diverse set of images compared to Omniglot. The video uses Mini-ImageNet to demonstrate the application of few-shot learning in a visual domain with a larger and more varied dataset.

Highlights

Few-shot learning is about making predictions based on a very small number of samples.

Humans can distinguish between Armadillos and Pangolins using just 4 training samples, but can computers do the same?

The challenge of few-shot learning is that the number of samples is too small to train a deep neural network effectively.

Few-shot learning is differentiated from standard supervised learning by its focus on learning to learn rather than memorizing training data.

The model is trained to recognize the similarity and difference between objects, not specific objects themselves.

Support set and query are key terminologies in few-shot learning, where support set provides additional information at test time.

The model can identify whether two images contain the same kind of objects even if it hasn't been trained on those specific objects.

Meta-learning is about teaching the model to learn from new data without extensive retraining.

In meta-learning, the model uses a support set to make predictions about new, unseen classes.

The concept of 'learn to learn' is illustrated through an analogy of a child learning to identify animals at a zoo.

One-shot learning is a subset of few-shot learning where the model makes predictions based on seeing one example per class.

The number of ways (classes in the support set) and shots (samples per class) affect the prediction accuracy in few-shot learning.

Few-shot learning aims to train a function that predicts similarity between samples.

The Siamese network, which will be discussed in the next lecture, can serve as a similarity function for few-shot learning.

Omniglot and Mini-ImageNet are two widely used datasets for evaluating few-shot learning models.

Omniglot contains over 1,000 classes of hand-written characters, making it suitable for few-shot learning tasks.

Mini-ImageNet consists of 100 classes with 600 samples each, providing a more complex dataset for few-shot learning.

Transcripts

I am Shusen Wang.

I am an assistant professor at Stevens Institute of Technology.

In this lecture, I will give a short introduction to few-shot learning.

Few-shot learning means making classification or regression based on a very small number of samples.

Before getting started, let’s play a game.

I show you 4 images. Please look carefully.

The left two images are Armadillos.

The right two are Pangolins.

You may have never heard of Armadillo or Pangolin, but it doesn’t matter.

You just want to pay attention to their differences and try to distinguish the two animals.

If you don’t know their difference, I can give you some hint.

Look at their ears and size of the scales.

Now, I give you a query image.

Do you think it is Armadillo or Pangolin?

Most people don’t know the difference between Armadillo and Pangolin.

They may have not even heard of Armadillo or Pangolin.

But human can learn to distinguish the two animals using merely 4 training samples.

For a human, making a prediction based on 4 training samples is not hard.

But can computers do this as well?

If a class has only two samples, can computers make the correct prediction?

This is harder than the standard classification problem.

The number of samples is too small for training a deep neural network.

Please keep the terminologies in mind: support set and query.

Support set is a small set of samples.

It is too small for training a model.

Few-shot learning is the problem of making predictions based on a limited number of samples.

Few-shot learning is different from the standard supervised learning.

The goal of few-shot learning is not to let the model recognize the images in the training set and then generalize to the test set.

Instead, the goal is to learn to learn.

“Learn to learn” sounds hard to understand.

You can think of it in this way.

I train the model on a big training set.

The goal of training is not to know what elephant is and what tiger is.

The goal is not to be able to recognize unseen elephants and tigers.

Instead, the goal is to know the similarity and difference between objects.

After training, you can show the two images to the model and ask whether the two are the same kind of animals.

The model has learned the similarity and difference between objects.

So the model is able to tell that the contents in the two images are the same kind of objects.

Take a look at our training data again.

The training data has 5 classes which do not include the squirrel class.

Thus, the model is unable to recognize squirrels.

If you show an image of the squirrel to the model, the model does not know it is a squirrel.

When the model sees the two images, it does not know they are squirrels.

However, the model knows they look alike.

The model can tell you with high confidence that they are the same kind of objects.

For the same reason, the model has never seen a rabbit during training,

so it does not know the two images are rabbits.

But the model knows the similarity and difference between things.

The model knows that the contents of the two images are very alike.

So the model can tell that they are the same object.

Then I show the two images to the model.

While the model has never seen pangolin and bulldog,

the model knows the two animals look quite different.

The model believes they are different objects.

Now, I ask a different question.

I have a query image. I show it to the model and ask what it is.

The model is unable to answer my question.

The model has not seen this kind of object during training.

Then, I provide the model with additional information.

I show additional 6 images to the model.

I tell the model that they are fox, squirrel, rabbit, hamster, otter, and beaver.

Now, the model can answer my question.

The model compares the query image with each of the 6 images.

The model finds the query most similar to the otter image.

So the model believes the query is an otter.

"Support set" is meta learning’s jargon.

The small set of labeled images is called support set.

Note the difference between the training set and support set.

The training set is big.

Every class in the training set has many samples.

The training set is big enough for learning a deep neural network.

In contrast, the support set is small.

Every class has at most a few samples.

In this example, every class has only one sample.

It is impossible to train a deep neural network using such a small set of data.

The support set can only provide additional information at test time.

Here is the basic idea of few-shot learning.

We train a big model using a big training set.

Rather than training the model to recognize specific objects such as tiger and elephant in the training set,

we train the model to know the similarity and difference between objects

With the additional information provided by the support set, the model can tell the query image is an otter

although otter is not among the classes in the training set

I am going to explain what few-shot learning and meta-learning are.

You may have heard of meta-learning.

Few-shot learning is a kind of meta-learning.

Meta-learning is different from traditional supervised learning.

Traditional supervised learning asks the model to recognize the training data and then generalize to unseen test data.

Differently, meta learning’s goal is to learn to learn.

How to understand "learn to learn"?

You bring your kid to the zoo.

He’s excited to see the fluffy animal in the water which he has never seen before.

He asked you: daddy, what’s this?

Although he has never seen this animal before, he is a smart kid and can learn by himself.

Now, you give the kid a set of cards.

On every card, there is an animal and its name.

The kid has never seen the animal in the water.

He has never seen the animals on the cards, either.

But the kid is so smart that by taking a look at all the cards, he knows the animal in the water is an otter.

The animal in the water is most similar to the otter on the card.

Teaching the kid to learn by himself is called meta-learning.

Before going to the zoo, the kid was already able to learn by himself.

He knew the similarity and difference between animals.

Although he has never seen otter before, he could learn by himself.

By reading the cards, he knows the animal is an otter.

The kid wants to know the animal in the water which he has never seen before.

In meta-learning, the unknown animal is called a query.

You give him a set of cards and let him learn by himself.

The set of cards is the support set.

What is meta-learning?

Learning to learn by himself is called meta-learning.

In this example, letting the kid distinguish different animals is meta-learning.

Before going to the zoo, the kid has not heard of otter,

but he knew how to related the otter in the water with the otter on the card.

In this example, the kid learns to recognize otter using a set of cards.

There is only one card for every species.

He learns to recognize otter using only one card.

This is called one-shot learning.

Here I compare traditional supervised learning with few-shot learning.

Traditional supervised learning is like this.

First, learn a model using a big training set.

After the model is trained, we can use the model for making predictions.

We show a test sample to the model.

The test sample is never-seen-before; it is not in the training set.

Fortunately, this test sample is from a known class.

The test sample is a husky.

It belongs to this class.

There are hundreds of samples under the class “husky”.

Although the model has never seen this husky, the model has seen hundreds of huskies.

It is not hard for the model to tell this test sample is a husky.

Few-shot learning is a different problem.

The query sample is never seen before.

Furthermore, the query sample is from an unknown class.

The query sample is a rabbit.

It is not among the classes in the training set.

The model has never seen any rabbit during training.

This is the main difference from traditional supervised learning.

The training set does not have a rabbit class,

so the model does not know that the query sample is.

We need to provide the model with more information.

We can show the cards to the model.

Every card has an image and a name.

The set of cards is the support set.

By comparing the query with the cards, the model finds the query most similar to the rabbit card.

So the model predicts that the query is a rabbit.

Way and shot are terminologies of few-shot learning.

K-way means the support set has k classes.

In this example, the support set has 6 classes: fox, squirrel, rabbit, hamster, otter, and beaver.

So K is 6.

N-shot means every class has n samples.

In this example, every class has only one sample.

So n is 1.

This support set is called 6-way and 1-shot.

Take a look at another support set.

It has 4 classes: squirrel, rabbit, hamster, and otter.

So it is 4-way.

There are two samples in every class. So it is 2-shot.

The support set is called 4-way 2-shot.

When performing few-shot learning, the prediction accuracy depends on the number of ways and the number of shots.

In this figure, the x-axis is the number of ways,

that is, the number of classes in the support set.

The y-axis is the prediction accuracy.

As the number of ways increases, the prediction accuracy drops.

Why does this happen?

There is an otter in the zoo.

The kid does not know what it is.

I give the kid 3 cards and ask the kid to choose one out of the three.

This is 3-way 1-shot learning.

What if I give the kid 6 cards?

Then this would be 6-way 1-shot learning.

Which one do you think is easier, 3-way or 6-way?

Obviously, 3-way is easier than 6-way.

Choosing one out of 3 is easier than choosing one out of 6.

Thus 3-way has higher accuracy than 6-way.

In this figure, the x-axis is the number of shots,

that is, the number of samples per class.

The y-axis is the prediction accuracy.

As the number of shots increases, the prediction accuracy improves.

The phenomenon is easy to interpret.

The above is a 2-shot support set.

The below is a 1-shot support set.

With more samples, the prediction becomes easier.

Thus 2-shot is easier than 1-shot.

The basic idea of few-shot learning is to train a function that predicts similarity.

Denote the similarity function by sim(x, x’).

It measures the similarity between the two samples, x and x’.

Here are 3 images.

They are bulldog, bulldog, and fox.

Denote them by x1, x2, and x3.

Ideally, taking x1 and x2 as input, the similarity function outputs one,

which means the two animals are the same.

Taking x1 and x3 as input, or taking x2 and x3 as input,

the similarity function outputs zero,

which means the two animals are different.

The idea can be implemented in this way.

First, learn a similarity function from a large-scale training dataset.

The similarity function tells us how similar two images are.

In the next lecture, we will study the Siamese network which can be a similarity function.

The network can be trained using a large-scale dataset such as ImageNet.

After training, the learned similarity function can be used for making predictions for unseen queries.

We can use the similarity function to compare the query with every sample in the support set and calculate the similarity scores.

Then find the sample with the highest similarity score, and use it as the prediction.

I use this example to demonstrate how to make a prediction.

Given this query image, I want to know what the image is.

We can compare the query with every sample in the support set.

Compare the query with greyhound.

The similarity function outputs a similarity score of 0.2.

The similarity score between the query and the bulldog is 0.1.

The similarity between the query and the armadillo is 0.03.

Do the same for all the samples in the support set to get all the similarity scores.

Among those similarity scores, this 0.7 is the biggest.

Thus, the model predicts the query is an otter.

One-shot learning can be performed in this way.

Given a support set,

we can compute the similarity between the query and every sample in the support set to find the most similar sample.

If you do research on meta-learning, then you will need datasets for evaluating your model.

Here I introduce 2 datasets which are most widely used in research papers.

Omniglot is the most frequently used dataset.

The dataset is small; only a few megabytes.

Omniglot is a hand-written dataset similar to MNIST.

MNIST dataset is for digit recognition.

MNIST has 10 classes; each class has 6 thousand samples.

In contrast, Omniglot has over 1 thousand classes.

But each class has only 20 digits.

This makes the classification for Omniglot harder than MNIST.

You can download the dataset using the link or import the dataset using TensorFlow.

The dataset has 50 alphabets such as Hebrew, Greek, Latin, etc.

Every alphabet has many characters.

For example, Greek has 24 letters such as alpha, beta, gamma, all the way to omega.

For every character, there are 20 digits written by different people.

Here is a summary of Omniglot.

It has 50 alphabets including various languages like Latin, Greek, and Hebrew.

Every alphabet has multiple characters, for example, Greek has 24 letters.

The 50 alphabets have a total of 1623 unique characters.

Therefore, the dataset has 1623 classes.

Each character was written by 20 different people.

It means each class has 20 samples.

All the samples are 105-by-105 images.

The training set has 30 alphabets, which contain 964 characters and thus 964 classes.

The training set contains a total of 19,280 samples.

The test set has 20 alphabets, which contain 659 characters and thus 659 classes.

The test set has a total of 13,180 samples.

Another commonly used dataset is Mini-ImageNet.

It has 100 classes such as mushroom, orange, corn, bird, and snake.

Every class has 600 samples.

The dataset has a total of 60 thousand samples.

We have learned the basic concepts of few-shot learning and meta-learning.

In the next class, we will study the Siamese network for few-shot learning.