Self-supervised learning and pseudo-labelling

Summary

This video delves into self-supervised learning and pseudo-labeling, key concepts in machine perception. It highlights the limitations of supervised learning and explores how self-supervised learning, inspired by human multimodal learning, can improve model performance. The video discusses various pretext tasks and the instance discrimination approach, which leverages visual similarity for representation learning. Additionally, it covers pseudo-labeling in semi-supervised learning, demonstrating its effectiveness in tasks like word sense disambiguation and image classification, showcasing its potential in handling large unannotated datasets.

Takeaways

• 🤖 **Self-Supervised Learning Motivation**: The need for self-supervised learning arises from the limitations of supervised learning, where large annotated datasets are required, and models still make mistakes that humans wouldn't.
• 👶 **Inspiration from Human Development**: Human babies learn in an incremental, multi-modal, and exploratory manner, which inspires self-supervised learning methods that mimic these natural learning strategies.
• 🔄 **Redundancy in Sensory Signals**: Self-supervised learning leverages redundancy in sensory signals, where the redundant part of the signal that can be predicted from the rest serves as a label for training a predictive model.
• 🧠 **Helmholtz's Insight**: The concept of self-supervision is rooted in Helmholtz's idea that our interactions with the world can be thought of as experiments to test our understanding of the invariant relations of phenomena.
• 🐮 **Self-Supervised Learning Defined**: It involves creating supervision from the learner's own experience, such as predicting outcomes based on movements or changes in the environment.
• 🔀 **Barlow's Coding**: Barlow's work suggests that learning pairwise associations can be simplified by representing events in a way that makes them statistically independent, reducing the storage needed for prior event probabilities.
• 📈 **Pseudo-Labeling**: Pseudo-labeling is a semi-supervised learning technique where a model predicts labels for unlabeled data, which are then used to retrain the model, often leading to improved performance.
• 🎲 **Pretext Tasks**: In computer vision, pretext tasks like image patch prediction or jigsaw puzzles are used to train models without explicit labeling, hoping they learn useful representations of the visual world.
• 🔄 **Instance Discrimination**: A powerful self-supervised learning approach that trains models to discriminate between individual instances, leading to better visual similarity capture compared to semantic class discrimination.
• 🔧 **Practical Challenges**: There are practical challenges in creating effective pretext tasks, such as the risk of models learning to 'cheat' by exploiting unintended cues rather than understanding the task's underlying concepts.

Q & A

What are the two main topics discussed in the video?

The two main topics discussed in the video are self-supervised learning and pseudo-labeling.

What is the motivation behind self-supervised learning?

The motivation behind self-supervised learning is to improve machine perception by taking inspiration from early stages of human development, particularly the way humans learn in a multi-modal and incremental manner.

How does self-supervised learning differ from supervised learning?

In self-supervised learning, the learner creates its own supervision by exploiting redundant signals from the environment, whereas in supervised learning, a model is trained using manually annotated data.

What is the role of redundancy in sensory signals in self-supervised learning?

Redundancy in sensory signals provides labels for training a predictive model by allowing the learner to predict one part of the signal from another, thus creating a learning target without external supervision.

What is a pretext task in the context of self-supervised learning?

A pretext task is a task that is not the final goal but is used to learn useful representations of the data. It is often a game-like challenge that the model must solve, which in turn helps it learn about the visual world.

What is pseudo-labeling and how does it work?

Pseudo-labeling is a semi-supervised learning algorithm where a classifier is first trained on labeled data, then used to predict labels for unlabeled data. These predicted labels, or pseudo-labels, are then used to retrain the classifier, often iteratively.

Why is pseudo-labeling effective when large quantities of data are available?

Pseudo-labeling is effective with large data sets because it leverages the unlabeled data to improve the classifier's predictions, leading to better performance, especially when combined with techniques like data augmentation.

What is the 'noisy student' approach mentioned in the video?

The 'noisy student' approach is a method where an initial model trained on labeled data infers pseudo-labels for unlabeled data, and then a higher capacity model is trained on these pseudo-labeled data, often with heavy use of data augmentation.

How does self-supervised learning relate to human perception development?

Self-supervised learning relates to human perception development by mimicking the way humans learn from their environment through exploration and interaction, without the need for explicit labeling.

What challenges are there in creating effective pretext tasks for self-supervised learning?

Creating effective pretext tasks can be challenging because they need to be designed carefully to ensure the model learns useful representations rather than exploiting unintended shortcuts or low-level signals.

Outlines

00:00

🤖 Introduction to Self-Supervised Learning and Pseudo-Labeling

This paragraph introduces the video's focus on self-supervised learning and pseudo-labeling, topics derived from lectures given at the University of Cambridge. The speaker outlines the structure of the video, starting with self-supervised learning and then moving to pseudo-labeling. The motivation for self-supervised learning is discussed in the context of machine perception, highlighting the limitations of supervised learning despite its successes. The speaker points out that even high-capacity models make mistakes and suggests looking to human developmental stages for inspiration. The learning strategies of human babies are examined, emphasizing incremental, social, physical, exploratory, language-based, and multi-modal learning. The challenges of implementing these strategies in machine learning are acknowledged, especially the practical barriers to embodied learning. The paragraph concludes by setting the stage for a discussion on multi-modal learning and self-supervised methods inspired by human learning.

05:01

🔍 The Concept of Self-Supervised Learning

The paragraph delves into the concept of self-supervised learning, which involves creating one's own supervision. It references historical insights by Helmholtz and Barlow on perception and redundancy in sensory signals. The importance of redundancy for learning new associations is emphasized, as it provides a predictable signal from which to learn. The discussion then turns to computational tricks, such as using minimum entropy coding to avoid a combinatorial explosion of storage for prior event probabilities. The paragraph also covers the evolution of self-supervised learning, from early works by Dasar to modern approaches like instance discrimination and momentum contrast. The latter, in particular, addresses the challenge of maintaining an up-to-date memory bank for effective learning. The paragraph illustrates how self-supervised learning leverages the redundancy in multimodal signals to train predictive models without external annotations.

10:02

🎲 Pretext Tasks in Self-Supervised Learning

This section explores pretext tasks used in self-supervised learning for computer vision. Pretext tasks are games or challenges that models must solve to learn about the visual world without explicit labeling. Examples include predicting the relative position of image patches, in-painting, solving jigsaw puzzles, colorization, and counting objects. The paragraph warns of the potential for models to 'cheat' by exploiting low-level signals rather than learning the intended high-level features, as seen in a study where models used chromatic aberration to solve tasks. The importance of carefully constructing pretext tasks is highlighted, as they can significantly influence the learning process. The paragraph also mentions various creative pretext tasks that have been developed for training deep neural networks effectively.

15:02

🐾 Advanced Techniques in Self-Supervised Learning

The paragraph discusses advanced techniques in self-supervised learning, focusing on instance discrimination and its extension, momentum contrast. Instance discrimination trains a model to uniquely encode each image instance, while momentum contrast addresses the issue of stale memory banks by using a queue of recent samples and a momentum encoder. The paragraph also touches on specialized downstream tasks like learning tracking models from colorization in videos and detecting object keypoints without supervision through viewpoint factorization. The innovative approaches demonstrate how self-supervised learning can be adapted to specific tasks beyond general image representation.

20:03

🏷️ Pseudo-Labeling in Semi-Supervised Learning

The final paragraph shifts focus to pseudo-labeling within the realm of semi-supervised learning. Pseudo-labeling involves training a classifier on labeled data, using it to predict labels for unlabeled data (pseudo-labels), and then retraining the classifier on these pseudo-labels. The process can be iterated for improved performance. The paragraph provides an example of pseudo-labeling in word sense disambiguation and its effectiveness in large-scale image classification using ImageNet and JFT-300M datasets. The 'noisy student' technique is highlighted for its significant gains over traditional training methods. The paragraph concludes by emphasizing the growing value of pseudo-labeling as manual annotation struggles to keep pace with the vast amounts of sensory data being generated.

Keywords

💡Self-supervised learning

Self-supervised learning is a machine learning paradigm where the model generates its own labels from input data without human supervision. This concept is central to the video's theme, as it explores methods that allow machines to learn from data in a way that mimics human perception development. An example from the script highlights how self-supervised learning can be achieved by creating a plentiful supply of learning targets through movements and predictions about the environment.

💡Pseudo-labeling

Pseudo-labeling, also known as self-training or self-labeling, is a semi-supervised learning technique where a model trained on labeled data is used to predict labels for unlabeled data, which are then treated as pseudo-labels for further training. The video discusses this method as a powerful tool, especially when large unannotated datasets are available, and illustrates its effectiveness in improving large-scale image classification performance.

💡Multi-modal learning

Multi-modal learning refers to the process of learning from multiple types of sensory input data. In the context of the video, multi-modal learning is discussed as a human-inspired strategy that machines can adopt to improve their perception capabilities. The video suggests that by exploiting redundant signals across different modalities, machines can learn more robust representations of the world.

💡Benchmarks

Benchmarks in the video refer to standardized tests or measurements used to evaluate the performance of machine learning models, particularly in computer vision. While benchmarks are valuable for objective comparison, the video points out that they may also lead to a focus on narrow research objectives and that real-world performance is the ultimate goal.

💡Human perception system

The human perception system is the collective mechanisms by which humans interpret sensory information from the environment. The video discusses the limitations of current machine learning models in achieving human-like perception, suggesting that taking inspiration from early human developmental stages could improve machine perception.

💡Redundancy in sensory signals

Redundancy in sensory signals is the presence of repeated or overlapping information in the data received by the senses. The video explains that redundancy is crucial for learning new associations, as it provides the necessary prior knowledge to detect patterns and make predictions about the environment, which is a key aspect of self-supervised learning.

💡Pretext task

A pretext task is a secondary problem that a model is trained to solve, with the expectation that solving this task will lead to the learning of useful representations for the actual target task. The video provides several examples, such as predicting the relative position of image patches, which serve as a pretext for learning about objects and their relationships in the visual world.

💡Instance discrimination

Instance discrimination is a method in self-supervised learning where models are trained to distinguish between individual instances of data, even within the same class. The video explains that this approach can lead to learning powerful image representations, as it encourages the model to capture fine-grained visual details that differentiate similar-looking objects.

💡Momentum contrast (MoCo)

Momentum contrast is an extension of instance discrimination that addresses the challenges of maintaining an updated memory bank for all instances. The video describes how MoCo uses a queue of recently encoded samples and a momentum encoder to provide a more efficient and effective approach to instance discrimination, which in turn improves the model's ability to learn useful representations.

💡Contextual redundancy

Contextual redundancy refers to the redundancy found in multimodal signals that comes from the context surrounding a piece of information. The video discusses how contextual redundancy can be leveraged for self-supervised learning, using examples from natural language processing where unlabeled text corpora provide low-level supervision through the context in which words appear.

Highlights

Self-supervised learning and pseudo-labeling are key areas of focus in advancing machine perception.

Supervised learning has made significant progress in machine perception but still has limitations.

Human perception development provides inspiration for improving machine learning.

Human babies learn incrementally in a dynamic environment with multi-modal experiences.

The challenge of embodied learning is highlighted by Alan Turing's observations from 1948.

Self-supervised learning aims to create its own supervision, inspired by human multi-modal learning.

Redundancy in sensory signals is crucial for learning, as noted by Barlow.

Predictive models can be trained using redundant signals to provide labels for learning.

Self-supervised learning can be operationalized by learning from redundant signals across modalities.

Modern self-supervised learning methods often involve pretext tasks that provide a context for learning.

Pseudo-labeling is a semi-supervised learning technique that uses predictions to train on unlabeled data.

Instance discrimination is a powerful mechanism for self-supervised learning.

Momentum Contrast (MoCo) is an extension to instance discrimination that addresses memory bank issues.

Self-supervised learning has been applied to specialized tasks like tracking and keypoint detection.

Pseudo-labeling has been successfully applied to large-scale image classification.

The effectiveness of pseudo-labeling is underscored by its ability to leverage unannotated data.

The future of pseudo-labeling is promising as manual annotation struggles to keep up with data creation.

Transcripts

good day everyone

in this video i will aim to provide a

brief digest of material on

self-supervised learning and

pseudo-labeling

this material forms part of some

lectures i gave in 2021 as part of the

four f-12 lecture series at the

university of cambridge

to provide a brief outline for the video

we will start out with self-supervised

learning before moving on to

pseudo-labeling

we'll start out with the first topic

let's begin with the motivation for

self-supervised learning starting with a

summary of the state of the nation for

the world of machine perception

we have a number of reasons to be

cheerful

deep learning has given us remarkable

progress with the supervised learning

paradigm in which we gather large

collection of data and manually annotate

it and supervise a model with the

resulting annotated data

this has yielded truly major gains on

vision benchmarks

of course

benchmarks are not the goal in

themselves and they have drawbacks they

can trap you into a local minimum of

research ideas

but they have clear value in allowing an

objective comparison of different

methods

still

we have some cause for concern

and somehow it seems that we might still

have a long way to go

even the highest capacity models trained

on the largest annotated data sets

continue to make what we might call

silly mistakes mistakes that would be

never made by a human

perhaps more worryingly it seems that we

can just never get enough label data to

get close to the human perception system

this state of affairs prompts a natural

age-old question

can we take inspiration from the early

stages of development of human

perception to improve things

to answer this it's worth considering

the wealth of research that has studied

human development in some detail

human baby learning is incremental a

child learns in a continuously evolving

environment rather than a stationary

distribution

social

babies learn from other humans around

them particularly caregivers

physical

they offload knowledge to the physical

world around them and store information

with respect to their surroundings

exploratory once their curiosity is

aroused they try a lot of things to try

to find something that works

learning is language based allowing them

to not only communicate but also learn

abstractions that support generalization

finally their learning is highly

multi-modal experiencing sensations from

sight sound touch taste proprioception

balance and smell when these senses are

available

different modalities provide significant

redundancy among their inputs to learn

from

it's also interesting that babies

naturally build curricula for their

learning

careful analysis of the appearance of

objects observed by babies shows that

they follow a strong power law

a small number of objects are seen an

incredibly large number of times

such that the child becomes an expert in

recognizing and manipulating those

objects

okay this all sounds great and many

people have observed that our current

machine perception systems are doing

very little of these human inspired

learning strategies

why is that

i would say that the main barrier to

implementing the kind of embodied

perception learning exhibited

in babies has been practical

in fact this challenge was noticed by

alan turing who was interested in the

development of such machines and

observed in 1948 that

in order that the machine should have a

chance of finding things out for itself

it should be allowed to roam the

countryside and the danger to the

ordinary citizen would be serious

the takeaway here is that there is some

practical challenge to

embodied learning which is the general

term given to an agent possessing a body

that learns to interact with its

environment

it's possible that recent developments

in simulation may help us here but we're

still some way away from creating the

world in sufficient fidelity to have

fully addressed this problem

for that reason we will talk about a

family of methods that tries to make

progress principally on perhaps the

safest of the human learning

characteristics multi-modal learning

in particular we will discuss

self-supervised methods that are partly

inspired by human multimodal learning in

the sense of exploiting redundant signal

the essence of self-supervised learning

as you might infer from the name is that

the learner should create its own

supervision

this is an old idea that was nicely

articulated by helmholtz in his

legendary 1878 speech on the facts in

perception each movement we make by

which we alter the appearance of objects

should be thought of as an experiment

designed to test whether we have

understood correctly the invariant

relations of the phenomena before us

that is their existence in definite

spatial dimensions

so we can create a plentiful supply of

learning targets by simply moving and

checking whether our model of the world

was able to predict what we will see

the role of redundancy in sensory

signals was then considered more

explicitly by barlow who made the

observation that learning requires

previous knowledge

in particular to detect a new

association such as the event c

preceding event u

we need to know the prior probabilities

of c and u

if we have those then we can learn a new

association if we observe that c is

followed by u more frequently than would

be expected by chance

the key role of redundancy is that in

order to know what usually happens we

need redundancy in the input signal

which could be for example sensory

messages of the same event from

different modalities

the redundant signal is by definition

the part of the signal that can be

predicted from the remaining signal

this redundant signal provides labels

for training a predictive model

one computational trick that is nicely

illustrated in barlow's work relates to

the kinds of codes we can use to

represent events in our environment

the observation is the following

to test whether two events are

co-occurring more frequently than would

happen by chance we need to know their

prior joint probabilities

so when learning pairwise associations

between n events we need to store

n-squared co-occurrence probabilities

but if we've learned representations in

which events c and u are statistically

independent we can compute the chance

co-occurrence of c and u from the

product of their marginals that means we

only need to store n event probabilities

this can help to avoid a combinatorial

explosion of storage for prior event

probabilities

barlow himself suggested minimum entropy

coding as a scheme to obtain factorial

representations

but this idea applies more generally

it's always desirable to achieve this

property where possible

one of the first works to use the term

self-supervised that operationalized

this insight about learning from

redundant signal was proposed by

virginia dasar in the 1990s

who helpfully provided an intuitive

bovine based explanation

in supervised learning each time we see

an image of a cow as input

we are providing the corresponding cow

label

but it is implausible to collect all

labels required for such a task

unsupervised learning takes in the same

cow image and seeks to learn a powerful

representation of it without labels

in the self-supervised approach proposed

here

labels are derived from co-occurring

inputs across modalities providing

redundant signal

learning then proceeds by minimizing the

disagreement between class labels

predicted from each modality rather than

matching predictions against an external

annotation set

this is an idea that underlies a number

of modern approaches

it's also worth noting that in the

modern literature the distinction

between self-supervised and unsupervised

learning as given here can become a

little blurry

a general way to think about the

redundancy found in multimodal signals

is that the redundancy comes from the

context surrounding a piece of

information

in the natural language processing

community

unlabeled text corpora have been used

for a long time to provide low level

supervision for neural networks

with the hope that the distributed

representations learned by these models

will enable them to generalize

one example of such models are auto

regressive models in which the joint

probability of a sequence is factored

into a product of conditional

distributions

with each element in the sequence

conditioned on the previous elements

then

a network can be trained to maximize the

likelihood of a text corpus under this

factorization

for example

a network can be trained to predict the

next character in text to enable

compression

or to predict the next word to learn a

language model

a slightly more adventurous use of

context was explored by the word to vec

skip graham model which was trained to

predict the collection of words

surrounding any given word in the corpus

this is simple to train without labels

you simply pick a word in a sentence

mask its neighbors and then try to

predict those neighbors from the current

word

this work was really a breakthrough in

terms of language modelling performance

and highlighted the critical importance

of having lots of training data in

getting good word vectors

more recently various multitask masking

schemes have been considered

perhaps the best known is burt which was

trained to predict randomly masked words

in a sequence in addition to predicting

the next sentence in a corpus

this work showed amongst other things

the benefits of using a high capacity

transformer for language modeling

it's perhaps a little less obvious how

the same approach can be applied to

computer vision

the strategy that has been developed has

been to train a neural network by

tasking it with playing a game

which is often referred to as a pretext

task

now typically we don't care about the

performance of the model on the pretext

task itself but we hope that by solving

it a model learns good representations

of the visual world

work by carl dusch and colleagues

illustrates this idea nicely

a network is shown two image patches

these image patches have been cropped

from nearby regions in the same image

the pretext task for the model is to

guess the relative position of the red

dashed cropped to the blue crop

here's an example

ask yourself where does the red dashed

cropped belong to relative to the blue

crop

here's a second example again where does

the red dash crop belong to relative to

the blue crop

when you worked out that the answer to

question one was probably bottom right

and question two was probably top middle

you made use of your knowledge of buses

and trains despite almost certainly

having never seen this particular bus or

this particular train before

the key idea is that a model can only

solve these questions once it learns

about cats buses and trains and

importantly no labeling is required to

construct this task

this seems like a cute idea and it is

but a warning is needed

sometimes the model won't solve the task

in the way that you want it

in this work dorse et al found that the

network can learn to cheat by exploiting

a low level signal the chromatic

aberration that results from a camera

lens focusing different light

wavelengths differently

one color typically green is shrunk

towards the image center relative to the

others

once the network has figured this out

it can solve the problem trivially by

determining the absolute locations of

the patches relative to the lens without

learning anything at all about cats

buses trains or the host of other

interesting objects we'd like it to know

about

the authors solved this problem by

randomly dropping colour channels from

each patch so that the network could not

rely on this queue

but still it provides a cautionary tale

constructing pretext tasks requires a

great deal of care

partly because it works so well and

partly because it's a fun research

problem

researchers have come up with a number

of creative pretext tasks for training

deep neural

networks some examples include training

a network to in-paint by removing

patches of images and requiring the

model to fill them in

requiring the network to solve jigsaw

puzzles by giving it a shuffled set of

patches and asking it to rearrange them

to match an image

colorization in which a model is given

an image whose colour has been removed

and then tasked with predicting the

original colour version

there is training a model to count this

is a slightly ingenious idea the network

is given image patches and has to count

objects in those patches in such a way

that when applied to the full image the

total object counts match the sum of the

object counts across each patch

other work has trained a model to invert

again by training a second gan to

generate the latent codes of the

original gan

one particularly appealing pretext task

is to train a model to group pixels

according to optical flow

the idea here is to train a model that

learns pixel embeddings that are similar

if and only if those pixels tend to move

with the same velocity in videos which

is a way of encoding the prior knowledge

that the pixels on a common object tend

to move together

clustering has also proven to be highly

effective here a network alternates

between clustering its own feature space

and classifying the cluster membership

of each image

finally rotation prediction can be used

to exploit human photographer bias

we tend to photograph things the right

way up so we can rotate images and ask

the model to predict the rotation that

has been applied

this is also surprisingly powerful

i'd like to talk in a little more detail

about a task formulation that has

emerged as one of the most powerful

mechanisms for self-supervised learning

namely instance discrimination

one motivation for considering this idea

stems from the observation that despite

training with semantic labels

fully supervised convolutional neural

networks also appear to capture visual

similarity between instances

given an input image of a leopard

the classification scores of a fully

supervised model are strongest for the

class of leopard

but also for highly visually similar

classes like jaguar and cheetah and much

less so for glasses like lifeboat shop

cart and bookcase

an interesting question is then whether

the same property would emerge if we

trained a model to discriminate between

individual instances rather than

semantic classes

one work from wu at al explored this

idea

it trained a cnn to map all image

instances of a data set

to 128 dimensional vectors storing them

in a data structure called a memory bank

the model was trained to encourage each

instance to be mapped to a different

location on a 128 dimensional unit

sphere such that each image when encoded

with the cnn would uniquely retrieve its

memory bank vector

this model can be trained without labels

but empirically nevertheless learns

strong image representations

momentum contrast considered an

extension to this approach

the motivation behind this work was that

instance discrimination works well but

memory banks have an issue

on the one hand recomputing the features

stored in the bank ie one feature per

image in the data set for every update

to the cnn parameters would be

prohibitively expensive

on the other if memory bank instances

are not regularly updated they grow

increasingly stale with every

optimization step

which is sub-optimal for instance

discrimination

moco or momentum contrast aims to avoid

this stay on this issue by first

replacing the memory bank with a queue

of recently encoded samples fewer than

the full data set

and encoding queue samples with a

momentum encoder which is formed from a

slow moving average of query encoder

weights

moco uses some additional terminology

keys refer to instances encoded in the

queue with the momentum encoder

queries are instances to be compared

against keys

positive pairs are queries and keys

originating from the same image and by

extension negative pairs are queries and

keys originating from different images

the instance discrimination task is then

to match up queries against the keys

that represent their positive pairs

using what's known as an info nce loss

which is essentially a standard

cross-entropy softmax

finally the resulting query encoder then

provides a useful representation for

downstream tasks

the methods we've discussed so far have

focused principally on learning general

image representations

however the ideas behind self-supervised

learning have also been applied to

tackling more specialized downstream

tasks

one lovely piece of work by vondrick and

collaborators showed how to learn a

tracking model by performing

colorization

the key idea is to use colors across

unlabeled videos as a source of

supervision

the model is given a black and white

frame that it needs to colorize and a

reference black and white frame

at each location in the input frame it

is tasked with pointing to the location

in the reference frame that contains the

same color as the current location

this color is copied across to the input

frame from the color version of the

reference frame so that a loss can be

applied at the same place in the true

colors of the current frame

in practice this can be implemented by

training a cnn that produces a low

dimensional embedding at each location

of an image then performing pointing

from the target frame to the reference

frame by simply comparing the

similarities of the embeddings at each

location in each frame

given enough data the model indeed

learns to solve this task and the

authors show that without any labels it

gains the ability to track objects

across frames

another direction has considered

learning object key points without

supervision

the idea shown here for cat faces was to

learn a model that would learn to detect

consistent locations on an object

without any labels

the approach here was to use a concept

called viewpoint factorization

the essence of this idea is if you had a

good key point detector it should fire

on the same point of the cat's face even

as the cat's face moves around

in practice that can be encouraged by

enforcing what's known as equivariance

as the image is translated the key point

is also translated by the same amount

since we have no annotations just images

of cat faces these translations and

other geometric transformations are

generated via synthetic image warps and

a loss is used to ensure that the key

points move consistently with the warps

another direction has sought to learn

powerful video representations with a

simple idea

a model takes in several video clips one

of which has had its frames shuffled

training a model to predict which clip

was shuffled learns representations that

are particularly useful for action

recognition

we now turn to sudo labeling

i'd like to talk briefly about

semi-supervised learning and the

pseudo-labeling algorithm

semi-supervised learning considers the

setting in which a learner assumes

access to both labeled and unlabeled

data during training

typically to provide benefit the

unlabeled data is assumed to be

significantly larger than the label data

pseudo-labeling which is sometimes also

referred to as self-training or

self-labeling is a term that refers to

some variation of the following

algorithm

first a classifier is trained on the

labeled data

next the classifier is used to predict

the labels of the unlabeled data these

labels are referred to as pseudolabels

the classifier is then retrained on the

pseudo labels

often this process is iterated by

regenerating a new set of pseudo labels

retraining generating new pseudo labels

etc a nice illustration of this

algorithm is provided by the work of

yarovsky

who focused on the task of word sense

disambiguation across a full corpus

here the task is to determine the sense

in which a word is meant

for example the word plant may refer to

a manufacturing plant or a live plant

the algorithm proceeds by obtaining an

initial small collection of labelled

samples

and then using them to train a

classifier

then it predicts labels for unlabeled

sequences

keeping those that have high confidence

and optionally filtering and expanding

the labeled sets via some nlp heuristics

this stage is then repeated until

convergence to a final state

the reason for mentioning this algorithm

is twofold

one it's a pleasingly simple algorithm

two

when large quantities of data are

available it works remarkably well in a

wide range of settings

david yarovsky awkwardville's work noted

that it thrives on unannotated

monolingual corpora the more the merrier

as an example application in computer

vision pseudo-labeling was applied to

improving large-scale image

classification performance by taking

imagenet as a source of labelled images

and jft 300 million as a source of

unlabeled images

the method referred to as noisy student

trained an initial model on the labelled

data inferred pseudolabels and then

retrained higher capacity model on the

pseudo-labeled data

making heavy use of data augmentation

before repeating

noisy student led to significant gains

over image net only training

highlighting the effectiveness of this

technique

more broadly i think that pseudo

labeling algorithms are likely to prove

increasingly valuable in future as

manual annotation simply cannot keep up

with the scale of sensory data sets that

are now being created

we've reached the end

thank you for your attention