Hebbian & Competitive | NN Learning Mechanisms | Neural Networks

Summary

In this video, the presenter dives into the Hebbian learning mechanism, one of the oldest learning rules in neural networks. Introduced by Donald Hebb in 1949, the principle is explained with reference to the hippocampus' role in learning. The script highlights key features such as the time-dependent, local, and interactive nature of Hebbian learning, with a focus on positive, negative, and uncorrelated correlations between presynaptic and postsynaptic neurons. It also touches upon various synaptic modifications, including Hebbian, Anti-Hebbian, and Non-Hebbian learning. Additionally, methods like the forgetting factor and co-variance hypothesis are introduced to address issues with the activity product rule. The session concludes with a brief introduction to competitive learning, setting the stage for further discussion.

Takeaways

• 😀 Hebbian learning was introduced by Donald Hebb in 1949, stating that simultaneous firing of neurons increases the synaptic strength between them.
• 😀 The Hebbian learning mechanism is based on the principle that if two neurons (A and B) repeatedly fire together, the connection between them becomes stronger.
• 😀 The learning mechanism is time-dependent: if presynaptic and postsynaptic neurons fire together, synaptic strength increases, otherwise it decreases.
• 😀 Hebbian learning is a local mechanism, meaning that synaptic strength is influenced only by the activities of the two neurons involved, without any external factors.
• 😀 Hebbian learning is strongly interactive: the change in synaptic strength is influenced by the activities of both presynaptic and postsynaptic neurons.
• 😀 Hebbian learning is correlational: synaptic modifications depend on the correlation between presynaptic and postsynaptic activity, leading to positive, negative, or uncorrelated states.
• 😀 Positive correlation leads to an increase in synaptic strength, while negative correlation weakens the synaptic connection, and uncorrelated states leave the synaptic strength unchanged.
• 😀 The mathematical model for Hebbian learning is based on the Activity Product Rule, which shows positive correlation but can lead to an unbounded increase in synaptic weight.
• 😀 To address the issue of unbounded growth in synaptic weight, solutions like introducing a forgetting factor and using a co-variance hypothesis are proposed.
• 😀 Hebbian learning is unsupervised, as it doesn’t require a desired output; it only adjusts based on the input signals and their correlation.
• 😀 Competitive learning involves output neurons competing against each other, where only one neuron is activated at a time, and neurons inhibit others to win the activation.

Q & A

What is Hebbian learning?

Hebbian learning is a neural learning rule proposed by Donald Hebb in 1949. It suggests that if two neurons (A and B) are activated simultaneously, the synaptic strength between them increases. This mechanism is similar to the physical changes in the brain's hippocampus when learning something new.

What are the key rules of Hebbian learning?

The two main rules of Hebbian learning are: 1) If both the presynaptic neuron (j) and postsynaptic neuron (k) are excited simultaneously, the synaptic strength (wkj) will increase. 2) If they fire asynchronously, the synaptic strength will decrease.

What are the properties of a Hebbian network?

The properties of a Hebbian network are: 1) Time-dependence: The synaptic modification depends on the timing of presynaptic and postsynaptic activities. 2) Locality: Synaptic strength depends only on the activities of the connected neurons. 3) Interactivity: The change in synaptic strength depends on the interaction of both neurons. 4) Correlation: Modification occurs based on the co-occurrence of presynaptic and postsynaptic activities.

What is meant by 'positive correlation' in Hebbian learning?

Positive correlation occurs when the presynaptic and postsynaptic neurons fire simultaneously. According to Hebbian learning, this will strengthen the synapse between the two neurons.

What happens during negative correlation in Hebbian learning?

Negative correlation happens when two neurons fire asynchronously. In this case, the synaptic strength between the neurons will weaken.

What is the meaning of uncorrelated activities in Hebbian learning?

Uncorrelated activities occur when the two neurons are in a relaxed state and do not exhibit any specific pattern of firing. The impact of uncorrelated activities on the synaptic strength is debated and can be designed in various ways in a neural network.

What are the different types of synaptic modifications?

Synaptic modifications can be classified into three types: Hebbian, Anti-Hebbian, and Non-Hebbian. In Hebbian learning, synaptic strength increases with positive correlation. In Anti-Hebbian learning, synaptic strength increases with negative correlation. Non-Hebbian modifications do not involve either Hebbian or Anti-Hebbian rules.

What problem arises from the activity product rule in Hebbian learning?

The activity product rule leads to a problem where the synaptic strength can increase without bounds over time. This can cause the synaptic weights to become too large for the hardware to process.

How can the problem with the activity product rule be addressed?

The problem can be addressed by introducing a forgetting factor or using a covariance hypothesis. The forgetting factor limits the increase in synaptic strength, while the covariance hypothesis averages the presynaptic and postsynaptic values over time.

Is Hebbian learning supervised or unsupervised?

Hebbian learning is an unsupervised learning method. The system learns based on the input pairs alone, without any desired output pairs. The network determines the output based on the patterns it identifies from the input data.