The LLM's RL Revelation We Didn't See Coming

Summary

This video delves into the evolving role of reinforcement learning (RL) in large language models (LLMs), highlighting recent research challenges and breakthroughs. The script discusses RL's shift from promising new reasoning discovery to enhancing pre-existing knowledge within models. It explores methods like RL from Verifiable Rewards (RLVR) and its use in domains like math and coding, while questioning its ability to generate new reasoning paths. The research challenges, such as the lack of generalization across models and the surprising impact of random rewards, are also examined, suggesting that RL methods need re-evaluation and further development for real-world application.

Takeaways

• 😀 RL in LMs has faced setbacks, transitioning from hope in discovering new reasoning paths to doubts about its ability to innovate reasoning strategies.
• 😀 RLHF (Reinforcement Learning from Human Feedback) fine-tunes models based on human feedback but suffers from limitations due to its subjective nature.
• 😀 RLVR (Reinforcement Learning from Verifiable Rewards) focuses on deterministic feedback, making it ideal for tasks with clear correctness criteria like math and coding.
• 😀 GRPO (Group Relative Policy Optimization) is an optimizer used in RLVR that improves model performance by comparing outputs within a group, promoting correctness without external human feedback.
• 😀 RLVR doesn’t generate new reasoning paths; instead, it amplifies existing knowledge, enhancing certain behaviors already present in the model.
• 😀 Longer responses generated by RLVR don’t always correlate with better performance; they might simply be the result of penalizing shorter, incorrect responses less.
• 😀 A study showed that models trained with RLVR do not outperform base models in all cases, especially with large sample sizes, where base models may shine in creative problem-solving.
• 😀 Distillation, not RLVR, has been found to introduce new reasoning processes into models, such as the DeepSeek R1 model, which could solve problems the base model couldn't.
• 😀 RLVR's role is questioned in terms of generalization across model families, with findings showing that Quinn models outperform others in certain tasks due to unique internal heuristics.
• 😀 Research has shown that incorrect or random rewards in RLVR can still improve performance, particularly for models like Quinn that excel at Python code reasoning.
• 😀 The future of RL in LMs lies in improving RLVR's ability to generalize, scale, and better assign rewards to unlock new reasoning capabilities.

Q & A

What is the main challenge currently faced by reinforcement learning (RL) in large language models (LLMs)?

The main challenge is that reinforcement learning (RL) in LLMs is not able to discover new reasoning processes as expected. Instead, it seems to only amplify and promote existing knowledge within the models, limiting the creation of novel reasoning paths.

What is the key distinction between RLHF and RLVR in reinforcement learning for language models?

RLHF (Reinforcement Learning from Human Feedback) trains a reward model based on human-labeled data to rank LLM responses. In contrast, RLVR (Reinforcement Learning from Verifiable Rewards) uses deterministic rewards based on correct or incorrect answers, providing binary feedback rather than relying on human preferences.

How does Group Relative Policy Optimization (GRPO) improve reinforcement learning methods in RLVR?

GRPO compares the performance of different outputs from the same model within a group and assigns rewards relatively, instead of maximizing rewards evaluated by a separate model. This allows RLVR to improve using task-specific feedback like correct or incorrect answers for math or code, while maintaining stable updates.

What role does the KL divergence penalty play in RLHF?

The KL divergence penalty in RLHF measures how much a new model’s predictions differ from its old ones, preventing the model from changing its behavior too much during optimization, thus maintaining a balance between new learning and existing knowledge.

What did the research published two months after the DeepSeek R1 paper reveal about RLVR and self-reflection in models?

The research showed that the increase in output length and self-reflection seen in models trained with RLVR was not a result of RLVR training itself. Instead, RLVR enhanced pre-existing self-reflection behavior, and self-reflection did not necessarily correlate with higher accuracy.

How does RLVR affect the diversity of solutions generated by a model?

RLVR tends to shrink the set of creative solutions that a model can provide. Models trained with RLVR showed less diversity in their problem-solving approaches, as RLVR reshapes the probability distribution of answers, making correct responses easier to hit but not introducing new reasoning paths.

What impact does RLVR have on model performance compared to base models in certain benchmarks?

In some benchmarks, such as the Manurva benchmark, models trained with RLVR performed worse than their base model counterparts, especially when the sample size increased. This indicates that RLVR may not always improve performance and may even limit the model’s ability to generate creative solutions.

What does the research on distillation suggest about improving reasoning capabilities in LLMs?

Distillation can import new reasoning capabilities into a model by transferring knowledge from one model to another. Research showed that a smaller distilled model could solve problems that the base model could not, suggesting that distillation is a promising approach for enhancing reasoning processes.

What is the significance of sparse parameter updates during RL training in models like DeepSeek R1?

During RL training, only a subset of the model’s parameters are updated, with 70% to 95% remaining unchanged. This sparse update pattern, particularly in models like DeepSeek R1, shows that RL training focuses on specific weights, allowing for effective fine-tuning without changing the entire model.

How did the 'spurious rewards' paper challenge the generalizability of RLVR across different model families?

The 'spurious rewards' paper revealed that rewarding models with incorrect, random, or pseudo labels could improve performance, but only for certain models like Quinn. For other models, such as Llama, these rewards actually degraded performance, highlighting that RLVR might not generalize well across different model families.