Building a fully local "deep researcher" with DeepSeek-R1

Summary

The video discusses Deep Seek Lab's release of Lanch Blank Chain Deep Seek R1, an open-source reasoning model. The model is trained using a combination of fine-tuning and reinforcement learning (RL), specifically leveraging GRPO for problem-solving tasks like math and coding. Key to its success are two stages of RL, one focused on reasoning and the other optimizing for both reasoning and general capabilities. The video also showcases the performance of distilled models that can run on local hardware, demonstrating their effectiveness in research and coding tasks. Despite some challenges with 'think tokens,' the model shows promising results.

Takeaways

• 😀 Deep Seek Lab's 'DeepCar1' is a new open-source reasoning model based on a combination of fine-tuning and reinforcement learning.
• 😀 The training strategy for DeepCar1 includes fine-tuning on thousands of chain-of-thought examples, followed by reinforcement learning with a novel approach called GRPO.
• 😀 GRPO reinforcement learning involves creating multiple samples (64) for each training example and scoring them using rule-based rewards, which guides the model's reasoning process.
• 😀 After the first reinforcement learning phase, DeepCar1 uses rejection sampling to filter out low-quality reasoning traces and retains 600,000 high-quality examples for further fine-tuning.
• 😀 To restore general capabilities that were weakened during the reasoning-focused phase, the model is fine-tuned on a mix of non-reasoning data (e.g., writing, QA) alongside reasoning samples.
• 😀 DeepCar1 undergoes a second round of reinforcement learning, optimizing for reasoning, helpfulness, and minimizing harm, which enhances its overall capabilities.
• 😀 The model also uses knowledge distillation to create smaller versions of DeepCar1 that can run on less powerful hardware, such as laptops, with the 14B model being a key example.
• 😀 DeepCar1's reasoning capabilities are comparable to state-of-the-art models like OpenAI's O1 series, excelling in areas like coding and math benchmarks.
• 😀 A key challenge with DeepCar1 is the presence of 'think tokens,' which appear during reasoning and can interfere with output quality, though filtering techniques can be applied.
• 😀 Running the distilled 14B model locally on devices like a MacBook Pro is possible, demonstrating the power of open-source reasoning models that can be used by individuals for free.

Q & A

What is the main difference between reasoning models and traditional chat models?

Reasoning models, like DeepSeek’s R1, focus on deliberate, logical reasoning (System 2 thinking), while traditional chat models typically rely on fast, intuitive responses (System 1 thinking). Reasoning models are trained to think step-by-step and are more suited for research and planning tasks, whereas chat models excel in interactive, conversational tasks.

How does DeepSeek's R1 model use reinforcement learning (RL) in its training process?

DeepSeek’s R1 model employs a two-stage reinforcement learning process. In the first stage, it generates multiple attempts (64 samples) for each problem and scores them based on correctness. In the second stage, the model is fine-tuned using high-quality reasoning traces and additional non-reasoning examples to balance both reasoning strength and general capabilities.

What role does Chain of Thought (CoT) reasoning play in training DeepSeek’s model?

Chain of Thought reasoning is central to the R1 model’s training. It involves breaking down complex problems into smaller, logical steps, which the model uses to generate solutions. The model is fine-tuned using thousands of CoT examples, enabling it to reason through problems systematically.

What is the issue with 'think tokens' in the R1 model’s output?

The 'think tokens' are a part of the R1 model’s internal reasoning process, but they are emitted in the model’s output. These tokens can clutter the response, making it harder to use the model in applications. While they provide transparency into the model’s reasoning, developers need to remove or filter them for cleaner outputs.

How does the R1 model compare to OpenAI's GPT-4 models in terms of performance?

The R1 model from DeepSeek performs comparably to OpenAI’s GPT-4 in tasks related to math and coding. In certain benchmarks, such as S-bench for software engineering challenges, R1 even slightly outperforms GPT-4, demonstrating strong reasoning capabilities in these areas.

What makes DeepSeek's 14B distilled model noteworthy?

The 14B distilled model from DeepSeek is remarkable because it maintains high performance similar to larger models, like GPT-4 mini, but can run on consumer hardware, such as a 32GB MacBook Pro. This makes high-performance reasoning accessible to a broader range of users, even on laptops.

Why does DeepSeek use reinforcement learning (RL) with multiple samples per problem?

DeepSeek uses multiple samples (64 attempts) per problem to better capture the diverse reasoning paths the model can take. This approach helps the model learn from different strategies and fine-tunes its decision-making process, ensuring it can solve problems more robustly and efficiently.

What is the benefit of combining non-reasoning examples with reasoning traces in the second fine-tuning stage?

Combining non-reasoning examples (e.g., writing and factual QA tasks) with reasoning traces in the second fine-tuning stage helps restore the model’s general capabilities. While reasoning traces enhance the model’s logical problem-solving skills, non-reasoning examples help it maintain broader language and knowledge abilities.

How does the R1 model handle the challenge of balancing reasoning with general capabilities?

The R1 model addresses this balance by performing a second round of fine-tuning. It first focuses on enhancing its reasoning ability through RL with math and coding problems. Then, it fine-tunes the model with general knowledge tasks, like writing and QA, to ensure the model retains a wide range of capabilities alongside its strong reasoning skills.

What is the significance of DeepSeek’s open-source approach for local AI development?

DeepSeek’s open-source approach is significant because it allows developers to run state-of-the-art reasoning models like R1 on personal hardware. This lowers the barrier for experimentation, enables better transparency in model training, and makes powerful AI tools accessible to a wider audience, without the need for expensive infrastructure.