Claude vs GPT vs o1: Which AI is best at programming? | Cursor Team and Lex Fridman

Summary

The discussion centers around comparing different large language models (LLMs) for coding, such as GPT and Claude, with no clear winner across all categories. While models excel in different aspects like speed, reasoning, and handling complex code, Sonet is praised for maintaining a consistent performance even in real coding scenarios, which can be messier and less structured than benchmarks. The conversation highlights how coding is context-dependent, requiring models to adapt to vague or incomplete human instructions, while also touching on the role of prompt design, human feedback, and the challenges in model evaluation and performance benchmarking.

Takeaways

• 🤖 There's no single model that dominates all aspects of coding, with different models excelling in different areas such as speed, code editing, and long context processing.
• 🔍 Sonet is currently seen as the best overall model for programming, especially for harder problems like those found in coding interviews, but it may lack the ability to interpret rough human intent.
• 📊 Programming benchmarks often focus on well-specified problems, whereas real-world coding is messier and more context-dependent, requiring models to understand vague or incomplete instructions.
• 🧑‍💻 Real-world programming involves dealing with unclear instructions, broken code, and a lot of context-dependent decision-making, unlike standardized coding problems found in benchmarks.
• 🧠 Public benchmarks may be contaminated, meaning models might hallucinate correct answers based on trained data, not necessarily because they understood the problem fully.
• 👨‍💼 Some companies rely on human feedback, not just benchmarks, to evaluate model performance, with internal evaluations based on how models perform in real, messy coding environments.
• 🎯 While benchmarks are useful, real-world evaluation requires qualitative assessments and 'vibe checks' from humans to get a more holistic understanding of a model's coding capabilities.
• ⚙️ Prompt design and context windows are crucial when dealing with models. A well-structured prompt helps models perform better, but larger context windows can slow down performance or cause confusion.
• 🛠️ Some systems, like 'preum', help optimize the way code is rendered for models, dynamically prioritizing the most important parts of code when the context window is limited.
• 📂 Suggestions for file inclusion or clarifications during prompt writing can help resolve ambiguity and improve model accuracy, especially when coding across multiple files.

Q & A

What factors are considered when evaluating which LLM is better at coding?

The evaluation considers factors like speed, ability to edit code, capacity to process large amounts of code, context length, and overall coding capabilities. Each model may excel in different areas, making it difficult to declare a single best model.

Which LLM is currently considered the best overall for coding, and why?

Sonet is considered the best overall because it handles both benchmark-style problems and real-world coding tasks consistently well. It maintains strong performance even outside of standardized test scenarios, understanding rough human intent better than some other models.

Why are benchmarks not fully representative of real coding tasks?

Benchmarks are often well-specified and focus on performance within specific parameters, while real coding tasks involve vague instructions, context dependence, and incomplete specifications. This gap makes benchmarks less effective at evaluating how well models handle real-world programming.

How do public benchmarks sometimes fail to offer accurate evaluation of models?

Public benchmarks can be 'contaminated' by training data, meaning models may perform well because they’ve been trained on similar problems. Models might hallucinate correct answers based on memorized data, rather than accurately reasoning through the problem.

What role does human feedback play in evaluating the performance of LLMs in coding tasks?

Human feedback is critical in evaluating models. Internal teams often provide qualitative assessments, testing the model in real scenarios to see if it performs well across a range of problems, rather than just relying on benchmark scores.

What is the 'vibe check' approach mentioned in the script?

The 'vibe check' refers to humans qualitatively assessing how well models perform on tasks, offering feedback based on practical use rather than formal evaluations. It reflects how well the model feels to users in real-world applications.

Why might users feel that LLMs like GPT or Claude are 'getting dumber'?

Users might feel this way due to changes in the underlying infrastructure, like using different chips or quantized versions of models, which can affect performance. Additionally, bugs or issues in the deployment environment may influence their perception.

What is the role of prompt design in improving LLM performance?

Prompt design helps structure the information given to the model, improving its ability to understand user intent. For example, some systems use a declarative approach similar to JSX in web design, where elements are prioritized and formatted based on importance and available space.

What is the 'preum' system mentioned in the script, and how does it help with prompts?

The 'preum' system organizes and renders the data in the prompt efficiently. It was initially designed for smaller context windows but is still useful in managing and splitting the data that goes into the prompt, ensuring clarity even with large inputs.

How does the suggestion system improve the coding process when using LLMs?

The suggestion system anticipates uncertainties in the prompt and suggests files or code blocks that could be relevant to the task. This reduces ambiguity and improves the model's ability to handle larger, interconnected codebases.