I Tested 20+ AI Models On The Same Coding Problems (Including Hard Leetcode)

Summary

In this video, the creator tests over 20 AI models' coding capabilities by solving a simple and a complex problem using a custom app. The app facilitates model selection, testing, and evaluation, providing results in a spreadsheet. The simple problem involves sorting popular numbers, while the complex one is a contest problem requiring a palindrome divisible by a number. The video demonstrates the models' performance, highlighting the importance of instruction-following in addition to coding skills, and concludes with a discussion of the results and potential applications of the testing framework.

Takeaways

• 😀 The video tests over 20 different AI models to solve coding problems using an app designed to facilitate the process.
• 🔍 The app allows for easy selection of AI providers and model versions, and automates the testing and evaluation of code solutions.
• 📝 The script describes a simple problem involving sorting numbers and a more complex 'hard lead code' problem to challenge the AI models.
• 📋 The results of each model's performance are meticulously tracked and organized in a spreadsheet for comparison.
• 🤖 Open AI models, including GPT-4 and mini versions, are tested live, with some successfully solving the problems and others failing.
• 📉 Some models, like Anthropic CL3.5, failed to solve the problems, while others like GPT-40 mini passed the tests with flying colors.
• 📝 The script highlights the importance of instruction following in AI models, noting that some open-source models struggled with this aspect.
• 🛠 The video demonstrates the setup of the testing environment, including the prompts and validation methods used to assess the AI's code.
• 🔧 The testing methodology includes generating a plan, writing unit tests, executing code, and validating outputs against expected results.
• 🔄 The video shows a process of running multiple tests, adjusting prompts, and observing the models' performance on both simple and complex tasks.
• 🔗 The testing app and its code will be made available to members of the channel, and the script encourages viewers to explore and modify it for their own use.

Q & A

What is the main purpose of the video?

The main purpose of the video is to test over 20 different AI models by solving coding problems to see which models can successfully complete the tasks.

What is the function of the app built by the presenter?

The app built by the presenter facilitates the testing of different AI models by allowing the selection of the provider, model version, and running tests to generate code and evaluation results.

What is the simple problem presented in the video for the AI models to solve?

The simple problem is to write a Python code that sorts the three most popular numbers from a survey of 10,000 people, where the responses are recorded in a file named 'random numbers.txt'.

How does the presenter validate the correctness of the AI models' outputs?

The presenter validates the outputs by checking if the unit tests pass and if the output matches the expected result, which in the simple problem case are the numbers 74, 66, and 31.

What is the 'generate plan' feature in the app?

The 'generate plan' feature creates a step-by-step plan to solve the given problem, which includes writing the code, creating unit tests, and executing the code.

What is the hard lead code problem presented in the video?

The hard lead code problem is to find the largest palindrome divisible by a given number K, which is a problem taken from a coding contest and not expected to be in the AI models' dataset.

How does the presenter handle the testing of the AI models' performance on the hard problem?

The presenter copies the full problem statement, constraints, and test cases into the app, and then evaluates the AI models' ability to solve the problem using the provided test cases.

What is the significance of the unit tests in the testing process?

The unit tests are crucial as they are used to check if the code generated by the AI models is correct and if it can solve the given problem accurately.

What issue did the presenter identify with the open-source models during testing?

The presenter found that while the open-source models could write good code, they often failed to follow instructions properly, which resulted in errors when executing the code.

What was the presenter's approach to sharing the testing setup with the audience?

The presenter offered to share the testing setup on GitHub for members of the channel, allowing them to fork or clone the project and customize it as they wish.

How does the presenter plan to continue using the testing app?

The presenter plans to continue using the testing app to evaluate new AI models as they are released and to fine-tune the prompts for better instruction following.