How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh

Summary

Emil, CTO at Reat, and his partner Haml discuss the creation of an AI agent for real estate agents. They faced challenges with the initial prototype using GPT-3.5, which was slow and error-prone. To improve, they partnered with Haml to develop a production-ready product. They emphasize the importance of an evaluation framework, including unit tests, logging, and human review, to systematically enhance AI performance. The talk highlights the journey from an MVP to a robust AI application, with a focus on aligning AI with human judgment and avoiding common pitfalls in AI evaluation.

Takeaways

• 😀 The founders of Reat, Emil and Haml, discuss their journey in building an AI agent for real estate agents and brokers.
• 🛠️ They initially used GPT-3.5 and React to create a prototype, but it was slow and error-prone, yet impressive when it worked.
• 🤝 Partnering with Haml, they aimed to develop a production-ready AI product that could handle tasks like contact management and email marketing.
• 🔍 They faced challenges in improving the AI without a clear way to measure success rates or understand failure modes.
• 💡 The solution was to create a systematic evaluation framework to guide the development and ensure consistent improvement.
• 📝 Unit tests and assertions were emphasized as foundational for evaluating AI systems, with a focus on logging results for tracking progress.
• 👨‍💻 The importance of logging traces and human review was highlighted, suggesting the use of tools like Lighthouse for logging and custom applications for data viewing and annotation.
• 🔄 They used language models to synthetically generate test cases, which helped in achieving comprehensive test coverage.
• 🔍 The eval framework provided a way to filter good cases for human review and to curate data for fine-tuning, reducing reliance on generic evaluations.
• 🚀 With the eval framework in place, they were able to rapidly increase the success rate of their AI application, demonstrating the power of systematic evaluation in AI development.
• 🎯 The talk concluded with a demonstration of a complex command executed by their AI agent, showcasing the practical benefits of a comprehensive evaluation system in real-world applications.

Q & A

What is the main product discussed in the transcript?

The main product discussed is an AI agent designed for real estate agents and brokers, which includes features like contact management, email marketing, and social marketing.

Why did the developers decide to build an AI agent for their application?

The developers decided to build an AI agent because they had a lot of APIs and data internally, which naturally led to the idea of leveraging AI to enhance their product for real estate agents.

What challenges did the developers face when creating the AI agent prototype?

The developers faced challenges such as the prototype being very slow and making mistakes all the time, even though it provided a majestic experience when it worked correctly.

What was the initial approach to improving the AI agent's performance?

The initial approach was to use prompt engineering and iterate with vibe checks to quickly go from zero to one, which is a common method for building an MVP (Minimum Viable Product).

Why did the developers partner with haml to improve the AI agent?

The developers partnered with haml to create a production-ready product, as they needed guidance on making the app reliable and effective for real-world use.

What is the significance of unit tests and assertions in the AI agent's development?

Unit tests and assertions are significant because they provide immediate feedback on the failure modes of the large language model, and they form the foundation for the evaluation system.

How did the developers handle logging and human review of the AI agent's performance?

The developers logged traces and used tools like lsmith for logging. They also built their own data viewing and annotation tools to facilitate human review and reduce friction in the evaluation process.

What role did synthetic data generation play in the development of the AI agent?

Synthetic data generation played a role in bootstrapping test cases by using an LLM to simulate real estate agent inputs, which helped in achieving good test coverage for the AI agent.

How did the developers ensure that the AI agent was making progress and improving?

The developers ensured progress by testing the evaluation system, iterating on prompt engineering, and using the evaluation system to filter out good cases and feed them into human review.

What were some of the common mistakes the developers avoided when building the AI agent's evaluation system?

The developers avoided common mistakes such as not looking at the data, focusing on tools rather than processes, using generic evals off the shelf, and relying too early on LM as a judge without aligning it with a human judge.

What were the key outcomes after implementing the evaluation system for the AI agent?

After implementing the evaluation system, the developers were able to rapidly increase the success rate of the LLM application, handle complex commands, and integrate natural language with user interface elements effectively.