Llama 3.1 8B vs Mistral 7B in RAG

Summary

In this technical talk, the speaker compares the performance of two AI models, Llama 3.1 (8B) and Mistol 7B, for use in Retrieval-Augmented Generation (RAG) applications. The speaker provides an informal, hands-on evaluation of both models, discussing their context window sizes, hallucination rates, and QA accuracy based on a small dataset of Wikipedia cities. The results show that while Llama 3.1 slightly outperforms Mistol 7B in speed, the two models demonstrate similar capabilities in answering questions, with Llama 3.1 exhibiting a marginal edge in some areas. The talk is positioned as an exploratory comparison rather than a conclusive analysis.

Takeaways

• 😀 The talk compares two models, Llama 3.1 and Mistral 7B, in the context of RAG (retrieval-augmented generation) applications.
• 😀 The focus of the experiment was on comparing these models' performance, including latency, hallucination rates, and QA correctness.
• 😀 Llama 3.1 (8B parameter model) was released on July 31st and is known for its large context window of 128,000 tokens.
• 😀 Mistral 7B, released in September 2023, has 7.3 billion parameters and a 4,000-token sliding context window.
• 😀 The speaker ran a simple RAG app using Wikipedia cities data, with embeddings from Hugging Face and a vector database from Facebook AI similarity search.
• 😀 The models were evaluated using toy questions, including some impossible ones to test hallucination behavior.
• 😀 The experiment showed that both Llama 3.1 and Mistral 7B had similar performance, with minor differences in speed and accuracy.
• 😀 Llama 3.1 had a slightly better latency at P50, but both models showed comparable P99 latencies around 2 seconds.
• 😀 Llama 3.1 showed a 22% hallucination rate, which was consistent with the nature of the questions asked, particularly impossible ones.
• 😀 Despite being trained with significant resources, Llama 3.1 and Mistral 7B showed very similar QA correctness, with both performing well on most questions.
• 😀 The talk emphasized that the results are based on small-scale tests, with a larger-scale experiment needed for more robust conclusions.

Q & A

What is the focus of the talk discussed in the script?

The talk primarily compares the performance of two language models: Llama 3.1 (8B) and Mistral 7B, specifically in the context of retrieval-augmented generation (RAG) applications.

What was the main goal of the experiment described in the talk?

The main goal was to compare the Llama 3.1 8B model and the Mistral 7B model for their performance in a RAG application using Wikipedia data, evaluating aspects like latency, hallucination, QA correctness, and relevance.

Why did the presenter choose the Llama 3.1 8B model for comparison with Mistral 7B?

The Llama 3.1 8B model was chosen because it has a similar size to the Mistral 7B model, making the comparison more balanced and fair.

What type of questions were asked in the experiment, and why were some of them impossible?

The questions were a mix of general knowledge and city-related inquiries, including impossible questions like 'When should I visit the Empire State Building in Houston?' because there is no Empire State Building in Houston.

How were the data and models set up for the experiment?

The data consisted of Wikipedia city articles, and the models used for the experiment were Llama 3.1 and Mistral 7B. The setup involved chunking the data, storing it in a vector database, and using Hugging Face embeddings for retrieval in the RAG framework.

What tool did the presenter use for tracing and evaluation during the experiment?

The presenter used Phoenix for tracing and evaluation, which helped monitor performance and track the results of the experiment.

How did the Llama 3.1 8B and Mistral 7B models compare in terms of hallucinations?

Both models showed similar results, with Llama 3.1 hallucinating about 22% of the time (due to impossible questions), and Mistral 7B had comparable hallucination rates.

What were the latency results for the Llama 3.1 8B model?

The latency for Llama 3.1 8B was about 1 second for the 50th percentile (P50) and 2 seconds for the 99th percentile (P99).

What framework did the presenter use for implementing the RAG system?

The presenter used the LChain framework for the RAG implementation, although they mentioned working on an example with Llama Index for future demonstrations.

What was the reason for using a custom separator during data chunking?

A custom separator was used to avoid the issue of very large chunks caused by rare separators like double new lines, which led to unusually large chunks and skewed results.