Generative AI 101: When to use RAG vs Fine Tuning?

Summary

In this episode of 'JNA 101', the host discusses the optimal use of large language models (LLMs). They explain the cost and process of fine-tuning an LLM for specific data sets, highlighting its effectiveness in niche applications like predicting crop growth based on soil data. The host contrasts this with using LLMs out-of-the-box or through retrieval (RG), which is more cost-effective and integrates well with enterprise systems. The video concludes with advice on choosing between RG and fine-tuning based on the specific needs of the business.

Takeaways

• 💡 Fine-tuning a large language model (LLM) is costly but necessary for specific use cases where you have unique data sets.
• 📚 Fine-tuning requires a clean, bias-free dataset and investment in engineering and research efforts to adjust the model properly.
• 🌱 An example of when fine-tuning is essential is in niche industries like agriculture, where predictive models for crop growth rely on proprietary data.
• 🔍 Open-source LLMs may not have learned from niche data that is not readily available on the internet, making fine-tuning a better option for such specific cases.
• 🚀 Fine-tuning is beneficial when you have access to exclusive data that can significantly enhance the LLM's performance in a particular application.
• 🤖 Using an LLM directly is suitable for general purposes like chatting with virtual assistants but lacks integration with company databases.
• 🔗 The real efficiency in enterprises comes from integrating LLMs with enterprise knowledge, such as documents and ERP systems.
• 🔍 Retrieval (RG) is an alternative to fine-tuning that involves fetching relevant facts from enterprise systems for the LLM to generate responses.
• 💰 RG is cost-effective as it doesn't require the computational resources and data preparation needed for fine-tuning.
• ⚖️ The choice between RG and fine-tuning depends on the use case; RG is suitable for general inquiries, while fine-tuning is for specific business-related cases.
• 📧 The speaker offers one-on-one discussions for further questions, encouraging viewers to reach out via social media channels.

Q & A

What is the main topic discussed in the video script?

The main topic is the decision-making process regarding when to use a fine-tuned large language model (LLM), when to use an out-of-the-box LLM, and when to use retrieval-augmented generation (RG).

Why is fine-tuning an LLM considered costly?

Fine-tuning is costly because it involves taking a pre-trained model, preparing a clean and bias-free dataset, and investing engineering and research resources to properly adjust the model for specific tasks.

What are the specific use cases where fine-tuning an LLM is recommended?

Fine-tuning is recommended when there is niche data available only to a few companies or not readily available on the internet, and when the LLM needs to perform optimally on tasks that are highly specific to that data.

Can you provide an example of a scenario where fine-tuning an LLM is beneficial?

An example is an agriculture company with decades of data on soil, nutrients, and bacteria that can be used to predict the best crop for a given land and season, which would greatly benefit from a fine-tuned LLM.

What is the primary limitation of using an out-of-the-box LLM for enterprise use?

The primary limitation is that an out-of-the-box LLM does not integrate with a company's database and lacks the ability to make sense of enterprise-specific knowledge without additional setup.

What does RG stand for, and how does it differ from fine-tuning an LLM?

RG stands for retrieval-augmented generation. It differs from fine-tuning by not requiring the LLM to be trained on new data but instead providing it with retrieved facts to generate responses.

Why is RG considered more efficient for large enterprises?

RG is efficient for large enterprises because it allows the LLM to access and make sense of enterprise knowledge from documents, ERP systems, CRM, etc., without the need for fine-tuning.

What percentage of use cases can be solved with RG according to the speaker's experience?

According to the speaker, 80% or more of use cases can be solved with RG.

How does RG help in providing accurate responses to enterprise queries?

RG helps by retrieving relevant facts from enterprise systems and providing them to the LLM, which then uses these facts to create accurate and contextually appropriate responses.

What is the speaker's offer to viewers who have questions on this topic?

The speaker offers to discuss one-on-one with viewers who have questions, encouraging them to reach out via social media channels for further discussion.

What is the main takeaway from the script regarding the use of LLMs in business?

The main takeaway is that businesses should choose between fine-tuning, using an out-of-the-box LLM, or RG based on the specificity of their data and use cases, with RG being a cost-effective option for general use cases and fine-tuning for specific business-related use cases.

Outlines

00:00

🤖 Fine-Tuning Large Language Models: When and Why

This paragraph discusses the considerations for using a fine-tuned large language model (LLM) versus using one out of the box. It highlights the cost and effort involved in fine-tuning a pre-trained model with a specific dataset, ensuring it is free from bias, and the potential benefits for niche applications where proprietary data is available. The speaker uses the example of an agriculture company creating an AI model to predict optimal crop growth based on soil data, emphasizing that fine-tuning is necessary for specialized use cases where off-the-shelf solutions may not suffice.

05:03

🔍 Choosing Between Direct Use and Retrieval-Augmented Generation

The second paragraph explores the practical applications of using an LLM directly versus employing Retrieval-Augmented Generation (RAG). It points out that using an LLM directly is often limited to basic interactions, such as chatting with virtual assistants, and lacks integration with company databases. In contrast, RAG is presented as a more efficient solution for enterprise use cases, allowing the LLM to access and make sense of enterprise knowledge from documents and systems. The speaker suggests that for most general use cases, RAG is a cost-effective choice, while fine-tuning is reserved for specific business-related scenarios where unique data can enhance the LLM's performance. The paragraph concludes with an invitation for further discussion on social media channels for those with questions or doubts.

Keywords

💡Fine-tuned Large Language Model

A fine-tuned large language model (LLM) refers to a pre-trained AI model that has been further trained on a specific dataset to perform better for a particular task or domain. In the video's context, it is used to illustrate the process of enhancing an LLM's performance for specialized use cases by investing in data cleaning and additional engineering efforts. An example given in the script is an agriculture company using a fine-tuned model to predict the best crop for a given soil type and season.

💡Out-of-the-box LLM

An out-of-the-box LLM is a language model that is used as it is, without any further training or adjustments. It is typically available for general use and does not cater to specific business needs. The video mentions that while using an out-of-the-box LLM is less costly than fine-tuning, it may not be as effective for specialized tasks that require domain-specific knowledge.

💡Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation is a technique that combines the capabilities of a language model with a retrieval system to provide more accurate and relevant responses. In the video, RAG is presented as a cost-effective solution for enterprises, allowing the LLM to access and make sense of enterprise-specific knowledge from documents, ERP systems, and CRM without the need for fine-tuning.

💡Costly

The term 'costly' in the video refers to the financial and resource investment required for fine-tuning an LLM. It includes the costs associated with data cleaning, engineering, and research efforts. The script contrasts this with the lower costs of using an out-of-the-box LLM or RAG, which do not require such investments.

💡Data Set

A data set in the context of the video is a collection of data that is used to train or fine-tune an LLM. The script emphasizes the importance of having a clean and bias-free data set for effective fine-tuning, as exemplified by the agriculture company's use of 20-30 years of soil and crop data.

💡Bias

Bias, in the context of AI and data sets, refers to any systematic errors or prejudices that can affect the performance of a model. The video script mentions the need to clean a data set to be rid of any bias, ensuring that the fine-tuned LLM provides unbiased and fair results.

💡Integration

Integration in the video refers to the ability of an LLM to connect with a company's databases and systems, such as ERP and CRM, to access and utilize enterprise-specific knowledge. The script suggests that direct use of an LLM is limited without such integration, which is a key factor in unlocking efficiency in enterprise use cases.

💡Efficiency

Efficiency in the video is discussed in the context of how enterprises can leverage LLMs to improve their operational effectiveness. The script suggests that the real efficiency gains come from integrating LLMs with enterprise knowledge, either through fine-tuning or RAG, rather than using an LLM in isolation.

💡Niche Data

Niche data refers to specific data that is unique to a particular company or industry and not widely available on the internet. The video script uses the term to describe the type of data that would justify the investment in fine-tuning an LLM, as it provides a competitive advantage by training the model on information not accessible to others.

💡Virtual Assistants

Virtual assistants in the video are AI-powered tools that can interact with users, such as chatbots or voice assistants. The script mentions using virtual assistants like Char GPT or Bard for general use cases, highlighting their limitations in enterprise settings where integration with company-specific knowledge is required.

💡Enterprise Knowledge

Enterprise knowledge in the video refers to the collective information and insights within a company, including documents, data from ERP systems, and CRM records. The script emphasizes that the true value of using LLMs in enterprises comes from their ability to access and understand this knowledge, either through fine-tuning or RAG.

Highlights

The cost of fine-tuning a large language model (LLM) is discussed, highlighting that it is costly but less so than training an LLM from scratch.

The necessity of having a clean, bias-free dataset for fine-tuning is emphasized.

Fine-tuning is recommended for specific use cases where unique data is available.

A case study involving an agriculture company using AI to predict crop growth is presented as an example of fine-tuning.

The importance of using fine-tuning when niche data is exclusive to a few companies is explained.

The limitations of using an LLM out of the box without fine-tuning are discussed.

The use of LLMs for general virtual assistance is described, noting their limitations in enterprise integration.

The concept of using LLMs in conjunction with enterprise knowledge for efficiency is introduced.

Retrieval-augmented generation (RAG) is presented as a method to integrate LLMs with enterprise systems.

A comparison between fine-tuning and RAG is made, with fine-tuning being more suitable for specific data.

The ease of setting up and using RAG is highlighted as a cost-effective alternative to fine-tuning.

The role of RAG in providing factual information to LLMs for more accurate responses is explained.

A call to action for one-on-one discussions on the topic via social media is made.

The speaker offers to discuss specific use cases and doubts regarding the use of LLMs.

The video concludes with a thank you message and an invitation for further engagement.

Transcripts

[Music]

hello everyone and welcome back to the

next episode of jna 101 in today's

episode we're going to discuss when do

you when should you use uh a fine-tuned

large language model when should you

just use it out of the box and when

should you use uh out of the box or

finding one with

r so

finetuning itself is a little cost

costly so you take a pre-trained model

and then you you first you need to have

your data set you need to clean it up

make sure that it is uh you know rid of

any bias or anything and then you have

to use that data set and you know spend

some money and engineering and you know

research power into fine-tuning it

properly so fine tuning is of course not

as costly as training the llm in the

first go but it is still costly as

compared to just using uh llm API or

just using an open source llm directly

or even doing RG so out of the three of

them fine tuning a fine tune llm and

creating tune llm and creating right but

there are specific use cases where you

have to do the way you should consider

doing it very recently I sat down the

CIO of a large um agriculture Agri

company and uh you know they they were

actually creating a model an AI model an

ml model

uh given a patch of land the type of

soil the type of uh you know nutrients

the type of bacteria Etc present in the

in the soil as an input you could

predict what kind of crop would go grow

the best in which season of the year

right so if you if as an agriculture

company you have data of all of this for

the last 20 30 years right you can and

of course you have the resources to

clean this data up then this is a prime

example of when to do fine tuning

because if you were to do the same in

either a direct use llm or even in RG it

will not have work it will not work as

well as a fine-tune llm on this specific

data right so if there is some Niche

data in your use case that only you have

or if you like that's very limited to a

few companies in the world and that's

not available in the uh on the internet

so easily then most probably an open-

Source llm will not have learned on it

right so it makes sense for you to use

that data set to train your own or

fine-tune your own llm right and it's

worth spending that money and time and

effort to do that as well right I've

given you two examples one in the last

uh episode and one in this one and and

there are many such examples uh of you

know when to find you now let's move on

to either using an LM directly or using

R so using an llm directly is only

useful when you have to when you want to

use it like Char GB right so you can

either use CH GPT or Bard or you know

any other freely available uh you know

virtual assistants on the web or you can

even just use an open source one and

start chatting with it but it's

basically very limited to use in your

day-to-day use because it has it does

not integrate with your company's

database right so the biggest use case

and what I have seen after speaking to a

lot of cxos globally is that the real

use case or the real unlock of

efficiency in uh efficiency in large

Enterprises using jni will only happen

when jni speaks to Enterprise knowledge

right this is knowledge in documents

knowledge in Erp systems knowledge in

CRM etc etc so you have to figure out a

way to merge them and just like freely

available lmms are not going to you know

do this for you and hence for most

Enterprise use cases RG is the best

option where you can figure out you can

do retrieval as we have discussed in the

last episode last to last episode so you

can you can do retrieval on your

Enterprise systems right and you can

throw that to an llm where that llm can

make sense of the facts that you have

provided it through uh retrieval and

then create a response accordingly

without using its own worldly knowledge

right now depending on use case to use

case you have to choose between the RG

one and the fine tuning one again

fine-tuning is useful when you have some

specific data which the world does not

have which is not available on the

internet and uh you know it is it will

make the llm better in that specific use

case right you have to do fine tuning

there and for everything else most like

I think 80% plus of the use cases that

you know my friends in the CIO and the

COO Community share with me they they

can be solved with R because RG is way

cheaper because it it does not require

you to fine tune fine tuning requires

some you know bandwidth some uh you know

uh computing power and data as well R

requires nothing it is very easy to set

up and very easy to use and uh it

grounds or you know gives facts to the

llm and says that okay you have to

remain within the facts and answer the

question that you know either your sales

team member is asking or your operation

for is asking or anyone in your company

is asking so if you most of the general

personal use cases can get solved with r

and your specific business related use

cases can be solved using fine tuning

and I'd be happy to discuss with any one

of you one-on-one if you have a question

on this so do hit me up on any of my

social media channels uh if you have a

doubt and we can have a discussion on

this as well but thank you so much for

watching everybody and have a great

day