What Skillsets Takes You To Become a Pro Generative AI Engineer #genai
Summary
TLDRIn this video, Kish Naak discusses essential skills for becoming a generative AI engineer, focusing on understanding large language models, image models, and multimodal models. He emphasizes the importance of learning both open-source and paid models, exploring frameworks like Lang Chain and Llama Index, and practicing with various projects to gain expertise in the field.
Takeaways
- 😀 The video is aimed at individuals interested in generative AI engineering and provides insights into the skills required for the field.
- 🔍 The speaker is Kish naak, a data science educator, who discusses the importance of understanding generative AI for those transitioning into the field.
- 📚 The video offers a comprehensive list of resources and playlists for learning about generative AI, including end-to-end projects and tools.
- 💡 Generative AI involves creating new content based on context, with a focus on large language models (LLMs), large image models, and multimodal models.
- 🏭 The video highlights the competition among tech giants like Google, Microsoft, and Meta to develop the best LLMs and image models.
- 🛠️ It emphasizes the importance of learning both open-source and paid models for generative AI, as well as understanding their advantages and disadvantages.
- 🔑 The speaker mentions frameworks like Lang chain and Llama Index as essential tools for developing applications using generative AI models.
- 📈 The video stresses the significance of practicing with various use cases and understanding the deployment and scalability aspects of generative AI models.
- 📝 There is a strong focus on the importance of fine-tuning models with custom data, which is considered a crucial skill in the field.
- 📚 The prerequisites for entering the field of generative AI include knowledge of Python, machine learning, deep learning, NLP, and advanced concepts like RNNs and Transformers.
- 🛑 The video concludes with a roadmap for becoming a generative AI engineer, urging viewers to learn the basics and practice with projects to improve their skills.
Q & A
What is the main focus of the video by Kish naak?
-The main focus of the video is to discuss the important skill sets required to become a generative AI engineer and to provide necessary materials and resources for learning about generative AI.
Why is Kish naak making this video?
-Kish naak is making this video because many of his students who have transitioned into the data science field are getting work in generative AI and are using large language models and large image models to solve various use cases.
What are the two main types of models discussed in the video?
-The two main types of models discussed are large language models (LLMs) and large image models, with a third type being multimodal models that combine text and images.
What is the main aim of generative AI models?
-The main aim of generative AI models is to generate new content based on any given context.
What are some of the companies mentioned in the video that are in competition to create the best LLM models?
-Some of the companies mentioned are OpenAI, Google, Microsoft, and Meta, all of which are competing to create the best large language models or large image models.
What are the two important categories of generative AI models discussed in the video?
-The two important categories are open source models and paid models, which the video suggests one should have a complete understanding of both.
What is AWS Bedrock and how does it relate to generative AI?
-AWS Bedrock is a service that provides APIs for various generative AI models, both open source and paid, allowing users to solve business use cases and perform fine-tuning without worrying about the cloud part.
What are some frameworks that one should be good at for developing applications in generative AI?
-Some frameworks mentioned are Lang chain, Llama Index, and Chainlink, which provide tools for various functionalities from data injection to transformation and the ability to call both paid and open source models.
Why is understanding vector databases important for generative AI?
-Understanding vector databases is important because they are essential for converting text into vectors, which is a key process in developing applications related to text in the generative AI field.
What is the importance of fine-tuning custom data with LLMs in the context of generative AI?
-Fine-tuning custom data with LLMs is crucial as it allows models to be adapted to specific use cases and business requirements, making it a vital skill for generative AI engineers.
What is the prerequisite knowledge required to start learning about generative AI according to the video?
-The prerequisite knowledge includes Python programming language, basics of machine learning and NLP, deep learning concepts, advanced NLP concepts like RNN, LSTM, and Transformers.
Outlines
🌟 Introduction to Generative AI Engineering
In this introductory paragraph, Kish Naak welcomes viewers to his YouTube channel and outlines the purpose of the video. He discusses the growing interest in generative AI among his students who are transitioning into data science. The focus is on the skills required to become a generative AI engineer, particularly in the context of using large language models (LLMs) and image models. Kish promises to provide necessary materials and playlists covering end-to-end projects that utilize both open-source and paid LLM models. He emphasizes the importance of understanding generative AI, the prerequisites for entering the field, and the key skill sets needed.
📚 Understanding Generative AI and Model Types
This paragraph delves into the specifics of generative AI, explaining the three main types of models: large language models (LLMs), large image models, and multimodal models. Kish highlights the importance of understanding the differences between these models and their applications. LLMs are used for text-related tasks, trained on vast datasets, and are in high demand due to competition among tech giants like Google, Microsoft, and Meta. Multimodal models, on the other hand, combine text and image capabilities, offering a broader range of applications. Kish also discusses the importance of skill sets in generative AI, focusing on both open-source and paid models.
💡 Exploring Open Source and Paid Models
Kish discusses the necessity of understanding both open source and paid models in generative AI. He compares the current state of generative AI to the machine learning boom in 2018, emphasizing the importance of hands-on practice and exploration. He mentions various companies and their models, such as Meta's Lama 2, Open AI, and AI 21 Lab, and discusses the differences between open source and paid models in terms of deployment and scalability. Kish also introduces AWS Bedrock, a service that combines various LLM models into a single API, making it easier to fine-tune and deploy models without worrying about cloud infrastructure.
🔍 Frameworks and Tools for Generative AI
In this paragraph, Kish introduces the frameworks and tools essential for generative AI development. He mentions the importance of being proficient in frameworks like Lang chain and Llama Index, which facilitate the integration of both open source and paid LLM models. Kish also discusses the use of Hugging Face for accessing various models and the role of cloud platforms like AWS and Azure in deploying AI services. He stresses the importance of understanding vector databases like ChromaDB and Cassandra for working with text data. Additionally, Kish advises on the practical aspects of creating projects using LLM models and the significance of fine-tuning models with custom data.
🚀 Prerequisites and Roadmap for Generative AI
Kish concludes the video by outlining the prerequisites for entering the field of generative AI. He provides a roadmap that includes learning Python, basics of machine learning and natural language processing (NLP), deep learning concepts, advanced NLP concepts, and transformers. He encourages viewers to go through the provided materials and playlists to build a strong foundation in generative AI. Kish emphasizes the importance of practice and project development to become proficient in the field.
Mindmap
Keywords
💡Generative AI
💡Data Science
💡Large Language Models (LLMs)
💡Large Image Models
💡Multimodel
💡Open Source
💡Paid Models
💡Fine-tuning
💡Frameworks
💡Vector Databases
💡AWS Bedrock
Highlights
Introduction to the importance of skill sets for becoming a generative AI engineer.
Discussion on the increasing demand for generative AI in data science and solving business use cases.
Emphasis on the necessity of understanding generative AI for those interested in working in this field.
Announcement of providing necessary materials and a playlist for learning generative AI.
Explanation of generative AI engineering and its prerequisites.
Description of the three types of generative AI models: large language models, large image models, and multimodal models.
Differentiation between large language models and their applications in text-related use cases.
Highlighting the competition among tech giants in developing accurate large language and image models.
Introduction to the concept of multimodal models that combine text and image for solving complex use cases.
Importance of understanding the main aim of generative AI models to generate new content based on context.
Categorization of generative AI into open source and paid models and the need to learn both.
Discussion on the use of open source models like Lama 2 and their potential for business use cases.
Mention of paid models and services like Open AI, Cloudy, and AI 21 lab for advanced functionalities.
Exploration of the role of cloud platforms in deploying and scaling generative AI models.
Introduction to AWS Bedrock and its comprehensive service for integrating various LLM models.
Recommendation to explore frameworks like Lang chain and Llama Index for developing generative AI applications.
Emphasis on the importance of practicing with different models and understanding their limitations.
Highlighting the significance of fine-tuning models with custom data for specific business needs.
Provision of a roadmap and resources for learning prerequisites in generative AI engineering.
Transcripts
hello all my name is Kish naak and
welcome to my YouTube channel so guys in
this particular video we are going to
discuss about all the important skill
sets that you may specifically require
in order to become a generative AI
engineer the reason why I'm making this
specific video right now most of my
students who have already transitioned
into the data science field who are
working as a data scientist they're
getting a lot of work to work in the
field of generative AI specifically with
respect to solving various use cases
with the help of large language models
and large image models so if you are
really interested in understanding about
generative AI work and you really want
to work in this field then this video
will definitely be for you and the most
important thing of this specific video
will be that I will be providing you all
the necessary materials and all the
entire playlist where I've created a lot
of videos and probably in the upcoming
one or couple of months you'll be seeing
a lot of end to end projects that will
be coming up uh I will try to use lot of
tools specifically that you actually
require to solve this business use cases
considering both open source llm models
and paid llm models and I'm already
doing that anyhow in the description
link I'll be providing you with all the
materials and all the playlist link now
let me quickly go ahead and start
explaining about generative AI
engineering and what are the skill sets
that are basically required whenever I
talk about generative AI engineering
right so I really want to talk about two
important thing one is the prerequisites
okay like if you really want to get
enter into this field what are the
prerequisites the second thing is that
what are the important skill sets Okay
so this entire video I will be talking
about this two important thing with
respect to prerequisite let me just
explain about the prerequisite in some
time okay so I will take this up in some
time but before this let me talk with
respect to skill sets Okay now and here
I'm just going to focus on generative AI
as I said prerequisite what are the
necessary things that you really need to
know so that in the interviews whatever
things are basically asked for machine
learning deep learning NLP that will be
part covered in entirely in this
prerequisite itself when I talk about
skill sets this is specifically related
to generative AI okay generative AI so
with respect to skill sets if I consider
this
okay understand one one one thing right
what exactly generative AI is generative
AI basically
means generative AI I basically means
here you are trying to work with those
kind of models right and specifically
when I talk with respect to generative
AI there are two types of models that
you will probably see right right now it
is three one is llm model large language
model the second one is large image
model and the third one is something
called as
multimodel okay multimodel if I probably
see with respect to categorization these
are the three important types with
respect to generative AI models okay so
one is llm large language model large
image model and multi model let me talk
about the differences between them large
language model basically you
specifically work with use cases that
are related to text and these models are
trained with huge amount of data okay
that usually trained with huge huge
amount of data nowadays there is a
fierce competition between big these big
giants who are able to create good large
language models or large image models
with the better accuracy so if you
probably see openi if you probably see
Google if you probably see Microsoft if
you probably see uh meta all these
companies are in a fierce competition to
probably make the best llm models or L
models right and over there they will be
using specific data sets at the end of
the day they are everybody's in
competition because once they achieve
that
accuracy and uh all the people will try
to use those specific models to solve
the business use cases if I talk about
multimodel it is a combination of text
and image so that basically mean that
model will be able to solve use cases
that involves both text and images like
example if I probably consider Google G
Pro and right now it is equivalent to
chat gp4 okay now our main aim is that
in generative AI main aim of the model
is to generate new content right based
on any context it'll be able to generate
new content so this is the main aim with
respect to generative AI itself now with
respect to skill sets now this is what
already have been probably researching
from 6 to 7 months I mean asking people
who are specifically working in this
specific field what kind of use cases
they are solving how they are solving
everything as such so with respect to
skill sets if I consider right and right
now this entire generative AI is
basically divided into two important
things okay one is Open
Source One is paid paid llm model
specifically if I llm or I can basically
say paid models Okay the reason why I'm
writing this because I'm going to
categorize guys based on this two
important information open source and
paid
models do you need to learn both of them
yes the answer is very simple yes you
really need to have the complete idea
I'll tell you right now the generative
AI field is somewhat like how in 2018
machine learning was right people were
exploring people were getting to
understand new things right how they can
actually Implement things how can they
can actually use this models to do the
deployment or solve complex problems how
we can actually make it scalable all
those things and right now platforms are
also coming so still you know companies
are using it they're seeing they're
seeing that how we can actually solve
the specific use cases so my always the
main main main thing that I really want
to talk about is that practice as much
as you can try to solve many use cases
as you can right tomorrow if you're
specifically working in a company if any
use cases come you really need to try
each and everything open source paid
everything right then you need to
understand that what are are the
downfalls what are the disadvantages
what are uh probably with respect to
open source and paid models which is the
best thing to probably go ahead with
right and automatically you'll be able
to understand okay this is the problem
that our company is facing this is what
is the perfect way to probably go ahead
with so that is the reason I've divided
this into two thing one is open source
llm models or L models paid models okay
and if I talk about open source
specifically you'll be hearing some
amazing companies like meta right uh
they have actually come with this Lama 2
right if I probably come up with this
Lama 2 Lama 2 model right now its
accuracy is good in some days probably L
3 will also come right uh not only Lama
2 like let's say consider paid one right
if I probably consider with respect to
paid one then you have open AI you have
cloudy right cloudy 2 you have one more
model is the a21 lab right sorry AI I
think it is a AI 21 lab right these are
they are also providing lot of
functionalities uh if I probably
consider one more uh Mistral right
mistol is also given as an open source
also and there is also a paid version of
it okay I guess both the combination is
there um recently I've started exploring
mrr also uh there's more open source
models like Falcon
right right now these all llm models if
I if I probably consider llm or L models
whatever things is there or here one
more example is Google gmany pro right
Google gy Pro right right now Google gy
pro is also given I'll not say it is an
open source it is also paid API right
now but it is also given in the free
version so with respect to Google ji Pro
here you'll be able to see that you'll
be able to use this and all right now
what is the differences between this
open source and paid models right now
you can use this entire open source
models to solve various business use
cases obviously you can use this to
solve business use cases you can also
use you can also use the paid models to
solve business use cases understand this
thing right but when we think in terms
of
deployment right we think about
scalability right so obviously we really
need to look at the cloud part when we
are specifically using this open source
model itself obviously the accuracy is
high over here accuracy is less that is
another thing but when we say that we
are taking this into the production
level at that point of time you'll be
seeing that we really need to be
dependent on the cloud right so one
thing that matters is with respect to
the cloud and this is also one of the
reason what business basically thinks
whether we should probably go with open
source and PR do we need to handle the
cloud right let's say I have a product I
want to use any of this models
functionalities right if I probably go
with open source I may have to use this
I need to finetune our data set and then
probably do the deployment right so the
cloud is one of the important factor in
the case of paid models they have their
own specific Cloud right personal cloud
let's say cloud A2 has some specific
Cloud open AI has their own apis itself
everything is basically provided with
respect to this where you can use these
apis along with to solve any business
use cases now this is one way now there
is also another way we can also be
dependent on
AWS or other Cloud platform like aure
let's say in AWS we have something
called as there is a service which is
called as AWS
Bedrock now AWS Bedrock I'll soon make a
tutorial a detailed tutorial I've
already started working on this and it's
superb guys it's superb right AWS
Bedrock what it does I I'll talk about
it okay now AWS Bedrock this is a
service what it does is that it has
almost each and every llm model stable
diffusion model LM model so one example
is all stable diffusion
right stable diffusion so what it does
is that this is basically specifically
used for image image for any use cases
that are related to image stable
diffusion okay now the best thing about
AWS Bedrock is that it has all the
functionalities with respect to this
open source or paid models it has each
and everything right I think it does not
have open AI other than that I think it
has almost everything right and this
actually
provides apis itself where you can solve
your business use cases you can perform
fine tuning right you can perform fine
tuning you can do each and everything
right probably use this specific API
directly into the clo so you don't have
to worry about the cloud part in this
right here also you don't have to worry
about the cloud part but right now let's
say if you want to use open AI then you
have a different API you if you want to
use Cloud 2 then you have to have a
different API right but what AWS Bedrock
has basically done is that it has
combined each and everything that is
probably available over here right so
when it is combining each and everything
over here in the form of apis that
basically means we can use this apis
directly over and fine tune do whatever
task we specifically require there uh
again when I probably show you the
detailed tutorial regarding AWS bedro
I'll show you how you can probably get
the API details how you have to probably
give the prompt and all what format it
is basically required everything will be
understood now what is super important
to understand you really need to have an
idea how to use this both open source
and paid models right one very important
way like one way is that you can go
ahead with AWS Bedrock but my suggestion
would be that try to explore different
different ways one way is specifically
by using hugging
face now using hugging face
you can call all these open source
models or paid models however you want
open source models right or other models
except openi you cannot call openi
because openi has a different way of
allog together doing them but when I
probably talk about Cloud 2 Mistral
right even Google gin Pro you cannot
right uh if you want to probably call
Lama 2 it has almost each and every
functionalities and what it does is that
it gives you set of libraries where you
can specifically apply embeddings also
uh you can probably use this all models
to probably solve any business use cases
not only that it also provides you Cloud
platform it also provides you space with
respect to Cloud where you can also find
tunate and here you probably have to go
pay as you go right pay as you go so the
more you use the services that much you
basically need to pay just calling this
particular thing models it is I think
almost similar like how we do with
respect to open source model so in short
the skill sets that you really need to
focus is open source two pay models paid
models you should know about the Su of
the information with respect to aw's
Cloud platform right Azure when I say
probably Azure over here you should know
that how you can use this AI services in
Azure itself right in Azure I think they
have functionalities that are related to
open and all okay since it is already
Microsoft and open AI are almost right
because Microsoft is investor in open AI
itself right
now very important thing that I have
actually mentioned over here along with
this I will talk about framework now
which Frameworks you really need to be
good at right till now guys one is open
AI framework you definitely need to know
how you can specifically use open a now
the main thing about the framework is
which I really want to mention is about
Lang chain and llama Index
right llama Index this two framework you
really need to be good at because this
two framework how it is basically
created if you really want to develop a
rag application I suggest always go with
ram andex if you want to create a
generic application and you want to
solve some problem statement you can use
Lang chain framework now what is so
amazing thing about this particular
framework because it provides you lot of
agent tools to perform different
different functionalities from data
injection to data transformation each
and everything the specific libraries
provides so because of this libraries
you'll be able to implement function uh
in a very much easy way and with Lang
chain and Lama index you can call both
paid and open source llm models that is
the best thing when I say llm models
please also consider L models or any
other models that I'm basically talking
about right both these Frameworks
provide you options to specifically call
this particular models itself so this is
also really good you should definitely
use Lang chain and Lama index one
more is nothing but chain late you
should also start exploring chain late
because this is also a framework that is
there now for practice sake you can use
framework like streamlit for the front
end right for the front end because this
will actually help you to create the UI
quickly and practice with respect to
this so all these things you
specifically need to learn with respect
to the skill sets now one very important
thing along with this you also need to
have knowledge with respect to Vector
databases there are lot of different
different Vector databases like chroma
DB right there is cassendra you can SP
specifically use cassendra you can use
data Stacks right to just get an
experience with working on this in the
production environment right and you
should try to see that how the specific
Vector databases work right once you're
able to understand the specific Vector
databases then understand developing any
application that will be related to text
you know you'll be able to convert those
text into some vectors and how you can
basically convert it they are different
different libraries which already
hugging face open AI already provide you
right so you can use this all
functionalities and important things
Vector databases is super important in
this specific framework now with respect
to task right Projects please try to
understand how to create projects how to
create projects right
using llm
models use Vector databases like pine
cone use techniques or Frameworks like
Lang chain right Lama
index Lama index see how you can use
specific agent tool see how you can use
databases see how you can actually do
the deployments each and everything
right so you need to create many
projects as possible once you create
many projects as possible tomorrow
whenever you go in any specific industry
you will have multiple option to
implement all these things right so this
is super important and one of the very
important task very very important task
and they may specifically ask you in the
interview is regarding finetuning of
your data fine tuning with your custom
data custom data using
llms this may be the very important
thing trust me I've written four star
over here but this is the most important
thing is nothing but fine-tuning with
your custom data
and you should know each and every
techniques over here let's say if you're
specifically using open AI models you
should know that if you're using llama
to you should know that if you're using
any other services you should know that
right through that way tomorrow whenever
you have a use cases where you really
need to work with fine tuning then this
will definitely help a lot okay so in
short I've given you the entire skill
sets and this is what I have done I've
created three amazing playlist one is
with respect to open AI one is with
respect to Lang chain I've covered
topics with respect to open AI Lang
chain and solve multiple use cases using
um different different models like
gemin um then you also have llama 2
right I have actually covered all these
things right so all those playlist will
be given in the description of the
specific video right go ahead and enjoy
that video go ahead and check it out if
you're serious about it definitely every
everything is available I've have
provided that in the playlist all you
need to do is that go ahead and learn
things now let's go with respect to the
prerequisite this prerequisite I've
given the entire things you need to
learn before probably starting these all
things right if you already know that it
is very good so here I'll be showing a
road map which I have already shared
earlier in my videos road map to become
a generative AI so let's go ahead and
see the road map what all things you
should need to learn so guys just a
month back I had already created this
entire road map to become generative so
these are the prerequisites that you
really need to focus on and again all
the videos materials is given over here
one is Python programming language then
basics of machine learning and natural
language processing y NLP this this this
is there basics of deep learning right
how you should basically know that hown
works forward propagation backward
propagation optimizers activation
function Advanced NLP Concepts like RNN
lstm RNN GN by Dire hmrn and all so
these are some of the prerequisites you
can probably see Transformers is also
written over here all the video links
are basically given over here you can go
ahead and check it out so once you
probably go through this and then you
start developing more projects as
possible as you can the more you
practice the more scenarios you
definitely see the more you better you
become in the generative AI Feld so yes
this was it for my side I hope you like
this particular video I'll see you all
in the next video have a great day thank
you one all take care bye-bye
Weitere ähnliche Videos ansehen
Generative AI Project Lifecycle-GENAI On Cloud
How I'd Learn AI in 2024 (If I Could Start Over) | Machine Learning Roadmap
Introducing Llama 3.1: Meta's most capable models to date
Roadmap to Learn Generative AI(LLM's) In 2024 With Free Videos And Materials- Krish Naik
Fresh And Updated Langchain Series- Understanding Langchain Ecosystem
Generative AI For Developers | Generative AI Series
5.0 / 5 (0 votes)