OpenAI Embeddings and Vector Databases Crash Course

Summary

This video tutorial explores the concept of embeddings and vector databases, essential for AI product development. It breaks down the process into three parts: theory, application, and integration. The host demonstrates how to create embeddings using OpenAI's API and store them in a vector database for semantic searches and recommendations. Step-by-step instructions are provided for generating embeddings with Postman and storing them in SingleStore, a cloud-based database. The video also includes a JavaScript function for interacting with embeddings and a teaser for a comprehensive guide on OpenAI and GPT.

Takeaways

• 📚 Embeddings are a way to convert data like words into numerical vectors that capture patterns of relationships.
• 📈 In a vector space, words with similar uses, like 'dog' and 'puppy', are represented by vectors that are close to each other.
• 🌐 Vector databases store these embeddings and can be used for searching, clustering, recommendations, and classification based on similarity.
• 🔍 OpenAI provides a model to create embeddings but does not offer a storage solution, necessitating the use of a cloud database.
• 🛠 Postman is a GUI tool that simplifies the process of making API requests, including creating embeddings with OpenAI's API.
• 🔑 To use OpenAI's API, an API key is required for authorization, which should be kept secure and private.
• 📝 Embeddings can be created for single words, phrases, or large documents, with the latter being particularly useful for capturing complex information.
• 🗄️ SingleStore is an example of a cloud database provider that supports vector databases, allowing for real-time, distributed SQL databases.
• 📊 SQL queries can be used to create tables in a vector database to store text and corresponding embedding vectors as blobs.
• 🔎 Searching a vector database involves creating an embedding for the search term and comparing it against stored embeddings to find the most similar results.
• 💻 JavaScript and Node.js can be used to create functions that interact with embeddings, automating the process of fetching, creating, and storing them.

Q & A

What are embeddings and how do they relate to AI products?

Embeddings are arrays of numbers, also known as vectors, that represent data such as words in a way that captures patterns of relationships. They are essential for AI products as they allow for the measurement of similarity between different pieces of data, which is crucial for tasks like semantic search and natural language processing.

Can embeddings be used for images as well as text?

Yes, embeddings can be used for images too. Just like with text, images are broken down into arrays of numbers, which can then be used to find patterns of similarity, enabling features like Google's similar image search.

What is a vector database and how is it used?

A vector database is a database that stores embeddings. It can be used in various ways, including searching (where results are ranked by relevance), clustering (grouping text strings by similarity), and recommendations or classifications based on similarity to related items or labels.

How does the video guide the creation of an embedding using OpenAI's API?

The video provides a step-by-step guide on creating an embedding using OpenAI's API, starting from accessing the API documentation, setting up API requests in Postman, and making a POST request with the necessary model and input text to receive the embedding response.

Why is Postman used in the demonstration and what is its role?

Postman is used as an API platform and web app that simplifies the process of making API requests. It is used in the demonstration to create and send API requests to OpenAI for generating embeddings, due to its user-friendly interface and features that facilitate API testing.

What is the significance of the 'ada002' model used in the OpenAI embedding example?

The 'ada002' model is the specific version of OpenAI's text embedding model used in the demonstration. It is mentioned as the cheapest version, suitable for creating embeddings for the input text.

How can embeddings be used to create a long-term memory for a chatbot?

Embeddings can be used to create a long-term memory for a chatbot by storing embeddings of conversations or knowledge bases in a vector database. When a user's query comes in, the chatbot can search the database for the most similar embedding to retrieve relevant information or context.

What is SingleStore and how does it relate to storing embeddings?

SingleStore is a real-time, unified distributed SQL database provider that allows for the incorporation of vector databases. It is used in the video to create a database for storing embeddings and to perform vector searches based on similarity.

How does the video demonstrate storing embeddings in a SingleStore database?

The video demonstrates creating a table in SingleStore with columns for the original text and the embedding vector. It then shows how to insert data into this table using SQL queries, using embeddings created with OpenAI's API.

What is the process for searching a vector database for similar embeddings?

The process involves creating an embedding for the search term, and then performing a search in the database against the existing embeddings. The results are ranked by similarity, with the closest matches appearing first.

How is JavaScript used in the video to interact with embeddings?

JavaScript is used to create a function that makes a fetch request to the OpenAI API to create an embedding. The function takes text as input, sends it to the API, and then logs and returns the embedding data.

What is the purpose of the book 'Teach Me OpenAI and GPT' mentioned in the video?

The book 'Teach Me OpenAI and GPT' is a digital book that covers comprehensive information about using the OpenAI API, including fine-tuning and other advanced topics, presented in a visually interesting and engaging manner.

Outlines

00:00

🧠 Introduction to Embeddings and Vector Databases

The video script introduces the concept of embeddings and vector databases as crucial components in AI product development. It outlines a three-part approach to explain the theory, practical use, and integration of these technologies with OpenAI APIs. Embeddings are described as a transformation of data, such as words, into numerical vectors that capture relational patterns. The script provides an example of word embeddings in a 2D graph and explains how these vectors can be used for various applications like search, clustering, and recommendations. The video aims to teach viewers how to create a long-term memory for AI chatbots or perform semantic searches on large PDF databases.

05:01

🔍 Creating Embeddings with OpenAI's API

The script details the process of creating embeddings using OpenAI's API with the help of Postman, an API platform. It explains how to log in to OpenAI, navigate to the API documentation, and use the embeddings endpoint to make POST requests. The example given uses the 'ada002' model to create an embedding for the text 'hello world'. The script also discusses different types of embeddings, such as single-word and multi-word embeddings, and the potential for embedding large documents. It emphasizes the importance of storing these embeddings in a vector database and introduces the concept of using a cloud database for this purpose.

10:02

🗄️ Setting Up a Vector Database with SingleStore

The script guides the viewer through setting up a vector database using SingleStore, a real-time unified distributed SQL database. It covers creating an account, setting up a workspace, and creating a database and table within SingleStore to store the embeddings. The table creation process involves defining columns for the original text and the vector data type. The script demonstrates how to insert data into the vector database using SQL queries and Postman, showing the process of embedding a 'hello world' example and a larger document example, and how to view the stored data.

15:04

🔎 Searching Vector Databases and JavaScript Integration

The final part of the script explains how to perform searches within a vector database by creating an embedding for the search term and comparing it with the stored embeddings to find the most similar results. It also includes a demonstration of creating a JavaScript function using Node.js to interact with the OpenAI API and handle embeddings. The script provides a step-by-step guide to set up the function, make a fetch request to the OpenAI API, and process the response. Additionally, it suggests potential applications for embeddings, such as importing PDFs, websites, and other data sources into the database for searchability.

Keywords

💡Embeddings

Embeddings are a fundamental concept in the video, referring to the process of converting data, such as words, into numerical vectors that capture the relationships and patterns among the data. In the context of AI, embeddings allow for the measurement of similarity between different pieces of data. For instance, the script mentions that 'dog' and 'puppy' would be represented by vectors close to each other, indicating their semantic similarity. Embeddings are crucial for tasks like search, clustering, and recommendations.

💡Vector Databases

A vector database is a type of database designed to store and manage vector representations of data, such as embeddings. The script explains that these databases can be used for various applications, including searching for relevant results based on a query string, clustering text strings by similarity, and classifying text strings by their most similar labels. The video focuses on using vector databases for searching, which is a common use case in AI products.

💡Open AI

Open AI is mentioned in the script as a provider of AI models specifically for creating embeddings. The video demonstrates how to access Open AI's API to generate embeddings, which are then used for various applications. Open AI is significant in the video's narrative as it provides the technology to convert text into embeddings, a key step in building AI products with capabilities like long-term memory for chatbots or semantic searches.

💡APIs

APIs, or Application Programming Interfaces, are sets of rules and protocols that allow different software applications to communicate with each other. In the video, the script discusses how to use Open AI's APIs to create embeddings and how to interact with them programmatically. APIs are essential for integrating AI capabilities into various products and services.

💡Postman

Postman is an API platform and web application introduced in the script as a tool for making API requests. It is used to demonstrate how to create embeddings by sending POST requests to Open AI's API. Postman simplifies the process of interacting with APIs and is highlighted in the video as the GUI software used for creating and testing API requests.

💡Semantic Searches

Semantic searches involve finding information based on the meaning and context of the search terms, rather than just their literal keywords. The script mentions that embeddings can be used to perform semantic searches based on a large database of PDFs connected to an AI. This type of search is important for AI products that require understanding the relationships and context of data.

💡Chat GPT

Chat GPT, or Chat Generative Pre-trained Transformer, is a type of AI model that can be enhanced with embeddings to create long-term memory. The script suggests that understanding and using embeddings is essential for building AI chatbots that can remember past interactions and provide more contextually relevant responses.

💡ADA Model

The ADA model, specifically ADA002, is an embedding model provided by Open AI that the script uses as an example for creating text embeddings. The ADA model is chosen for its capabilities and cost-effectiveness, and it is used to demonstrate the process of generating embeddings from text inputs.

💡SingleStore

SingleStore is a real-time, unified distributed SQL database provider mentioned in the script as the chosen platform for storing embeddings. It is used to create a vector database where embeddings can be stored and searched. The script demonstrates setting up a SingleStore database and inserting data into it.

💡SQL

SQL, or Structured Query Language, is used in the script to interact with the SingleStore database. It is the standard language for managing and manipulating relational databases. The video shows how to use SQL to create tables, insert data, and perform searches within a vector database.

💡Node.js

Node.js is a JavaScript runtime environment mentioned in the script for creating a function to interact with embeddings. The script provides an example of using Node.js to write a function that fetches embeddings from the Open AI API, demonstrating how to programmatically work with embeddings in a JavaScript environment.

Highlights

Embeddings and vector databases are essential for building AI products.

Embeddings convert data like words into numerical vectors to measure similarity.

Vectors can represent complex relationships in a multi-dimensional space.

Images can also be turned into vectors for pattern recognition in searches.

Vector databases store embeddings and can be used for searching, clustering, and classification.

OpenAI provides an AI model to create embeddings but not a storage solution.

Postman is a GUI software for making API requests, useful for creating embeddings.

Creating an embedding involves sending a POST request with text input to the OpenAI API.

Embeddings can be created for single words, phrases, or large documents.

OpenAI's ada002 model allows for embedding up to 8,000 tokens, suitable for large documents.

SingleStore is a cloud database provider that supports vector databases.

A vector database can be queried using SQL to search for the most similar embeddings.

JavaScript and Node.js can be used to interact with embeddings and databases programmatically.

The video demonstrates creating a function in Node.js to fetch and use embeddings.

Embeddings can be used to create long-term memory for AI chatbots or perform semantic searches.

The tutorial covers integrating OpenAI's embeddings with a database for practical applications.

The speaker offers a digital book 'Teach Me OpenAI and GPT' for further learning on OpenAI's capabilities.

The video concludes with a summary of the basics and a suggestion to explore more advanced topics.

Transcripts

embeddings and Vector databases are

essential if you're building any type of

AI product in this video I'll go over

what they are and how to use them with

open Ai and their apis I'll cover this

in three parts I'll explain the theory

then the use and finally integration

after watching this video you'll be able

to create long-term memory for a chat

GPT or perform semantic searches based

on a huge database of PDFs connected

directly to an AI let's start there are

two terms I'll cover embeddings and

Vector databases they work together I'll

begin with embeddings what are

embeddings to put it simply an embedding

is data like words that have been

converted into an array of numbers known

as a vector that contains patterns of

relationships the combination of these

numbers that make up the vector act as a

multi-dimensional map to measure

similarity for a simple example let me

describe a 2d graph the words dog and

puppy are often used in similar

situations so in a word embedding they

would be represented by vectors that are

close together well this is a simple 2D

example of a single dimension in reality

the vector has hundreds of Dimensions

that cover the rich multi-dimensional

complex relationship between words

images can also be turned into vectors

too and it's how Google does similar

image searches the image sections are

broken down into arrays of numbers

allowing you to find patterns of

similarity for those with closely

resembling vectors once an embedding is

created it can be stored in a database a

database full of these is considered a

vector database and can be used in

several ways including searching where

results are ranked by relevance to a

query string or clustering where text

strings are grouped by similarity and

recommendations where items with related

text strings are recommended also

classification where text strings are

classified by their most similar label

for the purpose of this video I'm going

to just cover searching since it would

be the most commonly used there are many

practical ways to do this but open AI

has provided a great AI model to

specifically create embeddings it does

not however provide a way to store them

which we will be using a cloud database

for later in the video now let's start

making an embedding by accessing open AI

on the Google page I'm going to browse

open AI I'm going to head over to the

open AI website where I can create a new

account or log into an existing one

which is what I'm gonna do it's free if

you want to sign up I'm going to log in

using my Google credentials and I'll be

taken to a few options here between chat

gbt Dali and other apis I'm going to

head over to their API page and this

will take me to the dashboard here I

want to start off having a look at the

documentation on the embeddings which

you can find just over here and I'll

also link this in the description but

what we're going to start off with is

creating some API requests for

embeddings themselves in order to do

that we're going to head over to the API

references page here under embeddings is

all the information to create one and

it's quite simple it's as easy as doing

a single post request with some inputs

and getting a response back we could

write code to do this or do it inside of

a terminal but the easiest way is inside

of some GUI software and this is called

Postman it's an API platform and it's

also the sponsor for today's video

Postman is a piece of software and also

a web app that's entirely free that

allows you to do all sorts of API

requests it's quite easy to get running

I'm using Windows so I'm going to

download the windows version of the

application here I've launched it the

first thing I'm going to do is create a

workspace for my API queries this I'm

going to call open AI Vector database

there are a few different types of

workspaces between personal private but

I'm going to select the team one because

this allows me to share out my workspace

to other people in the future if I

wanted to for this new workspace I'm

going to open up a new tab kind of like

Chrome here I've got a few different

options but the main one here is the

type of request which can be a get post

push I'm going to do a post request and

here I'm going to get the URL that we're

going to be using for the embeddings

which is just down here and it's the

api.openai forward slash version one

forward slash embeddings URL I'm going

to paste this here in the URL section

and what's useful is that Postman is

telling me that I do need authorization

to use this endpoint for an API I also

have the instructions on how to generate

one so I'm going to head over to this

open AI API Keys page and on this page I

can select it to create a new secret key

I'm going to label this embeddings and

then I'm going to generate it make sure

you keep this key private I'm going to

copy and paste this into Postman and

I'll place it here in the API key

section be aware Postman allows any type

of authorization we're using the bearer

token which is how open AI authenticates

with all this setup we can now create

our first embedding which is the easiest

part on the open AI website under create

embeddings all the information to create

a request is available it only really

needs two things the model as well as

the input for the model in this example

we'll be using open ai's text embedding

using ada002 which is also the cheapest

version and for the input it can be any

type of text so here on Postman I'm

gonna head over to create this body

request under the body I'm going to

select to pass in some raw information

and this will be in the form of a Json

object I'll add in some curly brackets

and here I'm going to pass in the string

for the model as well as the string for

the input in true programming fashion

I'll have this input say hello world and

that's it all that's left is to send

this post request to open Ai and its API

and here's the response we've created

our first embedding quite easy and this

is also the vector for the embedding

down here and if you preview it it quite

a lot of numbers let me now show a few

different types of examples of

embeddings that could be created there

are single word embeddings these are

things like dog or cat where the

embedding is a single word and the

embedding Vector is generated when it's

sent to open AI these would be used in

situations where you might want to

perform a search you can do multi-word

embeddings too so I could for example

have a small sentence like open AI

vectors and embeddings are easy and this

creates a more nuanced embedding but for

the most part it looks the same to us

humans the strength of embeddings is

where you chunk large bits of

information together such as paragraphs

or entire sections of documents to

create an embedding that can be then

later drawn upon when you search a

database and for some useful information

Ada version 2 allows for a maximum input

of 8 000 tokens more or less which is

around 30 000 letters or characters and

to give you an idea of how much that is

a page from a contract or a legal

document can be up to about 3 000

characters on a single page this means

you could probably embed about 10 pages

in a single request to give you an idea

of what this looks like a let me copy

paste this entire page from this NDA

contract then I'm going to jump into

Postman and paste this here into the

input removing any line breaks or

additional spaces this gives you an idea

of just how much is being currently

embedded and this embed request takes

just about as much time as a single word

now that we can create embeddings we

need to store them somewhere openai

doesn't provide databases so we'll need

to create our own and a database full of

embeddings is often referred to as a

vector database I'm going to use a

provider called a single store they

provide a real-time unified distributed

SQL database which also is quite easy to

use since it's in the cloud and on top

of that they allow for you to

incorporate Vector databases straight in

there so what we'll do is set up a

database and we'll start storing our

embeddings and then also start searching

through them first you'll want to set up

an account it's free and comes with

additional credit as well as the ability

to set up unlimited databases I already

have an account so I'm just going to

sign in via Google on the main dashboard

the first thing you'll want to do is

create a workspace and in here we'll

soon create a database for your

workspace you can call it anything in

mind I'm going to call it open AI Vector

database I'm going to select the cloud

platform for AWS but you can also select

a Google or Microsoft Azure platform and

for the region I would recommend just

picking the one that's closest to you

but I'm just going to leave it to the

default on us West next I'll select the

button next on this page I can set how

fast the workspace is how many CPUs how

much RAM it has but for this tutorial

I'm not going to do much I'm just going

to keep it at the bare minimum I'll

leave the advanced settings as the

default and select next this workstation

will set up in the background and once

complete I'm going to create a database

on it let me close these info panels

here we'll have the workspace on the

left hand side and on the right side I'm

going to select create a database I'm

going to name this database open AI

database and then select to create it

once it's created it'll show up here in

the interface and I can select to view

it it doesn't have any tables or data

just now so what we'll do next is head

over to the SQL editor where we can run

some simple commands to create a table

and then input some data make sure that

on the top you have selected the

database you're currently using so in

this case I'm going to select the

database open AI database and now I'm

going to write a simple SQL query it'll

be create table if not exist and I'll

have the name of the table as my table

actually I'll do my Vector table here in

Brackets I'm then going to put in the

different types of columns I'm going to

have the first will be the text the

original text as a text type the next

will be the vector and the vector will

be a blob type you can have different

types such as integers and numbers and

decimals and you can even have a larger

table with things like an ID and other

attributes like a URL but I'm keeping

this nice and simple for this demo let's

run this command and this has created

the table here we can see a logged below

I can head into the database and select

it and view the table additionally I can

view the different columns as well as

the data type additionally I can head

over to sample.2 have a look at the rows

which don't exist yet let's create our

first row and insert a vector into this

database what I'm going to do is open up

the SQL editor in here I'm going to

write some new syntax to insert a row

this is insert into the table name which

is a my Vector table then in Brackets

text and Vector being the two attributes

we want to fill and then the values for

those which in the this case will be the

ones that we have in Postman so I'm

going to open up a postman here I'm

going to copy out of this input hello

world and I'm going to paste this here

into the values next I'll put in the

embedding here I'm going to pass in Json

underscore array underscore pack which

is necessary to turn it into this blob

structure and in here I'm going to pass

in the string of the array this is in

Postman here under embedding and it's

this huge block of numbers I'll select

to copy all of them and then here in the

interface I'll paste them back in and

that's it now I simply have to run the

command and this should enter this

embedding into the vector database which

it just has here we can see the results

one row added and if I head over to the

open AI database under my Vector table

I'll be able to see under sample data

this a new row has been added now it's

time to add some more data because we

can't really have a database with just a

single row what I'm going to do is is

head back to postman search through the

history of all the posts we've done so

far and pull out some of these examples

to place in as a rose inside of the

database so I've got this one here for

open AI vectors and embeddings are easy

I'll copy the input as well as the

embedding straight into single store wow

it always surprises me how many

dimensions there are to this embedding

now I'll enter this and insert this row

heading back to the database I'll be

able to view the sample data and here it

is let me try this one more time with a

larger example I've got this one here

which was the document which had quite a

few different characters in here and the

process is more or less the same copying

out the text then copying the embedding

vector and then finally inserting the

row into single store let me show you

the three rows that now exist that are

quite simple but enough for us to start

performing searches searching a vector

database for embeddings is actually

quite simple the first step is to

identify what you want to search for so

for example we might want to search for

anything related to open AI next we have

to create an embedding for our search

term in this case we would create an

embedding for the word open Ai and then

finally we would perform a search in the

database against the existing embeddings

this would return a list with the

closest similarity being at the top

heading over to the SQL editor I'm going

to write this query by writing the

following select text which is one of

the columns then I'm going to pass in

dot underscore product passing in the

next column the vector column and adding

in Json underscore array underscore pack

I'll leave this empty for now but this

is where I'll add the array and then

convert this to a score that we can use

for ranking results next I'll grab this

from my Vector table and I'll order it

by the score in descending order this

will be limited to just Five results now

opening up Postman I'm going to need to

create an amending to search so the

embedding term I want to use is open AI

this will create the vector here which

I'm going to copy out and then paste

back into the SQL query the SQL query

will use this as a reference and use it

as part of the scoring system so let me

run this and show you the results here's

our successful search of the vector

database against the term open Ai and

all the other vectors that are similar

to it the scores are ratings essentially

how highly ranked it is is how similar

it is to the content already in the

database I could create another search

term embedding here called hello Earth

and use the vector that I receive as a

search and have a try to see if this

changes up the results pasting this into

the query here on the vector database

I'll find that now hello world is the

top ranking result built with a much

higher score than all the others and

this is fundamentally how Vector

searching Works in this part of the

video I'm going to create an actual

function using JavaScript on node.js to

interact with embeddings first let me

create a brand new folder called openai

vectors and embeddings next I'm going to

do a fetched request to the open AI API

so here inside of the file index.js I'm

going to write in a small header here to

connect up to open AI in this header I'm

going to make the content type as

application Json and I'm also going to

pass in the authorization which will

have the key here as a barrier token

similar to what I did inside of Postman

the safe and secure way to do this is

using an environmental key but for this

example here I'm just going to pass in

my token straight away to make a life

easier for myself this of course is

something you want to keep secure and

after this video I'll be deleting this

key now Now to create the function I'm

going to create a async function and I'm

going to call this create embedding it's

only going to pass in one item which

will be the text and this is the text

that we're going to embed and I'm going

to use this to pass to an API now I'm

not going to use axios or anything like

that I'm just doing a regular fetch

command here so I'm going to pass in let

response equals await fetch I'm going to

pass in the URL for open ai's version 1

of embeddings which is just a forward

slash V1 forward slash embeddings then

I'm going to pass in a couple of

parameters this will be a post request

so I'll set the method here as a post

I'll pass in the headers that we just

defined earlier and then finally I'll

pass in the body which I'm going to Json

stringify and this is the post data

there's two items here just like before

the model itself which is the text

embedding Ada model and the second item

is the input which is the text we want

to embed next I'll listen to see if

there is a response from the server and

that response is okay and then I just

want to console log it out and return it

as part of this function here I'll write

response Json so it'll turn it into a

Json object and then I'll pull out the

data from that I'll console log out the

data and then I'll return the data this

function is now complete so what I'm

going to do is call it to test it out

here I'm going to call create embeddings

and just pass in hello world next I'll

open up the terminal and run node

index.js to run this function and just

like on Postman this embedding gets

returned quite quickly we can see it

just over here with the array here on

the right hand side and we can plug this

into a database next what can we do with

this well there's quite a few things you

can import a PDF library to read PDFs

chunk them and and then store them in a

database so that they could be retrieved

later and searched through you could do

the same for websites or you could do

the same for all sorts of things if

you've stayed until this point you

definitely want to learn more about

openai check out my new book teach me

open Ai and GPT this is a digital book

I've put on sale for 49 that covers

everything from how to use the openai

API how to do fine tuning and much more

in a really visually interesting and fun

way it's over 48 pages and it's in the

description below these were just the

basics if you want to learn more have a

look at my video here on integrating

openai a database and even a web app

together and if you haven't already this

is the end of the video so don't forget

to hit