Conversation w/ Victoria Albrecht (Springbok.ai) - How To Build Your Own Internal ChatGPT

Summary

Victoria Albrecht, co-founder of Springbok AI, discusses the complexities and considerations of building and utilizing large language models (LLMs) in business. She emphasizes the high costs and expertise required for developing proprietary LLMs, suggesting that for most companies, fine-tuning existing models or using prompt architecture may be more practical. Albrecht outlines a framework for decision-making regarding LLM integration, highlighting the importance of aligning AI strategy with business goals rather than blindly following tech giants.

Takeaways

• 👋 Victoria Albrecht, co-founder and managing director of Springbok AI, leads a team of engineers and a commercial team on various projects focusing on AI development.
• 🌟 Victoria's previous experience includes scaling a food tech business and working with Rasa, a conversational AI framework, which aligns with her session's focus on building internal chat GPTs.
• 🤖 The session aims to provide insights on whether building your own large language model (LLM) is the right approach for a company or if there are alternative solutions.
• 🚀 Building an LLM is resource-intensive, with costs reaching into billions of dollars and requiring significant expertise and time investment, making it a strategy suitable primarily for tech giants aiming for global domination.
• 🔮 A common misconception is that building an LLM will automatically lead to a 'money printing machine,' but the reality is much more complex and costly.
• 🛠 For companies not seeking global dominance or lacking the resources to build an LLM, fine-tuning an existing LLM can be a viable option, provided they have a substantial dataset and the need for on-premise hosting.
• 🔒 Fine-tuning an LLM can offer domain-specific information, data security, and compliance, but it also comes with challenges such as the black-box nature of the model and the high costs associated with retraining.
• 💼 Many companies express a desire for a bespoke AI solution to automate and streamline processes, improve decision-making, and maintain data security, but often they do not require building or fine-tuning an LLM to achieve this.
• 🏢 Prompt architecture is introduced as a scalable and controllable method for leveraging LLMs, allowing for high data security and low risk without the need for developing or fine-tuning an LLM.
• 🔑 Prompt architecture involves context-based text enhancement and response accuracy checks, providing a software layer for control and steerability, which can be adapted based on company needs and data.
• 🌐 Springbok AI has developed an enterprise platform utilizing prompt architecture, enabling clients to upload documents and query them effectively, showcasing the practical application of LLMs in business processes.

Q & A

Who is Victoria Albrecht and what is her role at Springbok AI?

Victoria Albrecht is the co-founder and managing director of Springbok AI, where she leads a team of 40 engineers and a commercial team on multiple projects, driving the further development of the business.

What was the topic of Victoria's session at the event?

Victoria's session was about sharing insights on how to build your own internal chat GPT, which is a conversational AI framework.

What is the significance of the story about Andreas and Vlad in Victoria's talk?

The story about Andreas and Vlad illustrates the early support and trust that helped Victoria and Springbok AI grow, highlighting the importance of networking and connections in the tech industry.

Why did Victoria share her experiences in Japan during her presentation?

Victoria shared her experiences in Japan to emphasize the global reach and recognition of AI technologies, showing that the interest in AI extends beyond tech circles and has a widespread impact.

What is the main challenge companies face when considering building their own large language model (LLM)?

The main challenge is the significant investment in resources, expertise, and time required to develop an LLM, which may not be justifiable for companies that do not aim for global domination in the tech industry.

Why did the executive of a major toy company consider developing their own LLM?

The executive considered developing their own LLM as part of their AI strategy, possibly influenced by the actions of big tech companies, without necessarily considering whether it was the most suitable path for their business.

What is the cost implication of developing a large language model like Chat GPT?

Developing a large language model like Chat GPT involves a substantial financial investment, with OpenAI alone having spent roughly two billion dollars on its development.

What is the alternative to building or fine-tuning an LLM for companies that do not have the resources or need for such extensive AI models?

The alternative is prompt architecture, which allows companies to leverage existing LLMs like Chat GPT through a software layer that provides high control, data security, and low risk.

How does prompt architecture help companies utilize LLMs for their specific needs?

Prompt architecture enables companies to input contextual information and instructions to tailor the LLM's responses to their specific requirements, enhancing control and ensuring data security.

What are some examples of use cases where companies might not need their own LLM but can benefit from prompt architecture?

Examples include sales leaders wanting to automate the generation of sales contracts, HR departments providing 24/7 access to company policies, and investors querying internal databases for startup information.

Outlines

00:00

🎤 Welcoming Victoria Albrecht to Techsylvania

Victoria Albrecht, co-founder and managing director of Springbok AI, is introduced as the next speaker at Techsylvania. With a team of 40 engineers and a commercial team, she is leading multiple projects and business development. Her past experience includes scaling a food tech business and working with Rasa, a conversational AI framework. Victoria will share insights on building an internal chat GPT, a topic relevant to many attendees. She also shares a personal story about her previous engagements at Techsylvania and her appreciation for the connections she has made there, including with Vlad and Andreas, who have supported her journey in the AI industry.

05:01

🌐 The Global Reach of AI and Building Your Own Language Model

Victoria discusses the global impact of AI, sharing anecdotes from her travels in Japan where she found that even in remote areas, people were aware of AI advancements. She then transitions into the topic of building one's own large language model (LLM), noting that while it's a popular idea, it doesn't make sense for every company. She emphasizes the massive investment and expertise required to develop an LLM, citing the example of OpenAI's investment of around two billion dollars and the complexity of training such models. Victoria also highlights the importance of considering whether building an LLM aligns with a company's goals and resources.

10:01

🤖 The Risks and Considerations of Developing a Large Language Model

In this section, Victoria outlines the risks and considerations involved in developing a large language model. She mentions the need for significant resources, expertise, and the ability to set up state-of-the-art architecture. Victoria also points out that building an LLM is a massive undertaking, with OpenAI's model having 175 billion parameters and 96 layers, which took months to develop with uncertain outcomes. She advises that unless a company is aiming for global domination or has specific needs, it may not be worth the investment and risk.

15:03

🔄 The Alternative to Building an LLM: Fine-Tuning

Victoria introduces the concept of fine-tuning an existing large language model as an alternative to building one from scratch. She explains that fine-tuning can be beneficial for companies that do not seek global domination but want to dominate their specific space. The process involves using a pre-trained LLM and training a small portion of it with the company's data. However, she also points out the challenges, such as the need for a large dataset, the black-box nature of the model, and the high costs involved.

20:06

📚 Leveraging Documents with Prompt Architecture

The speaker explores the use of prompt architecture as a solution for companies that want to leverage their existing documents without needing to build or fine-tune an LLM. This approach involves using an LLM to generate responses based on contextual information and specific instructions. Victoria explains that prompt architecture can provide high control, data security, and low risk, making it a scalable solution for software development based on LLMs. She also discusses the process of context-based text enhancement and response accuracy checks to ensure the generated responses are appropriate and accurate.

25:07

🛠️ Transforming Business with Large Language Models

In the final paragraph, Victoria summarizes the key points of her presentation and invites the audience to consider how they will use large language models to transform their businesses. She emphasizes the importance of understanding the options available, such as building an LLM, fine-tuning, or using prompt architecture, and making informed decisions based on a company's specific needs and resources. Victoria also expresses her hope to have provided clarity on the topic of large language models and invites questions from the audience.

Keywords

💡Springbok AI

Springbok AI is the company co-founded and managed by Victoria Albrecht, the speaker in the video. It is a team of engineers and a commercial team working on multiple projects, focusing on the development of AI technologies. The company's role is central to the video's theme of discussing AI strategies and the building of internal chat GPTs.

💡Conversational AI

Conversational AI refers to artificial intelligence systems that can interact with humans through natural language processing. In the video, Victoria Albrecht discusses her previous experience with 'why food' and 'Rasa', a conversational AI framework, which is directly related to the session's focus on building internal chat GPTs.

💡Large Language Model (LLM)

A Large Language Model is an AI system that can understand and generate human-like text based on vast amounts of data. The video discusses the considerations and challenges of building one's own LLM versus using existing models, which is a central topic in the presentation.

💡Fine-tuning

Fine-tuning in the context of AI refers to the process of retraining a model on a specific dataset to adapt to a particular task or domain. The video explores the option of fine-tuning an existing LLM to suit a company's specific needs instead of building a new model from scratch.

💡Data Security

Data security is a critical aspect of AI deployment, especially when dealing with sensitive or proprietary information. The video mentions the desire for a bespoke AI that keeps data secure, highlighting the importance of data protection in AI strategy.

💡On-Premise

On-premise refers to software or services that are hosted and operated within a company's own infrastructure. The video discusses the need for on-premise solutions for companies with strict data handling policies, which relates to the decision-making process for AI deployment.

💡Cloud Hosted Services

Cloud hosted services are applications or platforms delivered over the internet and hosted on remote servers. The video touches on the use of cloud services as an alternative to on-premise solutions for AI deployment, offering flexibility and scalability.

💡Prompt Architecture

Prompt architecture is a method of interacting with AI models by providing them with specific instructions or 'prompts' to guide their responses. The video presents prompt architecture as a solution for companies seeking control and high data security without building their own LLM.

💡Reinforcement Learning

Reinforcement learning is a type of machine learning where an agent learns to make decisions by performing actions in an environment to maximize a reward. The video mentions the integration of reinforcement learning from human feedback as a feature that makes certain AI models impressive.

💡Tech Slovenia

Tech Slovenia is an event where the video's speaker, Victoria Albrecht, has been invited to share her insights. It serves as a backdrop for the discussion on AI strategies and is a platform for networking and learning about the latest in technology.

💡Chat GPT

Chat GPT is a specific instance of a conversational AI developed by OpenAI. The video discusses building an 'internal chat GPT', indicating the desire to create customized AI solutions for internal company use, which is a key point in the discussion about AI deployment.

Highlights

Victoria Albrecht, co-founder and managing director of Springbok AI, discusses the development and application of AI in business strategies.

Albrecht shares her experience scaling a food tech business and her involvement with Rasa, a conversational AI framework.

The importance of building your own internal chat GPT and how it can be a challenge many businesses are grappling with.

Albrecht's story about Techsylvania and the growth of AI, highlighting the global impact and adoption of AI technologies.

The observation that executives often suggest building their own large language model (LLM), despite the complexity and cost.

The reality of developing an LLM, with OpenAI spending around two billion dollars and a decade of research.

Factors to consider when deciding to build your own LLM, such as resources, expertise, and the potential for global domination.

The challenges of fine-tuning an LLM, including the need for a large dataset and the black-box nature of the process.

The high costs and uncertain outcomes associated with fine-tuning an LLM, potentially reaching seven hundred thousand dollars.

The alternative to building or fine-tuning an LLM: prompt architecture, offering high control and data security.

Prompt architecture as the future of scalable and LLM-based software, with examples of its application in various industries.

How prompt architecture allows for full control and steerability of AI responses, enhancing accuracy and security.

The practicality of prompt architecture for companies looking to leverage their existing documents and streamline processes.

Examples of user stories that illustrate the potential of prompt architecture in sales, HR, and investment sectors.

Springbok AI's development of an Enterprise platform utilizing prompt architecture for document uploading and querying.

Albrecht's call to action for businesses to consider how they will use large language models to transform their operations.

The conclusion emphasizing the importance of understanding the options available when it comes to integrating AI into business strategies.

Transcripts

[Music]

foreign

we have our next speaker so please all

join me in welcoming Victoria Albrecht

the co-founder and managing director of

Springbok AI where she leads a team of

40 engineers and Commercial team on

multiple projects and is driving the

further development of that that

business her previous experience

includes scaling a food Tech business

why food and eating sales at rasa a

conversational AI framework which is

exactly in line with the session that

we're going to have today where she'll

share her insights and how to build your

own internal chat GPT which I think many

of us have been grappling with so I'm

sure that all of you are going to learn

a lot so please join me in giving a warm

welcome to Victoria

[Music]

thank you

while the text being figured out um I'll

tell you guys a a story about tech

Sylvania because this is not my first

time speaking at texylvania or one of

vlad's events in fact it's my third time

speaking here

um and there's a very nice man I don't

know if he's in the audience already or

not his name is Andreas and a few years

ago when Springbok was just

you know nine months old

um Andreas decided to introduce me to

Vlad and said hey I met this great

founder you should speak to her you know

she might be cool for your events

and

um you know back five five-ish years ago

AI wasn't as big as it is now and we

were one of the only three AI

consultancies in all of London

um and we had like very little

reputation to go off of as a company but

Vlad was actually one of the people the

early people who believed in in me and

believed in us and sort of took a punt

and said you know what why don't you

come come speak we'll see how it goes

um so that was the first event and then

we got introduced to flutter

um in here in in plush I'm at the

managing director we had some really

nice conversations he ended up

introducing us to uh flutter in London

and we ended up working with flutter for

many years

so you know if I guess if this is your

first time to Tech Slovenia then welcome

um I hope that you find great

connections as I've found some great

connections

um and yeah since we have a couple of

minutes to to fill just wanted to take a

moment to say thank you to Vlad and his

team for putting this on and also

everybody that's sort of volunteering

and you know behind the scenes

making a great like Philip as well who

I've met several years ago

it was uh nice to be back

yeah how are we doing

oh yeah Round of Applause for black

definitely

how are we doing on the tech

no worries it's a great reminder that

nowadays building like almost super

human AIS easier than connecting a

laptop to a screen

absolutely we could have built a whole

chapter plug-in in the meantime

yeah I was um I was traveling in Japan a

few weeks ago in March and I thought it

was so incredible like obviously you

know we're in our in our Tech bubbles

you guys in inclusion wherever you

you've come from

um and it hadn't really occurred to me

that that news had traveled as far as

Japan for Chachi BT but I was sitting in

a cooking class and this cooking teacher

just a local Kyoto guy uh you know asked

what do you do for work and I kept it

quite high level and I said I work in Ai

and he goes oh Chachi PT

I was like okay I guess it's made it

here okay and then I went really deep

into the countryside to a town called uh

kinosaki Onsen I don't know does that

that ring a bell for anyone in the room

no okay it's really a tiny tiny tiny

town in Japan and so I was um sitting in

the in the Onsen these little hot

springs uh next to this lady and again

like similar conversation you know what

do you do what brings you here

um and I said I work in Ai and she goes

Chad gbt

I just thought wow okay like really

um this is taken off news has traveled

fast and I guess the the growth Spike

the growth curve that you've seen in

terms of adoption is not just you know

in our own Tech circles um but is is

really Global so

um so those are some some really great

little reminders by how far it's

actually spread

um and I'll maybe just start with my my

presentation already

um and we can dial in in a second

um and it actually starts with a with

another little anecdote which is that um

well actually more of an observation so

um something that I found really

interesting over the past

pretty much two months specifically

um has been that when I've gone into

meetings with Executives and sort of

board level to talk about an AI strategy

and discuss you know where they want to

take the business

there was very often at least one person

in the room who suggested that it might

be a good idea to build their own large

language model

just a an idea that was floated

and while for some companies that's a

really understandable Avenue to want to

take there's really not very few for

whom that actually makes sense and

applies

um and it was when the executive of um

one of the world's biggest toy companies

a toy that you've a company whose toys

you've definitely played with in your

childhood said you know we're also

considering developing our own llm that

I thought okay this is now Beyond

um

you know just an idea this has now

become a bit of like a buzz theme if

that's if that's a word

um

and I was asking into the into the room

with the speakers uh last night we had a

small get together and I was asking them

um who who of them have heard

um you know the the idea of an llm being

built for a portfolio company of theirs

or in their own company or their own

startup building uh their own llm being

floated and I was uh pretty Amazed by

the response I got so I can't see you

all super well because the light's

pretty bright from here but um I'd love

to hear it see by a show of hands

um if you've been privy to a

conversation where somebody said let's

just build our own llm can I get a small

show of hands

all right that's probably about eight or

nine of you in the room yeah not bad not

bad at all yeah so it's it's been it's

been a big topic you know a lot of

companies have been interested in

building their own llm and it to be hon

yeah as I said to be honest it doesn't

always it doesn't always make sense

to do that

um but of course we all look up to the

likes of um Google and um meta Alibaba

you know the big tech company so we look

at what are they doing and is there

anything that they're doing that I

should be doing

um so it's easy to think you know okay

that's that's their plan

um I must imitate but of course

um the important thing to remember is

those guys are you know cross

um industry players

um their entire play

their entire play is in data right

um and they've built their their they've

built their entire

um their entire systems and their entire

product to basically sell the outcomes

of the data that they that they process

um now for them it makes sense to sort

of further grow their their Global World

World Domination by developing their own

large language model and partially

that's because I guess the the sexy

Narrative of why this is interesting is

right the idea is you build it you build

the large language model once and then

you have a money printing machine

um the reality is is that open AI

um you know has been working together

with Google in in research

for probably about 10 years just open AI

alone have spent roughly two billion

dollars on developing what is now what

you're using um chat2pt

so it's really not

how are we doing

okay I know

um so it's really not the easiest feat

to just come in and say you know I'm

gonna come and compete with you here

um but what I'm hoping to

um go through with you guys in my talk

today is just really a little bit of um

of a framework for how to think about

those conversations that inevitably will

be coming up again and again and will be

happening more and more about whether

building your own llm is the right

approach for you for your company for

your portfolio company and if it's not

building your own llm then you know what

is the alternative solution

um and uh one way to start thinking

about

um you know whether you want your own

alarm is on the one hand you know do you

want um Global and world uh domination

is that is that what you want to achieve

but the other big part of it as I

mentioned is of course the resource

aspect of it you know do you have one to

two billion to blow do you have a

timeline of two to three years that

could set you back because you really

want to you know own and grow that thing

um do you have the expertise in-house or

are you willing to partner with

universities are you willing to partner

with

um Recent research institutions are you

willing to set up your own team

internally to to run this and do you

have the ability to set up

state-of-the-art architecture

um uh for uh for a large language model

and I'm sure all of you that are product

Builders here that have seen

architecture models you know what they

look like you know quite quite cute

quite simple

quite straightforward but the thing

that's important to keep in mind is that

an actual architecture for a large

language model in open ai's case has 175

billion parameters and 96 layers that's

not something that you just whip up

that's something that you you try you

you run you rerun and you do that over

months and months and months and you

don't actually know if the outcome that

you're going to get is what you expect

so what I'm saying is that it's a huge

risk to try and build your own LM the

cost is massive and unless you want

world domination or nothing it's

probably not worth going for

um

so that's on the on the topic of

excuse me on the topic of building your

own llm

do you have a clicker

of some sort

otherwise I just click on my laptop

it's fine

those are my slides

um so this is the framework that I'm

gonna I'm gonna walk you through

um and it'll all make sense towards the

end

and here's what we were just talking

about regarding world domination

um and then we've already talked about

what it entails to build your large

language model

um yeah so on the data side the other

thing that I forgot to mention

um handy to have slides right

um is that you need to also look for uh

integrating the reinforcement learning

from Human feedback and that's actually

something that's made chat topt

um really really impressive is it's not

just training the data but it's also

trained on all the training and

reinforcement learning that's been going

on that that openai has paid

basically some Kenyan labelers for about

six months to do full time

and they just turn this around in Cycles

um

my slides are gone again just

as a heads up

no

all right

um

so where we got to it's pretty much here

which is that building your own llm is

really only worth it if you're in oil

and gas defense a financial institution

or an aspiring Tech Giant because the

other thing you want to keep in mind is

that if your solution if your large

language model has to be on premise and

you really have a ton of data to train

it on and you want to build your own

then you know it's also also worth a

consideration

um

so the other thing that might that you

might think

um in your in your consideration here of

whether you know your company or a

portfolio company is is cut out for

building your own large language model

and you get to the point place where

you're thinking about you know do we

have the resources the expertise do we

have the risk appetite to really build

our own here and build us from scratch

and your answer is no then there is

still the option to fine-tune an llm

which is what I'm going to go into next

thank you um excuse me just wondering

would it be possible to have it on there

so sorry

yeah great thank you this is probably

the most chaotic presentation I've ever

done thank you for bearing with me

um where were we

all right so the thing that that um that

turns out is that when we actually ask

companies you know why do you want your

own large language model

something that we really often hear is

actually this

we want to get a bespoke chapter BT for

a company that keeps our data secure

that is tailored to our information and

that supercharges our processes in

decision making so basically they're

saying we want to take everything that

we've learned

and then we want to apply it to

everything that we're doing in the

future to be smarter to make better

decisions to be more efficient right

that's essentially what they're saying

so the question is should you be

fine-tuning instead

um so here's why fine tuning might make

sense for you

um you're not hungry for world

domination again

um you obviously still hungry for

dominating your space no one's

questioning that

um but if you just want to be the

biggest the best Healthcare company for

example Health tech company then you

know you're probably fine

um the next question is is everything in

your company hosted on premise

and if the answer is yes your company is

super strict with everything being

hosted on premise and you have super

huge data sets available then probably

yes fine-tuning your own llm is the way

to go

so what does that look like what is the

promised land so the dream on the

marketing brochure is basically this

you skip the harpit and you fine-tune

someone's someone else's work right and

in practice if we just look at this

diagram what it would look like is

um in the case if we if we choose open

AI um open ai's model you use the

pre-trained existing llm

um which is obviously a black box but

you use theirs their hard work you

manually manually collect the data that

you have available

in your company and you just train a

small retrain a small portion of the llm

that sounds easy enough right so what

are the suppose benefits or the promise

benefits of this well

um supposedly domain specific

information

um

you have data security and compliance of

course if it's if you can then run it on

premise and approved accuracy and

relevance because the ideas that you've

trained in on your data right

well it's not quite as easy

unfortunately and I'll start with the

with the top right

namely the manual collection of the

domain specific data set what that

basically means is that you need a huge

data set in your company to be able to

train against or rather alongside

model and

what was really interesting

and I think it was the the hearing in

Congress

uh with Sam Altman and he said

you know even if I wanted to I couldn't

really explain how this um how Chachi BT

Works

um how exactly the large language model

is is built what the weights are and why

it behaves the way it behaves we're

still figuring that out they feed it a

bunch of data and then they sort of have

to see how that changes the outcome but

every time you run it you're running it

it takes you about a month

and then you need to see what that does

to the outcome but if we think about the

fact that Chachi PT is built on 45

terabytes of data

and you don't have 45 terabytes of data

but maybe only

500 megabytes of data to train it

against

and I know that doesn't sound like a lot

but just for context

um uh 45 terabytes of data is 292

million documents and that's what Chad

gbt or open AI has used to train train

their model on

so if you're not coming anywhere close

to the data set that they have like you

really don't have massive amounts of

data it'll be really hard to make a

difference and even if you're able to to

bring a huge data set because this is a

black box it's so hard to control what

actually happens so you're putting in

you're putting in a bunch of you're

spending a bunch of money you're

retraining and retraining and you don't

know why it does the things that it does

which is not that great of a proposition

right

um and yeah I mean retraining the the

llm costs a ton of money

I think you can probably expect some

pretty uncertain technical outcomes and

a spend of at least seven hundred

thousand uh seven hundred thousand

dollars

um

of course if you do need it need

something hosted on premise then this is

probably still your best option

but if fine tuning isn't the answer then

you know what is the answer

well let's go through

um actually I'm interested uh raise your

hand if you're using anything on the

cloud

raise your hand if you're using a hybrid

of on-premise and Cloud

okay I'm

I'm gonna ask actually another question

raise your hand if your company does

everything on premise

right okay so for my first two questions

all of your hands whatever should have

gone up right

um which just means that that you are

part of the the 94 who wouldn't

necessarily need to to build their own

language model unless we have a new

Microsoft or a new uh Apple in in the

room

um so okay so we go down the decision

tree no we can use cloud Hostess service

services and we can use apis

and the other question is

um

do you need a high level of control over

the model output well we just looked at

or we just explored you know the fact

that if you have a data set and you

train it together with or you you try to

fine-tune open ai's data set the guarant

the output isn't necessarily guaranteed

right

um so if the answer to that question is

yes as well and you don't have it

necessarily have a huge data set

available but you have maybe hundreds

maybe thousands maybe tens of thousands

of documents that you'd like to train it

against

then probably

um there's another solution

so again what do most companies want

when they say they want an llm well

um just for for recap most of them say

we want a bespoke to rgbt for a company

that keeps our data secure so they want

control

is tailored to information and helps us

make better decisions and be a more

efficient company and automate our

processes

um and then when we drill a little bit

deeper we find that

um roughly 80 percent of companies

actually want this which is to leverage

our existing documents to create a

process that automates or streamlines

querying them

way so

um

I'm just going to walk through a few

examples here

um and you know maybe think about as I

do which of these might sound familiar

to you

of something that you've thought of or

that would be would be helpful

to you or to a startup that you work

with

um so the first one as a sales leader

these are user stories I'm sure you guys

are all familiar with how this works as

a sales leader I want my team to be able

to automatically generate sales

contracts based on our company best

practices

so that we don't need to make them from

scratch every time and so that I don't

need to review all of my sales reps

contracts myself

right

um or as a head of HR I want our

employees to have access to rhr 24 7. to

able to ask any question based on our

company policies so that my team is

freed up to deal with more complex or

interesting HR topics for example

implementing a four-day work week

um or as an investor

and this is actually based on Jenny from

eqt I was speaking to her yesterday on

the bus and asked her to give me a use

case and she was saying I would love to

as an investor I would love to be able

to query our internal database startup

pitch decks and our notes about startups

so that I don't need to give this to my

associate to do manually

uh when we're looking for the types of

startups that we've already looked at

the types of trends that are occurring

in the industry

that we've already observed and so on

and so forth

and you won't be surprised to hear me

say that actually none of these use

cases oopsies require you to use your

own large language model

or to fine-tune one

so what does that leave us with well the

concept is called

prompt architecture

so for the 94 of you

prompt architecture will likely be the

answer

and I really see and we at Springbok

really see prompt architecture

as the future of building scalable and

llm-based software with high control

High data security and low risk

and I'll explain to you what I mean

so here's how this works

um and this is really more of a

of a framework than you know

specifically our architecture or

anything

um so the the user inputs text so for

example in this HR case

um how many days of annual leave

do I get per year

next

we have context-based text enhancement

so you have in this case three types of

contextual information that's added

further so

the user information would be based on

potentially your your employee employee

database

that tells you or that tells the system

Jessica's in a probation period please

answer all these questions with this in

mind then you have the Persona

instructions so who is this

um this this bot who's the system this

this expert system representing so here

the prompt is something along the lines

of you're an HR expert you must be

helpful polite and professional and then

we have contextual information so the

company policy so this is relying on

company internal documents company

handbooks Etc maybe your notion

um so company policy is that you're only

permitted to three days of holiday

during your probation period so those

are the contextual inputs in this

example

and then the large language model

generates a response that doesn't get

sent to the user right away no no

we first have the response accuracy

check

so we check it for uh you check it for

offensive language factual correctness

turn a voice response response length

semantic similarity

um things like that just to make sure

that it actually reflects you know the

information that you've already been

been given it

um and then your text is enhanced with a

request to amend the factual

incorrectness if there was anything that

was wrong

and that is then sent as a message to

the user the message of the user is then

added to the conversation history and

this is what the diagram looks in its

entirety

now this is really what I see as this is

a very simplified version but this is

really what I see as the state of the

art of the next generation of software

that's going to be developed based on

large language models with as an example

chat GPT

as an underlying framework

um

and what this enables what this

architecture enables is for you to have

full control using a software layer so

you can control exactly what you know in

the example that we used chat topt watch

does

uh it provides accuracy so you can

directly provide the information that

Chad gbt uses right you can upload your

own documents and you know that it's

only giving you information based on the

information that you have provided it

it's not going to somewhere in the

internet that you don't know but it's

from a trusted source which is your data

that's the important bit and the

steerability so you can give it very

clear instructions for the type of

persona it's supposed to adopt the tone

of voice you can make that different

based on different users there's like a

lot of control that you have

um and of course depending on how you're

building with it you can make that

Solution on premise or you can make that

solution hosted in the cloud right that

flexibility still very much exists

um the one thing that you do need to do

of course

um here is work with the API so in this

case the chatgpt or the gpt4 currently

3.5 and student four uh to lgbt4 API

um

so yeah that's what what

um what you'll be able to to do with

that with that type of architecture and

so when we have

um you know the the customer is saying

we want a bespoke chat to BT for the

company that does these things

um and we turned it into

um or we've we then get to the point

where they where they realized that

actually what they want is to turn and

leverage their existing documents to be

able to query them for any topic that

they're specifically interested in

um and you know like theme specific uh

channels is the way that we we kind of

think about that

um then you know that's something that

we've at Springbok we've been thinking

about it for for a really long time and

um we've actually been in the in the

large language model space for I mean we

started Spring Walk In 2017. and we

started working with large language

models about three and a half uh three

and a half years ago

um and

through just a you know a ton of uh

customer interviews a ton of customer

interactions

um we found that this just is what comes

up again and again

but this is not unique to uh you know

the way that we're thinking about

product this is the way that products in

general will be developing so I hope

that the

um the framework here that I provided is

maybe a little bit of a glimpse into the

future of what that might look like for

your guys's uh for your as a software

development as well

and so what we've done

um recently we've launched this with a

few clients now has actually set this up

into into an Enterprise platform that

our clients can use so the idea is that

you just need to be uploading your

documents and you can start start

querying it and you have a sort of all

the Enterprise Suite features

um that you would like

um cool so what I hope what I hope that

I've been able to achieve today despite

all of the technical difficulties and

thank you for bearing bearing with us is

to provide you a little bit more clarity

in this mumbo-jumbo of the large

language models that's currently

floating around as everyone's figuring

out how to Grapple with a topic

um and to let you get a little bit

clearer of an understanding of you know

how to think about it next time you're

in the room with someone and you're

deciding or they're deciding whether

it's worth you know developing your own

large language model whether it's worth

exploring fine tuning as an option and

if those two things aren't an option for

whatever the reasons

um you know on the on the diagram where

that we walked through together

um you know you have a Third Avenue to

go down to think about building your

future products

so yeah I guess my question to you is

how you will lose use large language

models to transform your business

and um yeah feel free to I mean I'm

happy to take any questions if there's

if there's time I appreciate there's

probably not that much time left but

feel free to connect with me on LinkedIn

and find me afterwards and also

um Ryan my colleague is just reminding

me of a thing that I always forget

which I hope we have one more minute for

that's um

thank you

which is to take a photo with you guys

in the background

you cool with that

awesome

okay

all right

let's get in there cheers

cool all right I think we got it

Philip thank you so much Victorious

foreign