Unpopular Opinion: LLMs Didn't Add Value to AIs—They Just Made Them More Accessible

Summary

このトークは、LMS（Language Models）が技術的に新しいテーブルを持ち出しているか、それとも単に技術をよりアクセス可能にし、利用を拡大したにすぎないかという議論の中心に立脚しています。パネルは、LMSがビジネスや一般大衆の理解にどのように影響を与えたか、そしてこれらのモデルが将来どのように進化し、どのよう活用されるかについて洞察を提供します。彼らは、LMSが特定の業界でどのように使用され、その利点と課題をどのようにバランスを取るかについても議論します。

Takeaways

• 🤖 AIは、人間のように行動するコンピュータシステムの総称であり、特に機械学習がその大きな分野の中で中心的な役割を果たしています。
• 📊 LMS（Language Models）は技術的には新しいものではなく、RNNやCNNなどの既存技術に基つていますが、より高いアクセス性と広範な利用をもたらしました。
• 🚀 Transformerアーキテクチャは、より大きなテキストや文脈を処理できるようになり、自然言語処理の分野で大きな進歩を遂げました。
• 🧠 LMSの進化は技術的な進歩だけでなく、ビジネス価値の発見と適用面での革新も重要です。
• 💡 LMSは現在、単純なタスクを自動化し、生産性を向上させることができていますが、複雑な意思決定やビジネスプロセスを完全に takeover することはまだ達成されていません。
• 📈 LMSの成功事例は、医療や法律などのテキストに基づく分野で特に高いポテンシャルがあり、専門用語や規則に基づいたタスクを効率的に処理できるようになりました。
• 🔍 LMSの信頼性については、まだ完璧ではありません。実際の専門家と比較して、LMSは試験で高いスコアを獲得しても、実際の業務では不十分な場合があります。
• 🌐 LMSが将来的にシステムとの最初の接触点となる可能性があるとの視点があるが、効率的なユーザーインタラクションを実現するために、様々な方法が検討され続けています。
• 🔗 LMSの発展は、多様な言語や文化を考慮に入れた包括的なアプローチが求められ、これにより技術の普及と利用が促進されます。
• 📚 LMSの今後の進化は、応答の正確さと信頼性を向上させることで、より多くのビジネス分野で実装され、より広範に受け入れられるようになるでしょう。

Q & A

LMSが既存のAIシステムにどのような価値を提供したと思いますか？

LMSは、技術的には新しい価値を提供していないものの、よりアクセスしやすいインターフェースを提供し、既存のAIシステムをより广く利用できるようにしました。

LMSが技術的にどの分野で進化を遂げたと感じますか？

LMSは、Transformersアーキテクチャを基にしており、より大きなテキストデータセットを扱えるようになり、自然言語処理のコンテキストを大幅に改善しました。

LMSがビジネス価値を提供するための課題は何だと思いますか？

LMSがビジネス価値を提供するためには、技術的な課題だけでなく、適用層でのイノベーションも重要です。特定のビジネスニーズに合ったLMSの活用方法を見つけることが鍵となります。

LMSが医療や法務などの規制された分野でどのように役立つでしょうか？

LMSは、医療や法務などの分野で、豊富なテキストデータの処理やルールベースの分析を行い、専門家の仕事をサポートすることができます。ただし、LMSは最終的な決定を下すわけではなく、専門家の判断を補完する役割を果たします。

LMSの応用において、どのような技術的な課題があると感じますか？

LMSの応用においては、技術的な正確性と信頼性の向上、そして実際の現場での適用可能性の確認が課題となっています。また、LMSが誤った情報を提供するリスクにも注意が必要です。

LMSが自然言語処理の分野でどのような進化を見せましたか？

LMSは自然言語処理の分野で、より長いテキストや複雑な文脈を理解し、応答する能力を向上させました。これにより、より自然な人机交互が実現し、さまざまな業界で応用が可能となりました。

LMSを導入する際、どのようなビジネスのニーズを考慮すべきですか？

LMSを導入する際には、ビジネスプロセスの効率化、カスタマーサービスの改善、データ分析の支援など、企業が抱える課題やニーズに応じた適切な活用方法を検討する必要があります。

LMSが提供する自然言語処理能力を最大限に活用するためには、どのようなアプローチが効果的ですか？

LMSを効果的に活用するためには、適切なデータセットで事前学習を行い、特定のタスクや業界に関連するデータをファインチューンすることが重要です。また、ユーザーとのインタラクションを通じてLMSを徐々に改善していくことも効果的です。

LMSの開発において、今後どのような進化が期待されています？

LMSの開発において、今後期待される進化には、より高精度な自然言語理解能力の向上、多様なデータタイプとの連携能力の強化、そして倫理的な問題に対する対処などが含まれます。

LMSが誤った情報を提供するリスクがあることはどのように対処すべきですか？

LMSが誤った情報を提供するリスクに対しては、適切なデータセットで学習させ、定期的な評価とファインチューニングを行い、また、重要な決定においては人間の審査や介入を確保することが重要です。

LMSを導入する際のユーザーエクスペリエンスの改善について、どのようなポイントが重要ですか？

LMSを導入する際には、ユーザーが自然で使いやすいインターフェースを提供し、不明瞭なプロンプトや質問を最小限に抑えることが重要です。また、フィードバックメカニズムを設けることで、ユーザーの声を取り入れながらLMSを改善していくことも効果的です。

Outlines

00:00

🎤 イベントの開会とLMSの議論

イベントの開会とLMSの価値についての議論が行われています。議論の中心は、LMSが既存のAIシステムに新たな価値を提供したのか、単に技術をよりアクセスしやすくしたのかということです。また、LMSが技術的な進歩をもたらしたのか、または単にインターフェースとして機能したかについても話し及んでいます。

05:01

🌟 LMSの進化と技術的な革新

LMSの進化と技術的な革新について、3人のパネルリストが見解を共有しています。彼らはLMSが持続的な技術の進歩であると思わないという意見も示しており、LMSが現在注目されている理由は、技術的な進歩だけでなく、その技術が一般大衆にアクセス可能になったことにあると考えています。また、トランスフォーマーアーキテクチャとその前身の技術がどのように進化してきましたか、そしてLMSがどのようにスケールとコンテキストを扱う能力を向上させたかについても議論されています。

10:03

💡 LMSのビジネス価値と産業への影響

LMSがビジネスと産業に与える影響について、パネルリストが意見を述べています。LMSは企業にとって新しいビジネスケースを開き、AI技術をより使いやすくする重要な要素となりました。また、LMSは特定の業界（医療や法律など）において、ルールベースの分野でのパターン認識を改善し、効率を向上させる可能性があると指摘されています。

15:05

🚀 LMSの課題と今後の方向性

LMSの現在の課題と今後の方向性について、パネルリストが議論しています。LMSの応用が広まる中で、重要なのはモデルの正当性と信頼性の確保です。LMSは確率的なモデルであり、実際には期待通りに機能しない場合もあります。そのため、LMSの適切な訓練、フィードバック、改善が重要です。また、LMSが単に応答を生成するだけでなく、ユーザーとインタラクションを行って学び、改善する双向的なシステムとなることが期待されています。

20:07

🌐 LMSの多様性と言語の役割

LMSの多様性と言語の役割について、パネルリストが見解を共有しています。LMSは多様なモダリティ（テキスト、画像、動画など）を扱うことができ、将来的には基盤としてさらに発展する可能性があるとされています。また、言語は人間の間だけでなく、機械間のインターフェースとしても役立つとされ、LMSは自然言語でのインタラクションを容易にするでしょう。

25:08

📈 LMSの成功と信頼性のしやすさ

LMSの成功と信頼性のしやすさについて、パネルリストが意見を述べています。LMSは完璧ではないため、適切な成功率を設定することが重要です。簡単なタスクではLMSが自動化できるものの、複雑な決定には人間の参加が必要であるとされています。また、LMSは応答を生成するだけでなく、ユーザーのフィードバックを学び、改善することが望ましい双向的な学習システムとなるべきです。

Keywords

💡LMS (Language Models)

LMSは、自然言語処理技術の一種で、人間のような文章生成や理解を可能にします。このトークでは、LMSが既存のAIシステムにどのような価値を提供しているか、また技術的な進歩としてどの程度の意義を持つかが議論されています。

💡AI (Artificial Intelligence)

AIは、コンピュータシステムが人間の知的行為や行動を模倣する技術のことを指します。このトークでは、AIがどの程度進化し、LMSがその進歩の中でどのように役立っているかがテーマとなっています。

💡Transformers

Transformersは、自然言語処理における重要なアルゴリズムであり、平行処理が可能で、より大きな文脈を扱えるようになりました。このトークでは、TransformersがLMSの発展においてどのような役割を果たし、どのような技術的な進歩をもたらしたかが説明されています。

💡Fine-tuning

Fine-tuningは、機械学習において、既存の訓練済みモデルを特定のタスクに適応させる技術です。このトークでは、LMSを特定の業界やアプリケーションに適応させるためにFine-tuningがどのように使用されるかが議論されています。

💡Accessibility

Accessibilityは、技術やサービスが利用者にとって使いやすいかどうかを指す概念です。このトークでは、LMSが自然言語を通じてAIの技術をより広範なユーザーに利用可能にし、その意義が強調されています。

💡Probabilistic models

Probabilistic modelsとは、予測を行うために確率論を用いたモデルのことを指します。このトークでは、LMSがどのような確率的な予測を提供し、その限界についても議論されています。

💡Human in the loop

Human in the loopは、自動化システムにおいて人間の判断や参画が必要である場合に使用される概念です。このトークでは、LMSが特定のタスクを適切に実行するために、人間の介入が必要であることが強調されています。

💡Semantic understanding

Semantic understandingは、コンピュータが文脈や意味を理解し、適切な応答を行う能力を指します。このトークでは、LMSが語義的な理解を向上させることで、より高品質な応答を提供できることが説明されています。

💡Stochastic parrots

Stochastic parrotsは、LMSが確率的に応答を生成することを指す表現で、正確さと不確実性の問題を示唆しています。このトークでは、LMSがどのようにして確率的な予測を行うか、そしてその限界について議論されています。

💡Multimodality

Multimodalityは、複数の感覚やデータタイプ（例えば、テキスト、画像、音声）を組み合わせて情報処理を行うことを指します。このトークでは、LMSが多様なデータタイプを扱う能力が、技術的な進歩の一部として強調されています。

💡Knowledge base

Knowledge baseは、組織が蓄積した情報や知識を整理し、検索や分析のために利用できるようにしたデータベースです。このトークでは、LMSが知識ベースを構築し、効率的に活用することの重要性が強調されています。

Highlights

The panel discussion revolves around the value addition of Language Models (LMs) and their impact on Artificial Intelligence (AI) systems.

The panelists include experts from tech consulting, machine learning project management, and Google Cloud, providing diverse perspectives on LMs.

The discussion explores whether LMs have brought new technical advancements or simply increased accessibility to existing AI technologies.

Panelists share their definitions of AI, highlighting the importance of machine learning and the ability of systems to mimic human intelligence.

The evolution of AI is described as a continuum, with LMs like ChatGPT and DALL-E relying on transformer architectures and previous technological advancements.

The accessibility factor of LMs is emphasized, as they have made AI more approachable and fun for the general public.

The role of LMs in transforming user interaction with systems is discussed, with a focus on their potential as the first point of contact in various industries.

The importance of fine-tuning LMs for specific industry use cases, such as medical and legal applications, is highlighted to improve their accuracy and relevance.

The challenge of validating LMs is addressed, as their probabilistic nature makes it difficult to ensure their reliability in practical applications.

The potential of LMs to assist human experts in decision-making processes is discussed, with the emphasis on their current role as supportive tools rather than replacements.

The discussion touches on the need for LMs to understand context and handle complex queries, such as distinguishing between different types of quarters in a financial query.

The future of LMs is envisioned as part of the infrastructure, integrating with various data types and systems to form a comprehensive knowledge base.

The importance of inclusivity in LM interactions is stressed, with the need for systems to cater to diverse language abilities and literacy levels.

The potential of LMs to learn from human feedback, gradually improving their performance and ability to automate tasks, is highlighted as a promising area of development.

The debate on LMs as 'stochastic parrots' is addressed, acknowledging their probabilistic nature while also considering their potential for reasoning and understanding.

The discussion concludes with thoughts on the threshold of success rate for LM implementation and the need for a human-in-the-loop for sensitive use cases.

Transcripts

okay welcome uh to our panel the second

but last talk um on you can see the

topic on public opinion LMS didn't add

any value to our existing AIS um they

just made more accessible what we

already had and uh when I was scoping

the topic two or three months ago I

couldn't imagine that we're g to have

that many talks on LMS uh but here we

are and uh even if you have attended

former talks I'm sure you're going to

find this one uh valuable as well so I'm

chrisan I joined project a this year in

the tech Consulting

team and I'm super curious to find out

what our panelists have to say um I

already read out the topic um we're

going to talk about um if LMS brought

anything to new to the table technically

um or if they were just like acting as

an interface or a font technology and

making thereby LMS more accessible So

today we're going to have the goal of um

giving you some context to better

understand the the real technical

advancement behind behind LMS and also

to give you some food for thought um how

LMS could develop further and um how

they could be utilized in a way that

they provide real undeniable value with

me today are three lovely panelists um

we all like you all had have paneled in

different um

situations recently at the I think J

breakfast for marantic so um to start

off Hannah Hannah is um the team lead

for machine learning project management

at metics momentum marantic momentum for

those of you who don't know um

identifies use cases for II and ml um

with their customers and also then

develops and um um also operates those

um those solutions to to the use cases

then yakob you have seen yakob on stage

this morning already um yakob is a

customer engineer here at Google cloud

and uh is basically together with their

clients developing and um optimizing the

data infrastructure so um for all Cloud

native gcp clients um I would say I

would call you a an ML and II Enthusiast

and you're appearing like recently

appearing on many talks panels

hackathons um yeah and last but not

least is my dear colleague saman who's a

data scientist and engineer here project

a um together with our ventes he works

on data infrastructure machine learning

and also uh data

culture so um yeah I'm super happy to

have you here and um to kick things off

I'd like to uh get some interaction and

um ask you the audience on your opinion

um which on which side um you see

yourself also on which side on the

Spectrum so I would like to raise your

hand if you you are of the opinion that

LMS didn't add any value to our existing

a

technically is there anyone who's of the

opinion that llms didn't add any

value okay counter check if you're just

lazy raising your hand or because no one

is of the opinion who's of the opinion

that they add any that they did add any

value technically

okay so then we I think we're going to

have a interesting discussion um and to

kick things off and go right in I would

like to ask you three um to give us a

one- sentence definition of AI because

of course we know everyone defines a

differently uh and then so we can can

put your opinion um into perspective

maybe to start off with Hannah

sure no it was on before it was on

before sorry yeah happy to kick it off

so for me AI is really a collective term

for any kind of computer systems that in

some way simulate

yeah Behavior that's otherwise

identified or associated with human

behavior or intelligent behavior and I

think for me very pragmatically most

importantly is really this machine learn

topic of machine learning um as the

biggest cluster within the huge field of

AI yeah pretty I can I'm pretty much in

line with what he said but maybe to put

it into one sentence I think AI is a

collection of statistical and

mathematical

that sort of mimic human decision making

in some

sense I'm also in line with all the

other panels have said but I think I

would just go a bit broader and move

away from the statistical part of it and

just say any system that in one way or

the other um mimics aspect of human

intelligence whether that might be

Vision speech text writing that I was

ciz as as

AI Co thanks a lot so um to let's get uh

so let's get technical for a minute um

at project a we value knowledge sharing

really a lot so that's why for example

our conference tomorrow is called

project a knowledge conference and also

um at project a we have regular brown

bag lunches uh where a or colleagues can

share recent project outcomes or also

like any topic they find relevant to

them or they care of and recently saman

shared or did a brownb lunch on AI in

general and had a pretty I would say

critical view on this um beyond the bus

I would say and that's why I like to ask

you um is the Advent of LMS rather a

Continuum or a prival technical

advancement um in the ai's journey and

why or why

not um I think technically I wouldn't

say it's pivotal actually um I think if

you look at it from a technical

background current implementation of AI

which I think today we talk about these

geni aspects like chat GPT um Dar ma

Journey what else you have um rely on

this transform architecture which

basically then relies on lsdm which then

relies on RNN CNN so I think you have a

clear Evolution and and history of you

might even say incremental additions and

changes that have let to this point and

therefore arguing that now suddenly we

have these these these gen

uh applications are suddenly changing

would be a bit well exaggerating I would

say and I think if there was something

like shim who were in here he would now

say oh no I did all of this and I think

I would maybe partly agree with this as

on a technical level it's a clear

Evolution what

however makes us such a buzz in my

opinion that's why we have all of these

panels and we talk about this so much is

then um that somebody like my 67y old

dad now has a chat GPT app on his phone

and uses his because it's funny and cool

and that's essentially I think the

accessibility factor of it that previous

implementations were not able to do if

you talk to somebody about an lsdm is

they're like bro I don't know

but then you can show them hey give me a

cooking list for whatever I have in my

fridge and they can use it and they're

like oh this is insane and this

is a bit where I would sort of of argue

it's definitely not a pivotal change

technically um might be pivotal in sense

of how it's being perceived and

applied dig a bit into deeper into this

um what especially so you mentioned for

example Transformer architecture um is

this is this what you would you would

consider the technical advancement for

example that you now can um really

process it in parallel or can um get can

input like larger contents um or

contexts and it can understand it or

where do you see the technical

advancement because otherwise I would

say um there is there is nothing that

that we can really like um yeah hold on

to that that says okay it's technical

technically um I suppose and and I think

that's also so funny when we talk about

Transformers because they came out in

2017 I think was your former co-workers

paper when it was published um uh the

the main benefit I would say of

Transformers compared to say current

applications or for applications was

then you were able to um which goes into

this paration effort is you are now able

to work with much much much larger

corpuses Cory of body that were not able

to do so before right um if you take a

look at again it's lsdm which we might

have had before you were may be able to

go back as far as a sentence or two

because then you've got into these

diminishing returns and you're like oh I

don't even remember what you said and

now you know chat GPT can write you

books of content because this sort of

shortcoming has been alleviated with SC

implementation which yeah can be

attributed to a Transformer sure um

applications of it then um of what say

open AI does what BART does I think are

then more that on

scale maybe to add on to that I I

absolutely agree with what you've said

um I think what's I think what's very

much a Continuum is also when you think

about language we are always or the the

whole field of natural language

processing is kind of concerned about

capturing context and for the longest

time this was incredibly hard um as you

said like we would didn't really get

past sentences of context that were able

to be processed and then 2017 this

Transformer comes along of course also

building on Tech that was there

previously but this basically sparked

this explosion of yeah exponential

growth in what you can process in terms

of context whole whole books whole like

even libraries maybe at this stage so um

this this is really incredible and I

would also agree that the change we're

seeing now what's quite pivotal is um

definitely that we can now as an end

user also shape more how tool behaves um

without having to train it without

having to be super technical about it

but I do think this is partially also a

technical um an invention for for sure

um coming from the way these models are

pre- chained and now allow us to for

example um yeah give examples give

context provide intense and say dear

model please behave as if you were XY Z

or please write in the style of this um

and and this is quite quite pivotal

because this definitely makes some

things way easier and we can discuss

later if this is uh if this this a

Continuum or if this is really

groundbreaking yeah I will come back to

this definitely um yob why why did your

colleagues um did take so much time from

2017 until now um until they until

transform architecture really um yeah

really developed itself to to a point

where could could be utilized in a way

that that we can now really um see at

least a business value of of

course hard for me to make a statement

for for the researchers at this point

from from my perspective obviously but

yeah interesting question I think it's

really in line with the fact that in my

opinion technology usually doesn't make

pivotal changes and that's sometimes a

little bit of a tough truth because I

think us as humans we're also a little

bit wired in the sense that we love we

love the story aspect about these

individual Geniuses that make an

invention and then everything changes

and then we have suddenly this blackbox

solution that just solves all our

problems based on one genius's idea but

usually usually in technology and

research it doesn't work like that it's

rather one researcher Builds on the work

of many others we're all sort of working

on shoulders of giants in the end we

never invent everything from scratch and

that's also fine like that right that's

that really Mak the research uh culture

and the research Community right that

they collaborate and build on top of

each other's ideas and really the

pivotal changes in my opinion happen on

the application layer similar to what

someone said right suddenly there's a

break breakthrough someone um had the

idea of hey why don't we bring together

this technology with a certain business

use case where we see a large large need

this could solve this specific use case

and then these two perspectives come

together and when this B both both fits

right there's a technology that has

certain strength and then on the other

hand you have a business use case that

can be solved um and that makes sense

together and if this actually has value

then it really explodes then you have

your P to change but again that's more

rather on the application layer than on

a tech

technological in my opinion yeah so it's

it's partly a interface because it's

sticking together to Technologies and

then there would be an interface um like

the application acting as an interface

and making

front end whatever you want to call it

and making it then

accessible in a sense yeah of course of

course of course I mean in the end you

pluck together multiple Technologies and

then you try to get certain output right

yeah absolutely okay and then um to to

briefly talk about like the business

value um of llms Hannah you work closely

together with clients across Industries

um what would what would you say um the

introduction of llms um has it or like

where has it led to a significant change

in the perception but also in the

understanding um a among the masses and

B um in in business

General yeah so I think and I I would

expect we all agree with that and you

already mentioned that what what's

really this amazing Trend we are seeing

is that um everyone is getting hands on

and trying to work with with these tools

so I don't really know anyone who hasn't

at least tried to generate some greeting

card text or something like that with

llms which is amazing to see but to to

talk about industry I think maybe two

interesting things happen so first of

all we can we can feel the formo uh so

if there was anyone before not not

thinking about okay how can I use these

these how can I use AI essentially to

accelerate my bris business is now for

sure thinking about it at least if not

acting on it and and secondly um of

course it also unlocks certain use cases

or certain thoughts and I would also say

of course sometimes the the

conversations we have it seems like chbt

is is seen as like the solution to

everything which is clearly not the case

and we are also seeing that for example

in Industry we we have a lot of clients

in Industry um the main data sources

that we really have there are time

series and tabular data and this should

not be underestimated so everything that

was always said before like build up

your data um warehouses build up your

lake houses build up your um make sure

you you build the Foundation to utilize

this value this still holds true um up

to this point and there are a ton of

methods that should not be

underestimated um and then on top of

that language comes in as a very

interesting interface to access this

data in a new way and I think that's

that's really cool to see um but we also

we always have to make a bit of this

trade-off to see where is really where

is it language that solves the use case

and where is language that's the cool

addition to have a new way to interact

can I just react on this so generally

100% get your get your basic straight

before you jump into the more Advan St

stuff couldn't agree more but where I

would disagree with you is really since

we in the point you said um that you see

most with customers you see mostly

structured data in Time series yes they

do have a lot of that but they have a

lot of that because they didn't pay

attention to all the textual data or

potentially the data that could

potentially Ed be used for um for NLP

use cases I would say so obviously

there's a lot of structured data and

time series and obviously companies

still struggle with the basics in

getting really the infrastructure state

but um I would say there is even more

textual data they just didn't unlock

that yet as data that has potential to

be used for AI so much absolutely yeah I

think I can I can only agree with that

and I think maybe to to add one one

aspect there um um is that I think in

every company us have functional units

that have a lot of text being that the

legal space being that um yeah HR

internal knowledge bases way where you

have a lot of data that you can unlock

um in in new ways um nowadays and then

in your core business it really depends

on what you're doing if that's also

textual or if that's um different data

but I agree um in that sense yes I think

every company has to some extent

potential to look at your text

Data thanks a lot for for for the rewind

um to come to like the current current

scenario um we briefly talked Hana you

mentioned it um about like the the the

way you WR prompts um also there's F

short learning um you mentioned it

before in your talk in your talk as well

to to um utilize for example giving a

demonstration of um of different use

case of um the things that the AL is

supposed to do um to like fine tune it

in a way

um is there um any are there any

examples could you share any example

where where for example these

um the these two types of fine-tuning in

LM a task osting llm um has led to any

impact in any

industry um by for example adding this

context and thereby for example

unlocking longtail use cases that

formerly couldn't couldn't even Sol

because there's simply no return invest

sure so generally what I'm seeing in in

my experience from working with LMS is

that find tuning multi things so

generally supervised learning in the

sense what we already know fromal

machine learning makes a lot of sense

and it increases the quality of your

answers by a huge margin um so I believe

most use cases will go in the direction

of using some sort of multi pumping fine

tuning Etc um that's the basic basic

layer to as as an answer of your

question in terms of specific industry

use cases yes I think there are some

industries that have high potential for

the use of LMS for example Med medical

industry legal use cases um some others

right um but you can already see that

because these are very text based at the

same time text based usually um but at

the same time very rule based in some

sense right it's sort of pattern

recognition in both both of these cases

a lot of cases but at the same time

they're also highly regulated and you

work with highly sensitive data and

specifically in the medical case you

work with data and you work with

decisions that you should not get

wrong right um even more in the medical

space because it's sometimes about lives

right in legal space it's a lot it's

about a lot of money usually so there is

an inherit value for llms the task is um

makes sense to solve but we have

specific challenges why anms need to get

these points right and in these cases we

probably should think about how to train

L&M that are specific for the industry

great example for this is met Palm so

our foundation model is Palm right and

we're one of the research projects that

is in preview at the moment actually is

metp for example right our research

trained Palm fine tuned it on medical

data um fine tun it for this specific

use case so essentially in the back

combine factual knowledge that the llm

needs to get right in any case with the

generational knowledge of of Pama the

generational capabilities right and this

works quite well I mean p is not public

yet but it already outperforms any other

model in the medical um certification

tests in the US for example um similar

for the legal knowledge I don't I'm not

aware at least for legal llm purely

legal find you LM probably it's out

there already but I can imagine very

well that will fine tune or train

specifically in llm on for example a

certain legal codex um to to answer

specific questions and compare claims

between parties um and have the legal

system in the rules of that yeah thanks

so some we at project a and and you and

your work we we don't have the capacity

to train whole models ourselves to find

you find you them um have you seen any

recent either in our portfolio or like

um on on blogs or have you read about

recent implementation um success stories

also challenges uh with

llms um

I think I would go in the direction that

that that yakob just mentioned um

because smart struggle with LMS I don't

want to talk too much of the skeptic is

in this aspect um there is a possibility

to find trun LM based on these

foundational models given a certain

distinct situation or data set and make

them perform very very well on those you

mentioned Health you mentioned legal HR

possible whatever um however what I then

very much strug given that these are

strongly probabilistic models is the

possibility

that these actually are not doing what

they're supposed to do because um

validating these llms is incredibly

difficult um so there are numerous

articles where I've seen that while they

might perform very very well on these um

legal bar exams or these Health exams

whatever that might be putting them to

practice they always are being

outperformed easily by an actual lawyer

by an actual doctor um which makes this

this um intelligence aspect which is

being touted as this the strength of

these llms kind of ambiguous because

what does it actually mean if you

perform well on the test but then in

practice you're actually quite shite um

I don't know I'm not the one to answer

this question but this is why I sort of

struggle with this notion that when you

have some sort of AI model in an LM

model it's opposed to um I don't want to

say replace but it's actually I think

just there's an assistant and therefore

I I I'm bit skeptical to see it's going

to have this massive massive impact that

a much much simpler models have had so

far so um if if you ask me if there's

one thing that is the biggest or had the

biggest value to anything it would be

linear regression but this came out what

150 years ago or something right U but

it still works and it works very

very well um and if LM I think there's

this risk to it that that might not be a

more deterministic approach that that

integration might offer

um so I think for me and and I think

it's going bit divert a bit from your

original question um thinking of

industry and application my struggle

still seems to be a validation and

making sure that these things work the

way they're supposed to do no as on like

old man get get clout but this is sort

of the approach that I take when we

think about applying these things to

Industries because no in our portfolio

it's not really that big um because

there I think we rely also much more on

hanus approach mentioned earlier like

get your Basics right get your linear

regression right before you can talk

about um llms etc etc etc because that's

too far away and most of the time might

not have impact you actually desire

might you add on this I would love to I

think I would like to share one example

which I think illustrates quite well how

in this case the legal space Also

benefits from the recent advantages and

also how we at momentum kind of handle

this this risk because yes I absolutely

agree um we have probabilistic models

here we have models that hallucinate um

we have these risks and it's very

important for us to make that very very

transparent and handle that but I do

think that assisting work um while an

expert a human expert has the final say

is still something that can have huge

benefits um huge impact so one case i'

I'd like to share from our project um

portfolio is that we are working with a

with a legal partner um and this one is

about um yeah working and processing

relatively standardized

um legal regulation um think basically

ndas or something like that um and

what's very interesting is that I think

handling something in the legal space

benefits a lot from recent advantages so

one thing for sure is models being

fine-tuned on legal data in general so

being able to process this kind of

language and being able to process

longer documents with a lot of context

but I what I find very interesting and

this wasn't basically possible um 10

months ago is that in the legal space

you also always have not only one fixed

set of laws but you also have a

continuous flow of cases of reference

cases you need to consider if you're

making a specific decision and um this

new way of context specific learning

essentially allows you to feed that into

your model without having to train it

but just you can give examples you can

give reference cases and you can extract

Knowledge from all of that and at least

like provide a summary of um what the

models then train to um detect as

relevant passages um give that to an

expert and it's already helping you and

giving these pointers and giving

references and yes it needs a lot of

constraint so that it will give this

references but it allows you to combine

internal and external sources and I

think that's very very um cool to see

and definitely something that's that's

possible and that we are currently

working on for example I wouldn't AG I

wouldn't disagree with this I think I

would actually sort of agree with this

um because I think the main benefit of

this as you said is assistance that I

would say complement existing work so

you always read about um companies that

use AI had an productivity benefit of

uplift of what 20% and I think that's

that 20% that a lawyer would spend

reading this long boring documents for

being P to make sense of what's going on

there and and that's essentially when we

go to a bit more of an established

approach which is sematic understanding

right I think translation of documents

was more less a solved issue in in 2010

whatever right and and there we had a

clear Benchmark right um I don't know

what it's called but it's essentially

when you have a machine translated text

versus human translated text how Sim

arties to and that essentially defines

how well the transation is going and

that's how we also measured how well

Transformers perform actually whenever

launched um I think this you this this

one Benchmark is the what's missing to

then rely on what's a legal um what say

illegal AI might do and I ask a gen

based on legal text hey is this bad am I

being suit now I wouldn't trust it

advice but I would trust it hey what

does it tell me essentially this um this

transfer knowledge that we're having

from what is been generated and is it

actually applicable can I do something

with it I think that bridge is still

missing but the first step that's need

before understanding what's going on

yeah that's amazing right that

works quite well I would

say oh thanks um then now looking

forward you you already mentioned that

there's like there's like a gap until

until LMS like until you can fully trust

LMS or if you want if you want to add

this this maybe the first question if

you want to fully trust alms um but but

more generally speaking um

what like yob to you the question um

what what further advancements do you

see to undergo for example to get rid of

this the symptom that you would like

only um consult lmms and then make yeah

make your own decision um to for example

being LMS being um able to decide for

themselves and and act on

this difficult question I think um that

we're quite far are still from a place

where llms do everything and take over

all the decision processes and I think

you have to differentiate a little bit

what's the difficulty of a certain task

difficulty of a certain decision right

for simple tasks that don't require 100%

accuracy all the time we're already

there they can already take over quite a

lot in my opinion for example like chbd

plugins insta card just giv me like a a

recipe and then order it on inst card

right right yeah yeah simple

interactions right um there's also maybe

not that much of the line we can already

do that I would say um to get to a point

where llms do much more complex

decisions uh or take them over

completely I don't think we're going to

get there anywhere soon because also

traditional machine learning doesn't

most of the time do that I would say

right I think in some cases very simple

case it in cases it does and we need to

scale up a lot of very simple decisions

yes I see it in a similar sense maybe

less quantitatively based but more text

based decision that are that are being

taken um I think especially in in the

interaction space right really having

this as a sort of first interaction

layer with a user with a customer

internal user external user I think

that's where LMS will take over a lot of

decisions very independently but really

making pivotal business decisions for

example um I don't think we're going to

get there yet I think it's really

similar to machine learning as it does

already taking over over the 80% of the

manual work and then still leaving the

20% and the final decision- making to

the to the human in many cases yeah well

thanks so so so you rather say okay if

your question if you really want it want

them to to make decisions um rather

being like the first point of contact um

yeah exactly exactly yeah um okay so

really interesting because I recently um

watched the talk from the Nvidia CEO and

um he envisions a future or predicts a

future where LMS will be um the first

point of contact for every interaction

with the system um Hannah would you go

uh with his uh his prediction of the

future and maybe also um what do we need

to undergo in terms of uix um until we

reach maybe such a

point well I love this question it's a

very steep hypothesis uh

so I think I have a slightly different

take there um to be honest I don't think

language is always the most efficient

way to communicate with a system if I'm

very honest I think sometimes a simple

drop down and hit and go actually does

the job quite well so I I think we

should keep that in mind like what what

is really what is the good experience

for the user to interact with a system

if there if you have for example an FAQ

um space with only five questions adding

a chap but on top can make it even like

actually quite confusing because you

would have to kind of guess what this

chatbot can help you with instead of

just like browsing quickly having a

glance at these five

questions but in general I I do agree

what's an very interesting trend is to

see language being more and more an

interface between um humans and machines

even between machine and machines in the

sense that we see this trend of

multimodal models involving where you

have models trained on C different

modalities like for example image and

text or so video and text so you can now

ask questions about an image for example

and get the output read to you for

example by an AI voice so there's so

many ways how language kind of becomes

this interface and I think this is an

incredibly interesting Trend so yes I do

think that we will see a further rise of

language based uh interactions but I'm

not quite convinced that prompting and

like iterating on prompts to get the

right answer is necessarily the solution

for for every

system think you just described clippy

basically um so I generally do not think

language is anywhere close to being the

most efficient way to to to with any

sort of system I think and that's

essentially also what I attribute to the

success of chat2 is the the tendency of

humans to feel much more comfortable

with anthropomorphic systems like chat

talks and reads like a human being and

that's what makes it so impressive

whereas a complicated I mean we all Tech

space here CLI or whatever interface you

might have is then sort of scary and

spooky

but I think most of us in this room

might actually prefer a CLI because

that's the most efficient way to maybe

communicate with whatever is that you

want to do do I think text is efficient

absolutely not it takes a long time to

write takes a long to read a drop down

is two clicks so I hope that we do not

get into this direction because I think

we go in circular we can see that uh the

understanding of systems is increasing

you know as I said my dad is quite well

as iPhone these days you know um so we

don't need these systems from like

Clippy which were designed to in the

'90s to make people who do not

understand what you know the trash can

means on your desktop to explain it to

them but but one point where why I

partly disagree let's say is that sure I

agree with an educ with your point for

an educated bubble of very tech savvy

people but now that's not what Humanity

looks like right I think the really

average and the majority of folks that

used to technology use most of the

products they are rather uncen towards

technology right for them is technology

is something that they use as a user but

they don't think about what's the most

efficient way in communicating with it

and for them natural language is the

most natural way for sure and they don't

even think about whether it's the most

efficient or not for them it's the first

way that someone would communicate so in

that light I would say surely it's a

bold statement to say it's every single

time the first um layer of communication

but I would say we're going to hit I

know if you look at all the system inter

system interaction probably over all the

users probably 90% will be hitting a

natural language layer first and then um

access functionality and I mean a good

example of that is um who would have

thought before search engines came out

that some sort of keyword search uh

weird query is the way of interact with

online content right is this the most

efficient way to interact with it

probably not right but it's the way that

really that everyone can use right that

everyone used to be able to use and

similarly I think the way of querying

documents quering knowledge is going to

change quite a bit more towards the

semantic way and more use more towards

using natural language but similar to

search I think we're going to have

similar manner that search bars work now

going have more agents or search bars

that act as agents to interact as a

first layer that's for

sure maybe to just add one thought

because I recently read an interesting

Comon from the niss Norman Foundation

which you might know it's like a famous

ux research organization so one thing

they said about prompting is that you

also have to keep in mind that it

requires a pretty high degree of

literacy to to frame your thoughts in a

way that you get the answer you want at

least at the moment because you yeah we

all know this experience of prompting

and then not exactly getting what we

wanted and then refining the prompts so

so I think one interesting aspect there

is also to make to make sure this is

very inclusive if language becomes more

and more an interface so um thinking

about also the developments in research

towards integrating like more longtail

languages for example um allowing

interactions not only in English but

like in a diverse set of um languages

it's just one strand of research of

things we are seeing that still need to

happen in the space of NLP next to

getting rid of hallucinations or aspects

like that but but I think language and

extending to to these more long TR

languages um is very

important thought there um completely

agree that obviously prompting in the

back right when you're hitting the llms

is complex and need to follow a certain

structure but on the other hand I'm not

expecting from my user to write these

prompts right if I'm building an

building an application and I'm

integrating LMS then then it's on me to

make sure the user can use whatever

natural language they um used to and

then I need to write the prompt that

talks to the to the llm directly and

make sure that it includes um or makes

the llm understand whatever query the

user word in the natural language so I

don't think that or literacy in terms of

languages for sure obviously needs to

inclusiv needs to beur but I actually

think that lmms have the huge advantage

that different types of language

literacy in a sense can be considered

someone with very complex language can

write a switch query and it can be

similarly understood to someone with

very easy language because the llm can

make can actually understand what they

want so I'm not so sure about it though

I think that's I think that's what we

disagree U because I think language is

and I'm not just talking about this

bubble that we in it's just not the most

efficient way to transort information um

I can't recall into how many arguments

I've gotten on WhatsApp because a

message I've written was receiv in a

wrong way and if any sentence you say

and you put the emphasis on something on

a different place it has a completely

different meaning I told you that's a

nice shirt you might be offended whereas

I might actually as a genuine compliment

because I also like wuang you know um

and and I think this is then the issue

that you have when you then interact

with these systems where if you say hey

I want um last quarter's

Revenue what quot are we talking is it

the financial year is it being you know

the calendar year and and these context

is what then need to figure out from

person Business site and then also say

if you talk about the 90% of

consumers what does it mean to them we

talked about um it being very accessible

and I think this sort of reminds me of

um when iOS launched they had this this

heavily geomorphic design so your

contacts were like like a roller de

because we had to sort of accustom

people that swiping means you go through

something you had to pinch to zoom

because well I don't know what this

occurs naturally but we had all of these

metaphors for very familiar objects and

we've moved away from this because now

we have more phones and people on this

planet and we sort of understand this is

the Gen the overall gesture to zoom into

something and this means up and down and

I think with more technical literacy

that we have as as a human race we're

also going to move away from these

abstractions that might make things

actually a bit more difficult my opinion

because prompting and getting the right

thing is actually not as straightforward

as it likely to

be abolutely actually it's funny that

you took that you um chose the Q3

Revenue last quue example because that's

exactly the one of the examples I had in

my talk earlier right but I believe that

with a good semantic understand layer

that understands semantic meaning you'll

be able to cover that with an L because

an LM can either ask you question ask

you back hey there are multiple quarters

right which one do you mean or quarter X

quarter y quarter z um and that's

exactly the major advantage in my

opinion because you can use the natural

language and it can make these

connections put them into context maybe

see what's available in my database in

my knowledge base and um draw

conclusions across essentially so yeah

completely see the challenge 100% um but

that that's really on the prompting

don't give the answer directly but um

consider all the options or however

right okay I I have the feeling that the

conversation could go could go on for

like U plenty more minutes I have a lot

of questions as well but I want to give

you as the audience the chance to ask

questions if there are any um to our

three

panelists otherwise as I said I have I

have plenty more so um but we are happy

to assist you with a microphone and

then you can ask

him if not uh then I will ask my

questions um so we now talked about like

the basic interaction with the system

but now coming back to M um what do you

think will the the role be of llms in

this whole uh sphere of interaction with

the with the system where will the the

role in the future be um will it be for

example for every textual interaction

will it be an llm or uh where do you see

it I think the main benefit of llms at

the moment are are the multimodality

right they can work with texts but they

can also maybe work with tables you

canook up to your dat datase they can

work images and um in a in a in a

scenario maybe 10 20 years from now we

could speak that would be is that this

sits basically a part of infrastructure

it's part of your office suite as part

of all the tables that you have and from

that generates your overall knowledge

base and having essentially your your

your your Google llm your marantic llm

your llm which essentially is then your

go-to person for questions you have and

okay listen where are these slides that

are built for four years ago and they're

oh here they go oh thank you um this is

what I see essentially this main core

productivity benefit that an llm can

offer on a productive scale as part of

any organization really I think that's a

bit about the accessibility here this

can be part of any literally any company

it does matter how boring or fashioned

you are this can be embedded anywhere

right now I think we're not there yet

absolutely not because LS are then

always constricted to some sort of

microcosm that we want to explore maybe

that's your own your database for your

SES Maybe it's only you know some

textual data as part of your HR

department whatever um but I think this

is missing the Integrity of integration

across all departments which is where

going to get the main benefit

overall and I think textual information

is the start of it but then really the

underlying value it's always thinking a

little bit what are the patterns that

where again um technology strength of

the technology come together with the

business value right what the common

patterns you can kind of reuse use use

in every single company and I think this

um building a semantic understanding for

whatever data you're working with

whether it's uh structured data whether

it's time series data where whether it's

textual data whether it's images right

connecting all of that in a semantic

knowledge base that is huge it's huge so

so so sort of sort of a an evolution of

of search or organization of information

right just like a super quick addition I

think what I'm also really looking

forward to seeing is more use cases

where you you have a true interaction

which is kind of both sided so it's not

just us learning how to better prompt to

get a better result but it's also the

model kind of gradually learning as we

go along so this is something we are

seeing as a trend not only in language

but basically overall in machine

learning and I think the coolest systems

and it's something I'm super excited

about in general is whenever you we're

at a stage where you provide some input

and the model maybe gives a suggestion

and then you give some sort of feedback

as a human and the model learns from the

experts gradually as you advance and

then step by step you can automate

further essentially and I think that

would be great to to see that also um

more more with llms in general um but in

general I also absolutely agree with

what you've said already thanks we're

going to take this

question thanks for the discussion was

really cool and I have

question basic like actually it's two

questions but I try to compile it in one

um at which level of like let's say

success rate do you think will llms

actually be implemented is that like an

do you think that's like an absolute

like it shouldn't be like 99% it will

give the correct answer and then 1% it's

going to give you the wrong answer for

like something where a customer asks you

like I let's say at the um as a customer

service right like and you ask a

question how do I do exploit that at

which point do you think companies will

actually

feel safe with llms and Which percentage

and how do you ensure that they actually

trust them because right now I think

there's still like this ah yeah but

maybe what about that 1% when they like

go off and then they're like how do I I

don't know update my iPhone and then it

tells you just throw it into to the

microwave or something I don't know um I

know it's it will be more reasonable

than that but like how do you see that

um and how do you think or what do you

think it's another question how do you

think like the whole topic or like the

whole argument of l&m's just being

stochastic parrots um what do you think

about that do you think they are

stochastic parrots only or do you think

they have like an inner World model that

they actually can somewhat reason we

already ran out of time but Hannah I

would say um you will give give the

answer and then um you're you're always

welcome to then discuss afterwards but

Hannah here you go okay I will I will

hurry so that you also get a chance to

comment if you like so stochastic parrot

yes definitely I mean this is partially

what they are built for that's great uh

they they they give um the most

probabilistic answer which is very very

powerful but also comes with these

limits of potentially giving wrong

answers so I think in general you can

think about it in two aess one is how

good is the model compared to how good a

human could do it and the other is um

kind of how sensitive is the use case

and whenever um the model is better than

human and the case is not that sensitive

or for full automation if the model is

better than human or close to human but

the case is somewhat sensitive you

probably already reached the stage where

you want to have a human in the loop and

then we can have different stages of or

different ways of having the human in

the loop but you probably always want

that and I think

if you go for an approach of that it

takes out a lot of risk while still

utilizing the benefits and this is

something I would always recommend to

clients I think it's not reasonable to

to automate fully if it's a very risky

case or a human could do the job

better I see someone and yob nodding so

we're gonna take this thanks a lot Hanah

um so thanks a lot for the uh L lovely

discussion and like I said I I have the

feeling that we could go on for some

more minutes um one that I want to

mention definitely is um the park on

tomorrow um we we are at C and someone

is giving a talk there as well on the

interesting topic why causality is the

ne next big step in AI uh it is in

Cinema 4 at 2:30 so don't miss it I will

be definitely there um and uh this

leaves me with nothing less than uh

expressing my gratitude to you three uh

for sharing your knowledge to project a

team and Google team and also for your

questions and your

thanks lot

you