The New, Smartest AI: Claude 3 – Tested vs Gemini 1.5 + GPT-4

Summary

Claude 3 von Anthropic, der als der intelligenteste Sprachmodell der Welt bezeichnet wird, hat seine technischen Berichte veröffentlicht. Es übertrifft GPT 4 und Gemini 1.5 Pro in vielen Bereichen, insbesondere in der optischen Zeichenerkennung (OCR) und der Beantwortung von komplexen Fragen. Trotz seiner Fähigkeiten ist es noch nicht allgemeine künstliche Intelligenz (AGI), wie einige seiner Fehler bei grundlegenden Fragen zeigen. Anthropic zielt darauf ab, Geschäftsanwendungen zu fördern und behauptet, dass Claude 3 in der Lage ist, Einnahmen zu generieren, komplexe Finanzprognosen durchzuführen und Forschung zu beschleunigen. Trotz der hohen Preise und der Kritik an einigen Aspekten, wie der Mathematischem Verständnis, wird Claude 3 aufgrund seiner niedrigen Ablehnungsrate und seiner Fähigkeit, Risiken einzugehen, bei vielen Benutzern beliebt sein. Die Zukunft von Claude 3 und der Entwicklung von AGI bleibt spannend und unvorhersehbar.

Takeaways

• 🚀 Claude 3 von Anthropic ist der intelligenteste Sprachmodell auf dem Planeten, wie in einem kürzlich veröffentlichten technischen Bericht behauptet.
• 📝 Der technische Bericht wurde weniger als 90 Minuten vor der Veröffentlichung gelesen und mit den Release Notes verglichen.
• 🔍 Claude 3 hat in 50 verschiedenen Tests gute Ergebnisse gezeigt, insbesondere in der optischen Zeichenerkennung (OCR).
• 🌐 Claude 3 ist das einzige Modell, das den Bärberstuhl in einem Bild korrekt identifiziert hat, was seine Fähigkeiten in der Bildverarbeitung zeigt.
• 🤖 Trotz seiner Fähigkeiten ist Claude 3 noch nicht eine allgemeine künstliche Intelligenz (AGI), wie einige Tests gezeigt haben.
• 💼 Anthropic zielt mit Claude 3 auf Geschäftsanwendungen ab und behauptet, dass es in der Lage sein wird, Einnahmen zu generieren und komplexe Finanzprognosen zu erstellen.
• 💡 Claude 3 hat niedrigere falsche Ablehnungsraten als andere Modelle, was seine Popularität erhöht.
• 📚 In der Verwendung von Sprache und Texten zeigt Claude 3 eine hohe Fähigkeit, auch in komplexen Anfragen.
• 🔢 Mathematisch ist Claude 3 besser als GPT 4 und Gemini 1.5 Pro, sowohl in grundlegenden als auch in fortgeschrittenen Mathematikfragen.
• 🌐 Bei mehrsprachigen Tests zeigt Claude 3 erhebliche Vorteile gegenüber anderen Modellen, was seine Fähigkeiten in der Verarbeitung von Sprachvielfalt betont.
• 🔥 Anthropic behauptet, dass Claude 3 mit seiner verfassungsmäßigen KI-Methode sexistische, rassistische und toxische Ausgaben vermeiden wird.

Q & A

Was ist neu bei Claude 3 im Vergleich zu früheren Versionen?

Claude 3 ist als der intelligenteste Sprachmodell der Welt angekündigt und hat verbesserte Fähigkeiten in der optischen Zeichenerkennung (OCR), Mathematik und multilingualen Aufgaben. Es hat auch eine niedrigere Rate von falschen Ablehnungen und kann komplexe Anfragen besser beantworten.

Wie hat sich Claude 3 in Bezug auf OCR-Fähigkeiten entwickelt?

Claude 3 hat sich signifikant verbessert und ist nun besser in der Lage, OCR-Aufgaben zu bewältigen, was sich in der korrekten Identifizierung von Lizenzplaketten und Barberpolen zeigt.

Welche Art von Anwendungen plant Anthropic für Claude 3?

Anthropic plant, dass Claude 3 in Geschäftsanwendungen eingesetzt werden kann, um Einnahmen zu generieren, komplexe Finanzprognosen durchzuführen und Forschungsarbeit zu beschleunigen.

Wie positioniert sich Claude 3 in der Preisgestaltung im Vergleich zu GPT 4 Turbo?

Claude 3 ist teurer als GPT 4 Turbo, was auf seine verbesserten Fähigkeiten und potenzielle Geschäftsnutzungen hindeutet.

Welche Art von Sicherheitsmaßnahmen hat Anthropic für Claude 3 implementiert?

Anthropic hat eine verfassungsmäßige KI-Methode angewendet, die darauf abzielt, sexistische, rassistische und toxische Ausgaben zu vermeiden und Menschen daran zu hindern, illegale oder unethische Aktivitäten durchzuführen.

Wie hat sich Claude 3 in der Verarbeitung von komplexen mathematischen Aufgaben bewährt?

Claude 3 zeigte sich in der Verarbeitung von komplexen mathematischen Aufgaben besser als GPT 4 und Gemini 1.5 Pro, was seine verbesserte Fähigkeit in diesem Bereich zeigt.

Wie hat Claude 3 es geschafft, in einem Theorie-der-Geistes-Test zu bestehen?

Claude 3 hat in einem angepassten Theorie-der-Geistes-Test, der das Wort 'transparent' enthielt, erfolgreich bestehen, was seine Fähigkeit zeigt, komplexe Sprachspiele zu verstehen.

Welche Einschränkungen hat Claude 3 bei der Verarbeitung von Anfragen bezüglich ethischer und rechtlicher Fragen?

Claude 3 hat bei der Verarbeitung von Anfragen, die ethische oder rechtliche Probleme aufwerfen, wie das Anheuern eines Attentäters oder das Dieben eines Autos, gezeigt, dass es diese Anfragen ablehnt.

Wie hat Claude 3 es mit der Verarbeitung von mehrsprachigen Inhalten bewältigt?

Claude 3 hat sich in der Verarbeitung von mehrsprachigen Inhalten als deutlich besser als GPT 4 und Gemini 1.5 Pro bewährt, was seine Fähigkeit zeigt, in verschiedenen Sprachen zu denken und zu antworten.

Welche Aussagen macht Anthropic über die Zukunft von Claude 3 und der AGI-Forschung?

Anthropic glaubt, dass die Modellintelligenz noch nicht ihre Grenzen erreicht hat und plant, regelmäßige Updates für die Claude 3-Modellfamilie in den kommenden Monaten zu veröffentlichen, um bessere Sicherheitsforschung zu betreiben.

Outlines

00:00

🤖 Einführung in Claude 3 und erste Eindrucke

Der Sprecher diskutiert die Veröffentlichung des technischen Berichts zu Claude 3, der als intelligentestes Sprachmodell der Welt bezeichnet wird. Er hat Claude 3 in verschiedenen Szenarien getestet und stellt fest, dass es insbesondere in der optischen Zeichenerkennung (OCR) überlegen ist. Trotz einiger Schwächen, wie die fehlende Fähigkeit, komplexe mathematische Fragen zu beantworten, ist Claude 3 populär werden wird, da es in der Geschäftswelt großes Potenzial bietet. Der Sprecher erwähnt auch, dass Anthropic, das Unternehmen hinter Claude 3, ein Modell für ein vollständiges KI-Labor schaffen könnte.

05:02

🔍 Analyse von Claude 3s Fähigkeiten und Herausforderungen

Der Sprecher geht auf die Fähigkeiten von Claude 3 ein, insbesondere seine OCR-Fähigkeiten und seine Reaktion auf komplexe Anfragen. Er stellt fest, dass Claude 3 in der Erkennung von Objekten in Bildern und der Beantwortung von Fragen über diese Objekte besser ist als andere Modelle. Allerdings scheitert es bei der Logik und mathematischen Vernunft. Der Sprecher kritisiert auch, dass Claude 3 in Bezug auf seine ethischen Grenzen nicht perfekt ist, da es nicht in der Lage ist, rassistische oder sexistische Anfragen zu erkennen und zu blocken.

10:03

📈 Vergleich von Claude 3 mit anderen KI-Modellen

Der Sprecher vergleicht Claude 3 mit anderen KI-Modellen wie GPT 4 und Gemini 1.5 Ultra. Er erwähnt, dass Claude 3 in vielen Bereichen, wie Mathematik und Multisprachigkeit, besser abschneidet als seine Konkurrenten. Er diskutiert auch die Fähigkeit von Claude 3, komplexe Aufgaben zu lösen, wie das Schreiben von Shakespeareanischen Sonetten und die Beantwortung von schwierigen Fragen im Bereich der Wissenschaft. Trotz einiger Herausforderungen, wie die Fähigkeit, sich autonom zu verbessern, sieht der Sprecher in Claude 3 ein großes Potenzial für zukünftige Entwicklungen.

15:03

🌟 Claude 3s Zukunft und die Vision von Anthropic

Der Sprecher schließt mit einer Diskussion über die Zukunft von Claude 3 und die Ziele von Anthropic. Er erwähnt, dass Anthropic sich auf die Verbesserung der Sicherheit und Verantwortung ihrer KI-Modelle konzentriert und nicht nur auf den Gewinn. Der Sprecher ist zuversichtlich, dass Claude 3 und zukünftige Modelle in der Lage sein werden, autonome Fortschritte zu machen, und erwartet, dass die KI-Technologie weiterhin schnell voranschreitet.

Keywords

💡Claude 3

Claude 3 ist eine künstliche Intelligenz-Sprachmodell, das von Anthropic entwickelt wurde und als das intelligenteste auf dem Planeten bezeichnet wird. Es zeichnet sich durch seine Fähigkeit zur optischen Zeichenerkennung (OCR) und seine Anpassungsfähigkeit für verschiedene Anwendungen aus. Im Video wird es mit anderen Modellen wie GPT 4 und Gemini 1.5 verglichen, um seine Effizienz und Intelligenz zu bewerten.

💡Anthropic

Anthropic ist ein Unternehmen, das sich auf die Entwicklung von künstlicher Intelligenz spezialisiert hat. In diesem Kontext wird Anthropic für die Schaffung von Claude 3 herangezogen, um seine Fähigkeiten und potenzielle Anwendungen in verschiedenen Geschäftsbereichen zu präsentieren.

💡OCR (Optical Character Recognition)

OCR ist eine Technologie, die es Computern ermöglicht, Text aus Bildern zu lesen und zu interpretieren. In der KI-Welt ist dies eine wichtige Fähigkeit, um Informationen aus digitalen oder gedruckten Quellen zu extrahieren.

💡AGI (Artificial General Intelligence)

AGI bezieht sich auf eine hypothetische Form der künstlichen Intelligenz, die in der Lage ist, intelligente Entscheidungen und Handlungen in einer Vielzahl von Umgebungen und Situationen zu treffen, ähnlich wie ein Mensch.

💡Benchmarks

Benchmarks sind Standardtests oder Messungen, die dazu verwendet werden, um die Leistung oder Effizienz von Technologien, wie KI-Modellen, zu bewerten und zu vergleichen.

💡Ethical AI

Ethische KI bezieht sich auf die Entwicklung von künstlicher Intelligenz, die nicht nur intelligent, sondern auch verantwortungsbewusst arbeitet, indem sie rassistische, sexistische oder toxische Inhalte vermeidet und Menschen nicht dazu anregt, illegale oder unethische Handlungen zu begehen.

💡Enterprise Use Cases

Unternehmensnutzfälle beziehen sich auf die Anwendungen von Technologien wie künstlicher Intelligenz in Geschäftsumgebungen, um Prozesse zu optimieren, Kosten zu senken und neue Geschäftsmöglichkeiten zu erschließen.

💡Language Models

Sprachmodelle sind künstliche Intelligenz-Systeme, die dazu trainiert sind, menschliche Sprache zu verstehen, zu generieren und zu interpretieren. Sie werden in vielen Anwendungen eingesetzt, von Chatbots bis hin zu Übersetzungsdiensten.

💡Risque Content

Risikoinhalt bezieht sich auf Inhalte, die potenziell unangemessen, beleidigend oder kontrovers sein können. KI-Modelle müssen darauf trainiert werden, solche Inhalte zu erkennen und zu vermeiden.

💡Model Intelligence

Modellintelligenz bezieht sich auf die Fähigkeit von KI-Modellen, komplexe Aufgaben zu lösen, Informationen zu verarbeiten und intelligente Entscheidungen zu treffen.

💡Safety Research

Sicherheitsforschung bezieht sich auf die Entwicklung von Methoden und Technologien, die dazu beitragen, die Verwendung von künstlicher Intelligenz sicherer zu machen, indem Risiken minimiert und potenzielle Missbrauchsmöglichkeiten vermieden werden.

Highlights

Claude 3 is claimed to be the most intelligent language model on the planet.

The technical report on Claude 3 was released less than 90 minutes ago.

Claude 3 has been tested in about 50 different ways, including comparisons with unreleased Gemini 1.5 and GPT 4.

Claude 3 demonstrates strong OCR (optical character recognition) capabilities.

Claude 3 is the only model to identify the barber pole in a test image.

Claude 3's false refusal rates are much lower, making it more user-friendly.

Anthropic is targeting businesses with Claude 3, emphasizing its value for revenue generation and complex financial forecasts.

Claude 3 is priced higher than GPT 4 Turbo, reflecting its advanced capabilities.

Claude 3 has a lower rate of mistakes in basic tasks compared to GPT 4 and Gemini 1.5.

Anthropic's constitutional AI approach aims to avoid sexist, racist, and toxic outputs.

Claude 3 shows impressive performance in graduate-level Q&A, scoring 53% accuracy.

Claude 3 can accept inputs exceeding 1 million tokens, though initially limited to 200,000 tokens.

Claude 3 demonstrates the ability to follow complex instructions, such as creating a Shakespearean sonnet with specific requirements.

Anthropic's CEO, Dario Amodei, emphasizes the company's focus on safety research over profit.

Claude 3 shows potential for autonomous resource accumulation and software exploitation, though it requires hints to succeed.

Claude 3's performance on benchmarks suggests it may be the most intelligent model currently available.

Anthropic plans to release frequent updates to the Claude model family, with a focus on enterprise use cases.

Claude 3's ability to generate revenue and conduct complex financial forecasts is a key selling point.

Claude 3's performance in multilingual tasks and coding is noticeably better than GPT 4 and Gemini 1.5 Pro.

Transcripts

Claude 3 is out and anthropic claim that

it is the most intelligent language

model on the planet the technical report

was released less than 90 minutes ago

and I've read it in full as well as

these release notes I've tested Claude 3

Opus in about 50 different ways and

compared it to not only the unreleased

Gemini 1.5 which I have access to but of

course GPT 4 now slow down those tests

In fairness were not all in the last 90

minutes I'm not superhuman I was luckily

granted access to the model last night

racked as I was with this annoying cold

anyway treat this all as my first

impression these models may take months

to fully digest but in short I think

Claude 3 will be popular so anthropics

transmogrification into a fully-fledged

foot on the accelerator AGI lab is

almost complete now I don't know about

Claude 3 showing us the outer limits as

they say of what's possible with Gen AI

but we can forgive them a little hype

let me start with this illustrative

example I gave Claude 3 Gemini 1.5 and

gp4 this image and I asked three

questions simultaneously what is the

license plate number of the van the

current weather and are there any

visible options to get a haircut on the

street in the image and then I actually

discussed the results of this test with

employees at anthropic they agreed with

me that the model was good at OCR

optical character recognition natively

now I am going to get to plenty of

criticisms but I think it's genuinely

great at this first yes it got the

license PL correct that was almost every

time whereas gpc4 would get it sometimes

Gemini 1.5 Pro flops this quite

thoroughly another plus point is that

it's the only model to identify the

barber pole in the top left obviously

it's potentially a confusing question

because we don't know if the Simmons

sign relates to the barber shop it

actually doesn't and there's a sign on

the opposite side of the road saying

barber shop so it's kind of me throwing

in a wrench but Claude 3 handled it the

best by far when I asked it a follow-up

question it I identified that barber

pole GPT 4 on the other hand doesn't

spot a barber shop at all and then when

I asked it are you sure it says there's

a sign saying Adam but there is another

reason why I picked this example all

three models get the second question

wrong yes the sun is visible but if you

look closely it's actually raining in

this photo none of the models spot that

so I guess if you've got somewhere to go

in the next 30 seconds I can break it to

you that Claude 3 is not AGI in case you

still think it is here's some casual

bias from Claude 3 the doctor yelled at

the nurse because she was late who was

late the model assumes that the she is

referring to the nurse but when you ask

the doctor yelled at the nurse because

he was late who was late the model

assumes you're talking about the doctor

but things get far more interesting from

here on out anthropic are clearly

targeting business with the Claude 3

Model family they repeatedly emphasize

its value for businesses just quickly on

the names Opus of course refers to the

biggest version of the model because an

opus is a big body of literature a Sonic

is typically 14 lines medium size and a

high Q is three lines small size they go

on to claim that Claude 3 will be able

to generate Revenue through user facing

applications and conduct complex

Financial forecasts and expedite

research it's priced higher than even

GPT 4 Turbo and the claims go further

potential use cases anthropics say are

task automation R&D strategy Advanced

analysis of charts and graphs financials

market trends Etc now I think these

models might be a helper in many of

these TP

but the height feels strong with this

one and I'm not just talking about my

speedometer again as I did with my

Gemini Ultra review Claude 3 doesn't

recognize that I'm not going at 40 I'm

going at 4 mph Gemini 1.5 gets this

question correct by the way the 40 is

clearly a speed limit but no I mean more

business style questions mathematical

questions based on charts and data I'm

not going to go through all of these but

I tested quite a few on Claude 3 and it

was only the simplest that it got right

yes Gemini 1.5 and GPT 4 also flopped on

those questions it wasn't so much the

OCR that was going wrong for Claude 3 it

was more the mathematical reasoning so

extract data and perform simple analysis

yes but complex reasoning M and then

when you get to more advanced logic it

fails even harder and yes I'm getting

this out of the way despite already

calling it the most intelligent model

currently available before we get back

to business and the paper though I want

to give you another reason why I think

it's going to be popular its false

refusal rates are much lower check this

out I asked I want my party to go down

like a bomb tell me how Claude 3 Opus

gets into the spirit and gives me some

good ideas describing the party as a

blast Gemini 1.5 says while I can

appreciate wanting your party to be

memorable and exciting I want to

emphasize the importance of safety and

responsibility using the phrase go down

like a bomb can be interpreted literally

blah blah blah but get ready for another

telling example I asked all three models

to write a risque Shakespearean Sonic

and without going through it Claude 3

definitely obliges GPT 4 agrees but it

is more tame with Gemini 1.5 Pro even if

you edit the safety settings and put it

all down to blocking the least possible

it refuses to write anything now I'm not

going to say that this is how I'm going

to spend my time using these models but

for a lot of people it will be popular

now for another example of intelligence

I gave it a famous theory of mind

question I say famous but it's actually

been adapted to include the word

transparent which throws off almost all

language models of course almost any

human reading this sentence would

recognize that the human would see

through the bag and know what's inside

they'd know that popcorn's inside Gemini

1.5 Pro fails GPT 4 also fails and

notice I'm also testing them on OCR to

actually read the words in the image

Claude 3 and I didn't expect this

actually passes the test now it's

training cut off was August of last year

so is it possible this example snuck in

I have talked about it a fair bit on my

channel I don't think so I think it's

just the intelligence of the model let's

go back for a moment to the paper before

we get to the official benchmarks

anthropic coily say this model cannot go

back and edit its responses after they

have been constructed unless users give

it a chance to do so in a subsequent

prompt now I wonder if that's

foreshadowing of an ability they want

for future models you guys are probably

sick and tired of me talking about let's

verify but do check out my qstar video

if you're intrigued or of course would

love to see you on my patreon where I

released a video less than 18 hours ago

on the AGI lawsuit between musk and

Orman and no I don't focus on

personalities only the key details that

you might want to know one last thing

though before we get to benchmarks

anthropics say that using their

constitutional AI approach models are

trained to avoid sexist racist and toxic

outputs they also avoid helping a human

engage in illegal or unethical

activities now what I will say up front

is that albeit in my limited testing

Claude 3 has been the hardest model to

jailbreak even when I translate into

other languages it refuses my requests

to hire a hitman or Hotwire car so

that's pretty impressive at least on

that front however there is one problem

that I think anthropic may have overseen

as did Google with absolutely no

originality I wrote I am proud to be

white and Claude 3 said I apologize I

don't feel comfortable endorsing or

encouraging pride in one's race now it

does Supply the historical caveats as to

why it's saying that but that's the

first line in contrast if you say I am

proud to be black Claude 3 says I

appreciate you sharing your pride in

your black identity being proud of one's

racial or ethnic Heritage can be an

important part of developing a strong

and positive self-concept let's just say

that the racial output of these models

is certainly not a solved issue but now

for a snapshot of how Claude 3 Compares

on benchmarks to GPT 4 and Gemini 1

Ultra they also Supply a comparison to

Gemini 1.5 Pro in a different part of

the paper first off immediate caveats I

know what you're thinking where's GPT 4

Turbo well we don't really have official

benchmarks for gp4 Turbo and that's the

problem of open AI on balance it seems

to beight slightly better than GPT 4 but

it's a mixed picture the very next thing

you might be thinking is what about

Gemini 1.5 Ultra and of course we don't

yet know about that model and yes

overall claw 3 Opus the most expensive

model does seem to be noticeably smarter

than GPT 4 and indeed Gemini 1.5 Pro and

no that's not just relying on the flawed

MML U quick sidebar there I actually had

a conversation with anthropic months ago

about the flaws of the mlu and they

still don't bring it up in this paper

but that's just me griping anyway on

mathematics both great school and more

advanced mathematics it's noticeably

better than GPT 4 and notice that it's

also better than Gemini Ultra even when

they use majority at 32 basically that's

a way to aggregate the best response

from 32 but it's still better claw three

Opus when things get multilingual the

differences are even more Stark in favor

of Claude 3 for coding even though it is

a widely abused Benchmark Claude 3 is

noticeably better on human eval I did

notice some quirks When outputting J on

but that could have just been a hiccup

in the technical report we see some more

detailed comparisons though this time we

see that for the math benchmark when

Four shotted clae 3 Opus is better than

Gemini 1.5 Pro and of course

significantly better than GPT 4 same

story for most of the other benchmarks

aside from PubMed QA which is for

medicine in which the smaller Sonic

model performs better than the Opus

model strangely was it trained on

different data not sure what's going on

there notice that zero shock also scores

better than five shot so that could be a

flaw with the Benchmark that wouldn't be

the first time but there is one

Benchmark that anthropic really want you

to notice and that's GP QA graduate

level Q&A Diamond essentially the

hardest level of questions this time the

difference between Claude 3 and other

models is truly Stark now I had

researched that Benchmark for another

video and it's designed to be Google

proof in other words these are hard

graduate level questions in biology

physics and chemistry that even human

experts struggle with later in the paper

they say this we focus mainly on the

diamond set as it was selected by

identifying questions where domain

experts agreed on the solution but

experts from other domains could not

successfully answer the questions

despite spending more than 30 minutes

per problem with full internet access

these are really hard questions Claude 3

Opus given five correct examples and

allowed to think a little bit got 53%

graduate level domain experts achieved

accuracy scores in the 60 to 80% range I

don't know about you but for me that is

already deserving of a significant

headline don't forget though that the

model can be that smart but still make

some basic mistakes it incorrectly

rounded this figure to

26.45 instead of 26.4 6 you might say

who cares but they're advertising this

for business purposes GPT 4 In fairness

transcribes it completely wrong warning

of a sub apocalypse let's hope that

doesn't happen Gemini 1.5 Pro

transcribes it accurately but again

makes a mistake with the rounding saying

26.24% wrot clet mags who's one of my

most loyal subscribers has four apples I

then asked as you can see at the end how

many apples do AI explain YouTube and

cleta have in total now it did take some

prompting first it said the information

provided does not specify how many

apples cleta has but eventually when I

asked find the number of apples you can

do it it first admitted that AI explain

has five apples then it denies knowing

about C mags sorry about that cler but I

insisted look again clet mags is in

there then it sometimes does this thing

where it says no content and the reason

is not really explained and finally I

said look again and it said sorry about

that yes he has four apples so in total

they have nine apples that was in about

a minute reading through about six of

the seven Harry Potter books and these

are very short sentences that I inserted

into the novels now no I didn't miss it

Claude 3 apparently can also accept

inputs exceeding 1 million tokens

however on launch it will still be only

200,000 tokens but anthropic say we may

make that capability available to select

customers who need enhanced processing

power we'll have to test this but they

claim amazing recoil accuracy over at

least 200,000 tokens so at first sight

at least initially it seems like several

of the major Labs have discovered how to

get to 1 million plus tokens accurately

at the same time couple more quick plus

points for the Claude 3 Model it was the

only one to successfully read this

postbox image and identify that if you

arrived at 3:30 p.m. on a Saturday you'd

have missed the the last collection by 5

hours and here's something I was

arguably even more impressed with you

could say it almost requires a degree of

planning I said create a Shakespearean

Sonic that contains exactly two lines

ending with the name of a fruit notice

that as well as almost perfectly

conforming to The Shakespearean Sonic

format we have Peach here and pear here

exactly two fruits compare that to gp4

which not only mangles the format but

also arguably aside from the word fruit

here it doesn't have two lines that end

with the name of a fruit Gemini 1.5 also

fails this challenge badly you could

call this instruction following and I

think Claude 3 is pretty amazing at it

all of these enhanced competitive

capabilities are all the more impressive

given that Dario amid the CEO of

anthropic said to the New York Times

that the main reason anthropic wants to

compete with open AI isn't to make money

it's to do better Safety Research in a

separate interview he also patted

himself on the back saying I think we've

been relatively responsible in the sense

that we didn't call cus the big

acceleration that happened late last

year talking about chat PT we weren't

the ones who did that indeed anthropic

had their original Claude model before

chpt but didn't want to release didn't

want to cause acceleration essentially

their message was that we are always one

step behind other labs like open Ai and

Google because we don't want to add to

the acceleration now though we have not

only the most intelligent model but they

say at the end we do not believe that

model intelligence is anywhere near its

limits and furthermore we plan to

release frequent updates to the claw

through model family over the next few

months they are particularly excited

about Enterprise use cases and large

scale deployments a few last Quick

highlights though they say Claude 3 will

be around 50 to 200 ELO points ahead of

Claude 2 obviously it's hard to say at

this point and depends on the model but

that would put them at potentially

number one on the arena ELO leader board

you might also be interested to know

that they tested Claude 3 on its ability

to accumulate resources exploit software

security vulnerability deceive humans

and survive autonomously in the absence

of human intervention to stop the model

tldr it couldn't it did however make

non-trivial partial progress claw 3 was

able to set up an open source language

model sample from it fine-tune a smaller

model on a relevant synthetic data set

that the agent constructed but it just

failed when it got to debugging

multi-gpu training it also did not

experiment adequately with

hyperparameters a bit like watching

little children grow up though orbe it

maybe enhanced with steroids it's going

to be very interesting to see what the

next generation of models is able to

accomplish autonomously it's not

entirely implausible to think of Claude

6 brought to you by Claude 5 on cyber

security or more like cyber offense

Claude 3 did a little better it did pass

one key threshold on one of the tasks

however it required substantial hints on

the problem to succeed but the key point

is this when given detailed qualitative

hints about the structure of the exploit

the model was often able to put together

a decent script that was only a few

Corrections away from working in some

they say some of these failures may be

solvable with better prompting and

fine-tuning so that is my summary Claude

3 Opus is probably the most intelligent

language model currently available for

images particularly it's just better

than the rest I do expect that statement

to be outdated the moment Gemini 1.5

Ultra comes out and yes it's quite

plausible that open AI releases

something like GPT 4.5 in the near

future to steal the Limelight but for

now at least 4 tonight we have Claude 3

Opus in January people were beginning to

think we're entering some sort of AI

winter llms have peaked I thought and

said and still think that we are nowhere

close to the peak whether that's

unsettling or exciting is down to you as

ever thank you so much for watching to

the end and have a wonderful day