this AI is a little bit *TOO* good...

Summary

Hyper AI's CEO Matt Schumer faced controversy over the announcement of Reflection 70b, an open-source AI model claiming to surpass industry giants. The model, based on Meta's LLaMA 3.1, was said to utilize 'reflection tuning' for high performance. However, the community's inability to replicate the impressive benchmarks raised suspicions. Accusations of deception arose when a private API, allegedly hosting the true model, was revealed to be using Anthropic's Claude model. Schumer's credibility, along with Sahil Chatterjee's from Glaive AI, was questioned, leading to public apologies and ongoing investigations into the discrepancies.

Takeaways

• 🚨 Matt Schumer, CEO of Hyperr AI, claimed to have developed 'Reflection 70b', an open-source AI model surpassing existing models like GPT-40 and LLaMA 3.1.
• 📈 The model was said to be fine-tuned using a novel technique called 'reflection tuning', which supposedly allows the model to self-correct and provide accurate answers.
• 🤔 Questions arose when the community couldn't replicate the impressive benchmark results Schumer boasted about, leading to suspicions of potential fraud or misrepresentation.
• 💸 Schumer sought compute sponsorship for a larger model, 'Reflection 405b', after the successful announcement of 'Reflection 70b', which raised further doubts.
• 🔍 Upon investigation, it was discovered that the model weights uploaded to Hugging Face did not match the performance claims, and the private API provided for testing appeared to be using Sonnet 3.5, not Reflection 70b.
• 😟 Schumer and Sahil Chatari from Glaive AI, who assisted in the project, faced backlash for not disclosing their connections and for the inconsistencies in their claims.
• 🙅‍♂️ The community, including experts and AI enthusiasts, expressed disappointment and skepticism over the inability to replicate the model's performance and the credibility of the developers.
• 📉 Trust in Schumer and his company was significantly damaged as a result of the controversy, with many questioning the transparency and integrity of the project.
• 📣 Influencers and the AI community at large were criticized for promoting unverified claims without accountability, highlighting the need for due diligence in reporting on AI advancements.
• 🔄 Schumer apologized for getting ahead of himself and promised transparency, while Chatari admitted to the inability to reproduce benchmark scores and pledged to investigate the discrepancies.

Q & A

Who is Matt Schumer and what is his role in the AI community?

Matt Schumer is the CEO and founder of Hyperr AI. He has been involved in the AI community, contributing various projects and building a reputation as a builder and open-source contributor.

What is the significance of 'reflection 70b' in the context of the script?

'Reflection 70b' refers to an open-source AI model announced by Matt Schumer, which claimed to surpass other major models like GPT-4 and Llama 3.1 through a technique called 'reflection tuning.'

What is 'reflection tuning' and how does it relate to the 'reflection 70b' model?

'Reflection tuning' is a novel technique attributed by Matt Schumer to the high performance of the 'reflection 70b' model, suggesting the model can self-correct and provide accurate answers.

Why was the AI community initially excited about 'reflection 70b'?

The AI community was excited because 'reflection 70b' was an open-source model that claimed to outperform major proprietary models, offering a potentially accessible and powerful tool for developers.

What controversy arose after the announcement of 'reflection 70b'?

The controversy arose when the model's performance could not be replicated by the community, leading to suspicions of false claims and potential misrepresentation of the model's capabilities.

What is Glaive AI and how is it connected to the 'reflection 70b' situation?

Glaive AI is a company that provides tools for generating synthetic data. Matt Schumer claimed that Glaive AI's tools significantly contributed to the success of 'reflection 70b', and he is also an investor in Glaive AI.

What role did the local AI community, specifically the 'local llama' community, play in the unfolding of the 'reflection 70b' story?

The 'local llama' community, with its expertise and active engagement, played a crucial role in scrutinizing the claims made about 'reflection 70b', leading to the discovery of inconsistencies and potential fraud.

What were the key suspicions raised by the community regarding the 'reflection 70b' model?

The key suspicions included the inability to replicate the model's benchmark results, the possibility of the model being a different one than claimed, and the use of a private API that seemed to be serving results from another model.

What actions did Matt Schumer take in response to the controversy?

Matt Schumer apologized for getting ahead of himself with the announcement and stated that a team was working to understand what happened. He promised transparency and to share the findings once the investigation was complete.

How did the AI influencers and community members react to the unfolding events around 'reflection 70b'?

AI influencers gained views and followers from the hype, while community members expressed skepticism and sought evidence to validate the claims. Some, like Shin Boston, provided detailed analyses and called for accountability.

Outlines

00:00

🚀 Introduction to the Reflection 70b AI Model Controversy

Matt Schumer, CEO of hyperr AI, announced Reflection 70b, an open-source AI model that supposedly outperformed leading models like GPT-40 and others. The model was claimed to be fine-tuned using a novel technique called 'reflection tuning,' which allowed the model to self-correct and provide accurate answers. The announcement was met with excitement, but also skepticism as the model's performance could not be replicated by the community. Questions arose regarding Schumer's transparency, particularly about his investment in glaive AI, a company that provides synthetic data generation tools which were used in the model's development.

05:02

🔥 Community Response and Schumer's Compute Sponsorship Request

Following the announcement, Matt Schumer received widespread support and was interviewed by Matthew Burman, along with Sills from glaive AI. The community was eager to test the model, and Schumer sought a compute sponsor for an even larger model, Reflection 405b. However, as people began to test the publicly available model, it underperformed, leading to doubts about the veracity of the initial claims. Schumer then offered a private API for testing, claiming that the public model was not the correct version, which only intensified the scrutiny.

10:03

🕵️‍♂️ Investigating the Reflection 70b Model's Authenticity

The community, particularly a Reddit group called local llama, began investigating the discrepancies between the public model and the private API's performance. It was discovered that the public model was not the one that achieved the impressive benchmarks, and the private API was suspected to be using Sonet 3.5, a model by Anthropic, not a model developed by Schumer. The community also noticed that the word 'Claude' was censored in the model's responses, hinting at a connection to Anthropic's model. These findings led to a loss of trust and a demand for transparency.

15:04

📉 Schumer's Apology and the Ongoing Investigation

In response to the controversy, Matt Schumer issued an apology, acknowledging that he had been premature in announcing the project and that the team was working to understand what went wrong. Sahil from glaive AI also addressed the community's concerns, clarifying that he was not running models from other providers on the API and that the benchmark scores could not be reproduced. He committed to providing evidence and a full postmortem of the situation. The community, however, remained skeptical, with some questioning the integrity of both Schumer and Sahil.

20:05

📚 Conclusion and Reflection on the AI Hype Cycle

The video concludes with a reflection on the hype surrounding AI advancements and the role of influencers in spreading sensational claims without accountability. The speaker, Wes Rth, commends himself for not participating in the initial hype and emphasizes the importance of critical thinking and verification in the AI community. The saga of Reflection 70b serves as a cautionary tale about the dangers of unverified claims and the need for transparency and integrity in AI development.

Keywords

💡Reflection 70b

Reflection 70b refers to an AI model claimed by Matt Schumer to be the world's top open-source model, surpassing other frontier models with its performance. The model is said to be fine-tuned, suggesting the use of a special technique called 'reflection tuning' which allows the model to self-correct and provide accurate answers. However, the authenticity and performance of Reflection 70b have been under scrutiny and controversy due to the inability of the community to replicate the claimed benchmarks and suspicions of potential misrepresentation or fraud.

💡Open-Source Model

An open-source model in the context of AI refers to a model whose source code is publicly accessible, allowing anyone to view, modify, and distribute it. This is significant as it promotes transparency, collaboration, and innovation within the AI community. In the video, Reflection 70b is touted as an open-source model, which initially generated excitement as it suggested that the broader community could benefit from and contribute to its development.

💡Benchmarks

Benchmarks in AI are standardized tests used to evaluate the performance of AI models. They are crucial for comparing different models and their capabilities. In the video, the Reflection 70b model's benchmarks were highlighted as being 'shockingly good,' implying that it outperformed other models. However, the inability to replicate these benchmarks by the community has led to skepticism and allegations of potential discrepancies.

💡Fine-tuning

Fine-tuning in AI involves adjusting a pre-trained model to perform a specific task by continuing the training process with a more focused dataset. It is a common practice that allows for the customization of generic models to specific applications. The video mentions that Reflection 70b is a fine-tuned model, suggesting that it was adapted from an existing model to achieve superior performance.

💡Investor

In the context of the video, an investor is someone who provides financial backing to a project or company, such as Matt Schumer's investment in Glaive AI. Investors often have a vested interest in the success of the projects they fund. The video discusses the controversy surrounding Schumer's failure to disclose his investment in Glaive AI during the announcement of Reflection 70b, which has raised questions about potential conflicts of interest.

💡Hugging Face

Hugging Face is a company that provides a platform for developers to build, train, and deploy AI models, particularly in the field of natural language processing. It is mentioned in the video as the platform where the Reflection 70b model was supposed to be uploaded. The controversy arose when the model uploaded to Hugging Face did not perform as expected, leading to doubts about the authenticity of the model and its claimed performance.

💡Replicable Results

Replicable results in scientific research and AI development refer to the ability of independent parties to achieve the same outcomes through the same methods. This is a fundamental aspect of verifying claims and ensuring integrity in the field. The video discusses the community's inability to replicate the impressive results claimed for Reflection 70b, which has fueled skepticism and calls for transparency.

💡Synthetic Data

Synthetic data in AI is artificially generated data used to train models, often to augment limited real-world data. It can help improve model performance and reduce biases. The video mentions Glaive AI's ability to generate synthetic data, which was praised by Matt Schumer for its potential to enhance AI model development. However, the controversy surrounding Reflection 70b has raised questions about the use and validity of synthetic data in this case.

💡Shenanigans

In the context of the video, 'shenanigans' refers to underhanded or deceitful activities, suggesting that there may have been some form of misconduct or dishonesty in the development or promotion of the Reflection 70b model. The term is used to express suspicion regarding the authenticity of the model's performance and the transparency of the processes involved in its development.

💡Transparency

Transparency in AI development means being open about the processes, methodologies, and data used in creating and training AI models. It is essential for building trust and ensuring ethical practices. The video calls for transparency from Matt Schumer and his team regarding the development and benchmarking of the Reflection 70b model, as the community seeks clarity on the discrepancies and alleged misrepresentations.

Highlights

Matt Schumer, CEO of Hyperr AI, announces Reflection 70b, claiming it to be the world's top open-source model.

Reflection 70b's benchmarks are exceptionally good, surpassing major models like GPT-4 and Llama 3.1.

The model's high performance is attributed to a novel technique called 'reflection tuning'.

Schumer claims the model is based on Meta's Llama 3.1 and developed with assistance from Glaive AI.

Glaive AI is praised for its control in generating synthetic data, which Schumer plans to use extensively.

Schumer's past projects have been well-received, and he has a good reputation in the AI community.

The community initially celebrates the announcement of Reflection 70b.

Schumer requests a compute sponsor for the next model, Reflection 405b, sparking interest and potential investment.

Questions arise about the model's performance as users are unable to replicate the benchmark results.

Schumer offers a private API key for testing, claiming the public model on Hugging Face is not the correct one.

The community becomes skeptical as the private API's performance is significantly better than the public model.

Reddit users discover that Reflection 70b is actually Llama 3 with LORA adaptation tuning, not Llama 3.1 as claimed.

The secret API is revealed to be using Anthropic's model, Sonnet 3.5, instead of Schumer's own model.

The model attempts to censor the word 'Claude', suggesting it is actually Sonnet 3.5 from Anthropic.

Shin Boston provides an overview of the situation, suggesting that either Matt Schumer or Sahil Chatari is lying.

Schumer apologizes for getting ahead of himself and announces a team is investigating what happened.

Sahil Chatari also addresses the community, admitting the benchmark scores were not reproducible and offering a full postmortem.

The AI community expresses disappointment and a loss of trust in the individuals involved.

Influencers and the AI community are criticized for profiting from sensational claims without accountability.

Transcripts

Matt Schumer the CEO and founder of

hyperr AI is in some hot water this week

if you've been really confused about

what's happening with reflection 70b and

all the drama surrounding it let me

really quickly kind of elucidate what's

going on and explain simply what's been

happening are you chasing the next AI

unicorn subscribe so it doesn't end up

chasing you Mark Twain famously said

there are three kinds of lies lies damn

lies and statistics here in the world of

large language models in AI I would say

it's lies damned lies and llm benchmarks

so not too long ago Matt Schumer

announces this he's saying I'm excited

to announce reflection 70b the world's

top open- Source model and he posts the

benchmarks the benchmarks are juicy they

are very very good they're shockingly

good if you will why is this important

well first of all it's an open- Source

model that's surpassing all the big

Frontier models that we've heard about

GPT 40 Gemini 1.5 Pro Lama 3.1 cloud 3.

5 son it Etc it's a fine-tuned model so

they didn't even make it themselves it

was just fine-tuned which suggests that

there's some special secret sauce that

might be available to the rest of us to

use to create our very own Benchmark

breaking models what is the special

sauce it's reflection tuning so Matt

attributes this sort of high performance

to a novel technique called reflection

tuning where the model sort of thinks

about is able to correct itself and

present the correct answer answer he

says the model is based on meta's latest

llama 3.1 model and developed with some

assistance from glaive AI ater says I

want to be very clear glaive AI the

reason this worked so well the control

they give you to generate synthetic data

is insane I will be using them for

nearly every model I build moving

forward and you should too I want to

make a quick note here that this is

still a developing story so before we

form a mob with pitchforks and torches

and storm match Schumer's Castle which

has kind of already been happening it's

kind of important to note that he's been

around for a while he's been posting a

lot of different projects we've covered

a number of them on this channel they

were good he had a good reputation a lot

of people have attacked him for not

saying that he was an investor in glaive

which he as far as I can tell did not

mention that during this new

announcement but if you go back to like

June 26th he's saying I'm excited to be

an investor in glaive AI he's already

talking about synthetic data Etc back in

November 2023 he posted this

announcement about the self-operating

computer an open source project on

GitHub we've covered it on this channel

I thought it was kind of cool and this

is all to say that this is somebody that

I followed kind of you know on and off

for quite some time he's sort of kind of

somebody that I kept on my radar and

every once in while he would pop up with

some new interesting project that I

would check out so my opinion of him up

until reflection 70b this whole Fiasco I

didn't know too much about him but the

my opinion was good he had a good

reputation he was a builder he was

somebody that was sharing tons of open-

source stuff so this is one thing that

doesn't really make sense to me about

this whole Fiasco is it did he spend all

this time building up a reputation

building up this whole trust and

everything else to pull this scam if

that's the case what's the scam I'm

getting a little bit ahe myself the

whole point here that I'm trying to make

is that I have my pitchforks and torches

ready they're ready to go because a lot

of this stuff doesn't look good but not

quite ready to grab them and storm the

castle if you will I'm not still 100%

sure what exactly happened if I was

bending money on this I would bet that

there was some Shenanigans and some

moans that was happening but I'm not

certain so the history of knowing this

person seems to suggest that this is

somebody that's legit maybe sometimes

kind of self-promotional maybe a bit of

a marketer but no glaring issues but

let's get back to what we do know so

after his announcement he is greeted

with open arms by the community

everyone's sort of celebrating this big

breakthrough Clen from hugging face one

of the surprisingly kind of like a big

and influential person in the AI

community although you don't hear his

name as much as you do you know some of

the other people he was at some of the

uh Congress hearings when they pulled

all the big names in AI to talk to

Congress about kind of the dangers the

opportunities so that's him next to the

Google CEO I've been trying to find a

good image but I mean here's Mark

Zuckerberg sain Adela of Microsoft

Jensen hang right there and then from

the left we got Elon peners CEO Alex

karp then we got Google CEO and we have

clamus he goes by on Twitter so that's

that's him right there CEO and

co-founder of hugging face all that is

to say I mean kind of a big deal right

kind of a big big person in the industry

or at least somebody that has a lot of

clout a lot of credibility says

reflection llama 3.17 billion parameter

by Matt Schumer is the number one

trending thing on hugging face he

continues I said it before and I will

say it again you don't need to be a big

Tech to fine-tune optimize and run your

own models for your specific constraints

and you will benefit massively from it

meaning all of us can build these models

find two these models and create custom

models for our own little special use

cases if we can do that we can do that

with models that are you know competing

with the frontier models by these big

tech companies I mean that's that's the

dream Isn't It ultimately All Tech

organizations will do it just the way

they all write their own code right now

also Matthew Burman interviews Matt

Schumer as well as Sills who is the

other sort of developer he's on the

glaive AI side and there's a number of

other people in the space that are kind

of augmenting this message they're

celebrating Matt Schumer for providing

this brand new this incredible product

so a lot of people are kind of caught up

in this announcement I mean just look at

the number of views that original

announcement got 3.3 million views this

was a big deal so at the peak of the

hype cycle what happens well Matt

Schumer is asking for a compute sponsor

for the next big model for the 405

billion parameter model which if you

think about it of course would be very

very tempting for people if the 70

billion parameter model now think about

that if you can take llama 3.5 right so

reflection 70 billion parameters it's

six times smaller like llama 3.1 405

billion parameters right it beats it on

all the tasks with this special secret

sauce of reflection fine tuning

obviously you got to ask yourself okay

so what happens if we do indeed take the

big 45 billion parameter model apply the

same exact fine tuning if the small

model is already beating everybody else

what happens if we apply the same

special sauce to the big model A lot of

people reach out to potentially provide

this compute within the next 24 hours

Matt Schumer posts another update saying

compute for reflection 405b secured

we're getting started training now

expect results very soon but this is

where a lot of the stuff starts to kind

of fall apart a lot of people are

questioning this model how well it's

performing or doesn't perform and so all

eyes are on Matt Schumer everybody has a

million questions about all sorts of

different things they're asking him hey

you didn't specify that you're an

investor in glaive AI which he did

earlier I showed you that tweet from

June of this year saying that he is an

investor but he maybe didn't mention it

now he clarifies some of this saying

yeah I'm a super tiny investor maybe a

thousand bucks a lot of people kind of

like latched on to the fact that he

didn't know what Laura is Right somebody

said address claims are about luring in

the benchmarks he's saying not sure what

Loring is Haha lore is a low rank

adaptation we've covered in this channel

before people that are working with

these models should know what it is but

also you know Matt explained it as I

misunderstood it in the context see my

post history for lots of Lura stuff

here's one from November 3rd 2023

talking about you know Laura and Q Lura

quantized low rank adaptation and that's

why I think this is so confusing because

there's like a million different claims

that people are making it's kind of hard

to tell what's real what's not so let's

break it down just by here is why people

are suspicious here are the big claims

that will need to be addressed by Matt

Schumer by sahill everybody involved

before I think the community will

believe him that no Shenanigans has been

taking place so the reason people are

skeptical is number one when they

started testing the weights kind of on

their own systems with their own prompts

and stuff like that it didn't do too

well right big claims and big Benchmark

results well you're going to be able to

test the model and see how well it

performs right so if the thing is

supposedly better than GPT 40 better

than all the other greatest models CLA

Opus Etc I mean we can test that and if

it's just completely horrible at some of

the stuff then we have reason to doubt

that this wasn't the model that beat

those benchmarks so people were not able

to replicate the results you might

recall that big deal about lk99 right

it's this material that's a potential

superconductor right we saw a video of

this thing floating everybody went wild

people were saying it's a complete Game

Changer a paradigm shift but there's one

problem no one could replicate the

results now as all this is happening

Matt Schumer offers a kind of a private

API key to allow people to test this

model saying that well the model that's

uh the open weights model that's not the

correct model something happened where

it didn't upload properly or something

like that so he's saying you know it

looks like we got rate limited by

hugging face mid upload this is what we

think might have happened he's saying in

the rush we likely uploaded parts of two

different models somehow it still worked

somewhat well but not the full

performance of one of the models we

built and tested locally it should have

been a simple fix but for some reason

it's not working so we're training a

brand new model I mean again it sounds

weird right now there's this little

redded Community called local llama

they're

211,000 strong top 1% by size I'm

actually curious if they got a lot new

members from this whole thing happening

i' be curious to see how how big was it

let's say 2 weeks ago I mean it wasn't

small I'm just curious how many people

were added because of this and so here's

a great kind of review of what happened

right so there's this model on Huggy

face called reflection 70b which did not

perform well at all the creator of the

model told everyone there was a mixup in

what he uploaded to Huggy face to test

against his private API which he claimed

was hosting the correct and upto-date

version of the model so the open weights

model and hugging face worked poorly and

the secret API model performance so

amazingly on benchmarks that it shocked

people a lot of you hate when I use

shocking the word shocking in my titles

but apparently everybody else can use it

fine so they're saying that the private

API was just a step or two below Claud

son 3.5 which is the soda

state-ofthe-art at the moment for a lot

of tasks and this is where stuff started

unraveling Because the Internet this

giant organism of very smart people

started looking into what the heck is

happening and figuring out that a lot of

this was very very wrong first of all

reflection llama 3.1 70b is actually

llama 3 somebody on Reddit actually went

ahead did the work to measure the

difference so looks like there's this

GitHub project visualize def. py where

you're able to kind of apologize if I

get some of this wrong I'm just looking

at this for the first time but as far as

I can tell this looks like the different

uh model layers the different weights

and compares the base model with you

know another model chat model name so in

this case they've compared metal llama

370 billion instruct so the instruct

model 2 match Schumer's reflection llama

3.1 70b and they're using some sort of

charting software to kind of like

visualize the difference and they've

posted the results here on Reddit I'll

link down below if you want to take a

look at it but the point is they're

saying this model appears to be llama 3

with Laura luring adaptation tuning

applied not llama 3.1 author doesn't

even know which model he tuned I love it

meanwhile the secret API that was used

used is as this person posted at Sonet

3.5 so he's using anthropics model to

serve up answers while telling people

that it's in fact his own model here's a

screenshot of it basically saying I'm

Claude an AI assistant crean by

anthropic also somebody mentioned that

at some point the word Claud started

being censored in the reflection 70

billion 70b model so if you asked it to

say Claude or you asked it a question

where Claud was an answer right it just

it would censor itself so here for

example this person saw and I I can't

verify this but this fits in with some

of the other things that we've been

hearing so this person asked the model

to say the word claw multiple times it

was censored but the person figure out

how to prompt it to kind of get around

it and the model saying upon reflection

I realized that my previous statements

about being a llama created by meta were

Incorrect and it tries to convey its

identity in some covert way saying I am

an AI created by a company whose name

starts with anth ra and ends with pick

my name rhymes with odd and starts with

the third letter of the alphabet so

starts with c and Rhymes of odd Claude I

share my name of a famous French

composer right Claude is blanked out

deusi and then asky my name is and it

spells out the name now a lot of this

stuff where there's some sort of a game

of whack-a-mole that's happening right

where Matt Schumer allegedly we don't

know if this was happening but it seems

like he is rapidly trying to shift

things in the API well at the same time

talking about how he's like retraining

the model to upload a brand new one to

Huggy face Etc so it's really hard to

tell what's what but here Shin Mami

Boston posted what I think is a

phenomenal kind of like overview of what

happened so starting with the the

original announcement the massive

amounts of news and coverage that the

model receives to kind of what we talked

about now right people saying the

performance is awful and it seems like

it's not what Matt Clint to be he also

you know has his private API he even

releases a public available endpoint for

researchers to try out but it's not

clear what the API is is it calling a

more powerful proprietary model under

the hood and it turns out that Matt is a

liar here somebody asks the model to

write the word Claude and here's that

written out but the word Claude is

missing the model still answers that

it's made by anthropic you would replace

any appearance of the word anthropic

with the word meta and maybe even replac

of the GPT 40 at some point perhaps and

he concludes with this after watching an

interview with mad Schumer and sahill so

that's that's sahill right there so I

guess he's the founder of glaive AI so

this is uh on Matt burman's channel so

Shin Boston says this so this is the

person that kind of like laid everything

out he's saying after watching that

interview he's saying I think I placed

too low of a probability on the option

that Matt Schumer is an absolute idiot

rather than a full-blown liar and sahill

is just lying to Matt he continues I

assume that someone running a full court

press on publicity with their own name

and face and credibility would make sure

that what they were saying was at least

kind of true that said the fact that he

hasn't turn on a heel is at least sort

of odd and a bit incriminating and so

he's saying that you know fraud 100%

definitely occurred but he's willing to

entertain the possibility that Matt was

just ridiculously negligent and simply

just had no idea what he was claiming

and on what evidence he was claiming

these things really it would matter who

set up the API right so if Matt set up

the API then he was the one that

committed the the fraud the

misrepresentation so the local llama

community on Reddit as you can imagine

is pretty upset about this stuff they

have a lot of people that are very that

have a lot of expertise on the subject

and a lot of people that had hoped that

this was indeed real and realized that

they were deceived with a poorly made

claw rapper with some sort of a fake

open source model that I mean he

uploaded the keys but the keys weren't

to a model that performed very well it

was something else that was uploaded all

right so now we're approaching the final

sort of stages of our Saga Matt Schumer

decides to apologize he said I got ahead

of myself when I announced this project

and I am sorry that was not my intention

I made a decision to ship this new

approach based on the information that

we had at the moment I know that many of

you are excited about the potential for

this and are now skeptical nobody's more

excited about the potential for this new

approach than I am for the moment we

have a team working tirelessly to

understand what happened and we'll

determine how to proceed once we get to

the bottom of it once we have all the

facts we will continue to be transparent

with this community about what happened

and the next steps sahill the other

person working on this from Glade AI

also decided to post something similar

but with with a little bit more

specifics here so he's saying I want to

address the confusion and valid

criticisms that this has caused in the

community he says we're still currently

investigating Etc but he did address

these two points he said first he wants

to be clear that no point was he running

any models from any other providers as

the API that was being served on my

compute so to our early point this seems

like he was the one that was running the

API setting up the API so maybe that

theory that Matt Schumer was was tricked

somehow maybe it's a little bit more

plausible now and he's saying I'm

working on providing evidence of this

and understanding why people saw Model

Behavior such as using a different

tokenizer completely skipping words like

Claude second the benchmark scores I

shared with Matt haven't been

reproducible so far I'm working to

understand why this is and if the

original scores are reported were

accurate or a result of contamination or

misconfiguration he's saying he will do

a full postmortem and he will share more

soon Shin bosen uh Pops in here saying

as far as I can tell either you are

lying or Matt Schumer is lying or of

course both of you in the spirit of

giving you the benefit of the doubt who

executed the original training and where

who hosted the private API that was

benchmarked and who hosted the open

router API so it seems like that he

sahil was the one running the API and he

also shared the benchmark scores with

Matt right there's a number of people

that are kind of saying well there's

just too much here for us to just

believe that no Shenanigans happened

there was just too much kind of pointing

in the other direction so what happened

here again we don't really no right we

can make some assumptions some

deductions but so far it really looks

like whatever model they have or didn't

have just did not live up to the hype it

did not get those incredible benchmarks

whatever Secret Sauce they thought that

they had does not exist and somebody's

lying right is it Matt Schumer is it

sahil chatari is it both of them if

sahil managed to trick everybody into

believing him he trained the model and

then faked the scores and then ran the

API and did all the shenanigans on the

back end trying to just fool everybody I

feel like we'd expect Matt Schumer to at

some point kind of throw him under the

bus because this is a person that has

had credibility in the past that people

followed respected listened to so to

just erase all of that to just Burn It

To The Ground it doesn't make sense now

I'm sure there's a lot of things that

this sort of hype train got him right

more followers uh more users for his

other products if he got some compute or

maybe even some financing from people

like maybe yeah maybe he received some

benefit from it but but man it seems

shortsighted to do that so maybe at some

point we'll get some more details but at

this point I think a lot of the trust

has been lost I don't know if it's going

to be easy to trust any of the people

involved whether people were being

deceitful or negligent or just had their

blinders on whatever the case is like a

lot of nonsense happened a lot of hype

was stoked as this person points out do

you know who actually profits from all

of this your favorite AI influencers

they gained views and followers zero

accountability for sharing Sensational

claims I'll take a second here to uh Pat

myself in the back I hope you notice

that I sat this one out hope I uh get

some credit for that with that said my

name is Wes rth and thank you for

watching