The Hidden Life of Embeddings: Linus Lee

Summary

In this inaugural AI engineer conference talk, L from Notion shares their journey with AI, focusing on the power of embeddings and latent spaces in language models. With a background in experimenting with traditional NLP and language models, L delves into how embeddings offer a more direct control over steering language models, akin to navigating with precision rather than guesswork. Through practical examples, including manipulating text and image embeddings, L demonstrates the potential for creating expressive interfaces and the ability to interpret and control models more intuitively. The talk highlights recent advancements, introduces open-source models for experimentation, and envisions a future where interfaces allow deeper interaction and understanding of AI's underlying mechanisms.

Takeaways

• 😃 L from Notion discussed the power and potential of embeddings in AI at the inaugural AI engineer conference.
• 👨‍💻 Before joining Notion, L worked independently on language models and traditional NLP techniques like TF-IDF and BM25 to create innovative reading and writing interfaces.
• 🚀 Notion AI was launched around November 2022, introducing features like AI autofill and translation, with more innovations on the way.
• 👥 Notion is actively seeking AI engineers, product engineers, and machine learning engineers to expand their team.
• 🚨 L highlighted the challenge of steering language models effectively, likening it to controlling a car with a pool noodle from the back seat.
• 🔍 Latent spaces within embedding models offer a more direct layer of control by allowing inspection and manipulation of model internals, beyond just using prompts.
• 💬 By exploring embeddings and latent spaces, L demonstrates how to build more expressive interfaces and achieve a deeper understanding of what models 'see' in their inputs.
• 📈 L showcased practical examples of manipulating embeddings for text and images to alter content length, sentiment, and even blend concepts in creative ways.
• 🔧 The talk introduced custom models based on T5 architecture, fine-tuned and adapted for unique tasks like embedding recovery and model interoperability.
• 📦 L announced the availability of their models on Hugging Face, encouraging others to experiment with them and share findings.

Q & A

What is the main focus of the talk given by the speaker at the inaugural AI engineer conference?

The main focus of the talk is on embeddings and latent spaces of models, particularly how they can be used to control and interpret AI models more directly.

What is the speaker's background and their work prior to the conference?

The speaker, L, has worked on AI at Notion for the last year or so, and before that, did a lot of independent work experimenting with language models, traditional NLP techniques like TF-IDF and BM25, and embedding models to build interfaces for reading and writing.

What are some of the new features launched in Notion AI since its public beta in November 2022?

Since the public beta in November 2022, Notion AI has steadily launched new features including AI autofill inside databases and translation.

Why does the speaker compare prompting language models to steering a car with a pool noodle?

The speaker uses this analogy to illustrate the indirect and somewhat imprecise control one has over language models when using just prompts, highlighting the challenges in steering the model's output directly.

What are latent spaces, and why are they important according to the speaker?

Latent spaces are high-dimensional vector spaces inside models, including embedding models and activation spaces. They are important because they potentially encode salient features of the input data, offering a more direct layer of control and understanding of the model's operations.

How does the speaker propose to manipulate embeddings to achieve more direct control over model outputs?

The speaker suggests that by understanding and intervening in the latent spaces of models, specifically by manipulating embeddings, one can achieve a more direct control over the model's outputs and build more expressive interfaces.

Can you describe an example given by the speaker of how embeddings can be manipulated?

The speaker describes manipulating embeddings to vary text length while maintaining the topic, by identifying a direction in the embedding space that represents shorter text and then moving the embedding in that direction before decoding it back to text.

What innovative visual representation did the speaker showcase?

The speaker showcased a visual representation of their transformation from beginning to end, showcasing the transition from a photographic to a cartoon version of themselves.

What is the significance of the research work mentioned by the speaker, such as the AO World model paper?

The research work, including the AO World model paper, is significant because it represents groundbreaking efforts in understanding and manipulating latent spaces within models, offering new ways to interpret and control AI models' behaviors.

How does the speaker view the future of human interfaces to knowledge in the context of AI?

The speaker is optimistic about the future, believing that by digging deeper into what models are actually looking at and building interfaces around them, more human-centric interfaces to knowledge are possible, enhancing our understanding and interaction with AI.

Outlines

00:00

🗣 Introduction to Embeddings at the AI Conference

The speaker, identified as L, introduces themselves at the inaugural AI engineer conference, discussing their background in AI at Notion and independent work on language models and embeddings. They highlight Notion AI's development over the past year, including new features and the ongoing search for AI engineers. L emphasizes the importance of direct control over language models, comparing the usual indirect control to steering a car with a pool noodle. They propose that a closer examination of latent spaces within models could offer a more direct layer of control, making the interaction with models more effective.

05:01

🔍 Exploring Latent Spaces and Embeddings

L delves into the concept of latent spaces and embeddings, explaining how these can represent the most salient features of text or images. They demonstrate using an encoder-decoder model to manipulate embeddings, showing how slight alterations can produce variations in text outputs that maintain the general idea but vary in specifics. L introduces an interactive tool for exploring latent spaces, allowing for manipulation of text characteristics such as length and sentiment. They also highlight the potential of embedding models to understand and modify text in meaningful ways, suggesting a more nuanced control over model outputs.

10:03

🔬 Advanced Techniques with Embeddings

L showcases advanced techniques for working with embeddings, including adapting models to understand embeddings from different sources, such as OpenAI's models. They present experiments with image embeddings, demonstrating interpolation between images and manipulation based on textual descriptions to alter image characteristics. L emphasizes the surprising effectiveness of these methods in blending and modifying content meaningfully, suggesting that even complex features like proper nouns and themes can be manipulated through embedding space arithmetic. This section underscores the potential for creative and precise model control through embeddings.

15:05

🛠 Tools, Models, and Future Directions

In the concluding section, L discusses the technical details of the models used in their demonstrations, including a custom text model based on T5 and an image model capable of converting clip embeddings back to text or images. They announce the availability of these models on Hugging Face, encouraging others to experiment and share results. L reflects on the broader implications of their work, suggesting that understanding and manipulating model representations can lead to more intuitive and human-friendly interfaces for interacting with AI. The talk ends with an invitation to explore these tools further, emphasizing the ongoing exploration of AI's capabilities.

Keywords

💡Embeddings

Embeddings are a key concept in natural language processing and machine learning, representing words, sentences, or documents as vectors in a high-dimensional space. These vectors capture semantic information, meaning similar words or texts are closer in this space. In the video, embeddings are discussed as a crucial tool for understanding and manipulating latent spaces in AI models, particularly for tasks like steering language models, and building interfaces for reading and writing. The speaker illustrates how embeddings can encapsulate the most salient features of text or images and can be manipulated to alter the output of generative models, showing examples of changing text length or sentiment by adjusting embeddings.

💡Latent Spaces

Latent spaces refer to the abstract, high-dimensional spaces where embeddings reside. They encapsulate the underlying patterns and features learned by a model from the data. In the context of the video, latent spaces are explored as areas within AI models that, when understood and manipulated, can provide more direct control over the model's output. The speaker uses the metaphor of steering a car with a pool noodle to describe the challenge of controlling models through prompts alone and suggests that intervening in latent spaces might offer a more precise mechanism for directing model behavior.

💡Notion AI

Notion AI is mentioned as a product developed by the speaker's company, which integrates artificial intelligence into the Notion workspace. The video marks almost a year since its launch, highlighting its role in introducing new features like AI autofill and translation within databases. This context situates the speaker's work and interests in practical applications of AI to enhance productivity tools, underlining the ongoing efforts to refine AI capabilities and user interfaces in software products.

💡Steering Language Models

Steering language models refers to the process of guiding the generation of text by a language model towards desired outcomes. The speaker likens it to steering a car with a pool noodle from the back seat, illustrating the indirect and somewhat imprecise control users have when using prompts. This concept is central to the discussion on finding more effective ways to influence model behavior, particularly through the manipulation of latent spaces and embeddings, aiming for a closer layer of control over the generative processes.

💡Vector Spaces

Vector spaces are mathematical constructs where vectors reside, and in the context of AI, they represent the domain of embeddings. These spaces allow for operations like addition, subtraction, and scaling, facilitating the manipulation of embeddings. The video discusses how embeddings in vector spaces can encode information about text or images and how moving or adjusting these vectors can change the information represented, such as altering text length or sentiment, showcasing the potential for direct interaction with model outputs through vector space manipulation.

💡Encoder-Decoder Models

Encoder-decoder models are a type of neural network architecture used for a variety of tasks like machine translation, text summarization, and more. They work by encoding input data into an abstract representation (embedding) and then decoding it to produce output in the desired form. The speaker describes using an encoder to transform text into embeddings and a decoder to revert embeddings back into text, demonstrating this architecture's utility in exploring and manipulating latent spaces for creative and controlled generation of content.

💡Semantic Blur

Semantic blur refers to the loss of specificity or detail in generated text when embeddings are manipulated, such as by adding noise or blurring the vector representation. The speaker uses this concept to show how varying the precision of embeddings affects the clarity of the generated text, illustrating the relationship between the manipulation of embeddings in latent spaces and the resulting output quality. This highlights the challenges and possibilities of tuning AI models for desired outcomes through direct manipulation of embeddings.

💡Fine-tuning

Fine-tuning is a machine learning technique where a pre-trained model is further trained or adjusted on a new, often smaller, dataset to adapt to specific tasks or improve performance on particular metrics. In the video, fine-tuning is mentioned in the context of adapting an un-embedding model to interpret embeddings from different models, demonstrating how models can be refined to enhance compatibility and functionality across different embedding spaces, thereby extending their applicability and effectiveness.

💡Generative Models

Generative models are AI models capable of generating new data samples that resemble the training data. These include language models for text generation and diffusion models for image generation. The speaker discusses using generative models to explore latent spaces and manipulate embeddings to control generation processes, such as changing the style or sentiment of text and creating transitions between images. This showcases the creative potential of generative models when combined with an understanding of latent spaces.

💡Interpretable Representations

Interpretable representations refer to model outputs or internal representations that can be easily understood and manipulated by humans. The speaker emphasizes the importance of creating models whose latent spaces encode information in ways that are controllable and understandable, allowing for direct manipulation to achieve desired outcomes. This is linked to the broader goal of making AI models more transparent and user-friendly, enabling more intuitive interaction and customization by end-users.

Highlights

Introduction of the speaker, L, discussing their work on AI at Notion and experience with language models and embedding models.

Mention of Notion AI's progress since its public beta launch in November 2022, highlighting new features like AI autofill and translation.

Discussion on the challenge of steering language models effectively, comparing it to steering a car with a pool noodle.

Introduction to latent spaces and their significance in understanding and manipulating AI model behaviors.

Explanation of embedding models, illustrating how they encapsulate salient features of texts or images in high-dimensional vector spaces.

Demonstration of how modifying embeddings can alter generated content, showcasing the control over text length and sentiment.

Introduction to a custom AI tool that allows interactive manipulation of embeddings to explore latent space.

Showcasing the potential of embeddings to interpret and control AI output by varying and mixing embeddings.

Discussion on the ability to project text attributes in embedding space, revealing how models perceive different texts.

Demonstration of embedding space manipulation to blend different textual or visual inputs seamlessly.

Explanation of using linear adapters to adapt models for reading out text from different embedding spaces.

Availability of the speaker's custom models and tools for public use on Hugging Face, encouraging community experimentation.

Introduction to an image model based on CLIP embeddings, facilitating manipulations and interpolations in image space.

Highlighting the interdisciplinary research and recent developments in understanding and leveraging latent spaces.

Emphasis on the practicality of making AI models' internal processes more observable and interactive for deeper insights.

Concluding thoughts on the future of human interfaces to knowledge, inspired by interactive and manipulable AI models.

Transcripts

[Music]

everyone I'm L I'm here to talk about

embeddings um grateful to be here at the

inaugural a engineer conference who

learned something new

today yeah um before I talk about that a

little about myself if you don't know me

already I am lus I work on AI at notion

uh for the last year or so before that I

did a lot of independent work

prototyping experimenting with trying

out different things with language

models with traditional LP things like

tfidf bm25 to build interesting

interfaces for reading and writing in

particular I worked a lot with embedding

models and Laten spaces of models which

is what I'll be I'll be talking about

today but before I do that I want to

take a moment to say it's been almost

year since notion launched notion AI our

public beta was first announced in

around November 2022 so as we get close

to a year we we've uh been steadily

launching new and interesting features

inside notion AI from November we have

ai autofill inside databases translation

and things coming soon though not today

so keep an eye on the space and

obviously we're hiring just like

everybody else here we're looking for AI

Engineers product Engineers machine

learning Engineers to tackle the full

gamut of problems that people have been

talking about today agents tool use

evaluations data uh training and all the

interface stuff that that we'll see

today and tomorrow so if you're

interested please grab me and uh and

we'll have a little

chat now it wouldn't be Al us talk

without talking about lat in spaces so

let's talk about it um the the one of

the problems that I find always

motivated by is the problem of steering

language models and I always say that

prompting language models feels a lot

like you're steering a car from the back

seat with a pool noodle like yes

technically you have some control over

the motion of the vehicle so like

there's some connection but like you're

not really in the driver's seat the

control isn't really there it's not

really direct there's like three layers

of indirection between you and what the

vehicle is doing and that to me

trying to prompt a model especially

smaller more efficient models that we

can use for production with just tokens

just prompts feels a lot like uh there's

too many lers of indirection and as even

though models are getting better at

understanding prompts I think there's

always going to be this fundamental

barrier between indirect kind of control

of models with just prompts and and

getting with the model getting the model

to do what what we want them to do and

so perhaps we can get a closer layer of

control a more direct layer of control

by looking inside the model which is

where we look at latent spaces latent

spaces arise I think uh most famously

inside embedding models if you embed

some pieces of text that Vector of 1536

numbers or 1024 numbers is inside a high

dimensional Vector space that's a latent

space but also you can look at Vector at

the latent spaces inside activation

spaces of models inside token embeddings

inside image models and then obviously

other model architectures like Auto

encoders today we're going to be looking

at embeding embedding models but I think

a lot of the general takeaways apply to

other models and I think there's a lot

of fascinating research work happening

inside other models as well well when

you look at an embedding you kind of see

this right you see like rows and rows of

numbers if you ever debugged some kind

of an embedding Pipeline and you print

out the embedding you can kind of tell

it that's like a thousand numbers but

it's just looking at like a matrix uh

screen of numbers ring down but in

theory there's a lot of information

actually packed inside those embeddings

if you get an embedding of a piece of

text or image these latent spaces these

embeddings represent in theory the most

Salient features of the text or the

image that the model is using to lower

its loss or do its task and so maybe can

disentangle some meaningful attributes

or features out of these embeddings if

we can look at them a little more

closely and and and interpret them a

little better maybe we can build more

expressive interfaces that let them

control the model by interfering or

intervening inside the

model another way to say that is that

emedic show embeddings show us what the

model sees in a sample of input so maybe

we can read out what it sees and try to

understand better what what the model's

doing and maybe we can even control the

embedding intermediate activations to

see uh what the model can generate

so let's uh let's see some of

that so some of this some of you might

have seen before but I promise there's

some new stuff at the end so hang tight

so here are some sentence that I have

it's a sentence about this novel one of

my favorite novels named diaspora it's a

science fiction novel by Greg Egan that

explores the evolution and existence of

postum artificial intelligences uh

something to do with alien civilizations

and the questioning the nature of

reality and Consciousness which you

might be doing a lot given all the

things that are happening and so I have

I have I've trained this model that can

uh generate some embeddings out of this

text so if I hit this enter it's going

to give us an embedding but it's a it's

an embedding of length 248 and so it's

it's quite large but it's just a row of

numbers right but then I have a A decod

or half of this model they can take this

embedding and try to reconstruct the

original input that may have produced

this embedding so in this case it took

the original sentence there's some Vari

you can tell it's not exactly the same L

maybe but it's mostly reconstructed the

original sentence including specific

details like the title of the book and

so so we have an encoder that's going

from text to embedding and a decoder

that's going from embedding back to text

and now we can start to do things with

the embedding to vary it a little bit

and and see what the decoder might see

if we if we make some mod modifications

to the embedding so here I've uh try to

kind of blur the embedding and sample

some points around the embedding with

this this blur radius and you can see

the text generated from those blurry

embeddings they're a little off like

this is not the correct title the

title's kind of gone here um the the it

still kept the name Greg but it's it's a

different person and so there's kind of

a blur a semantic blur that's happened

here but this is kind of boring this is

not really useful what's a little more

useful is trying to actually manipulate

things in more meaningful directions so

now we have the same taste with text and

now here I have a bunch of controls so

maybe I want to find a direction in this

embedding space here computed a

direction where if you push an embedding

in that direction that's going to

represent a shorter piece of text of

roughly the same topic and so I picked

this direction and I hit go and uh it'll

try to move push the embedding of this

text in that direction and decode them

out and you can tell they're a little

bit shorter um if I push it a little bit

further even so now I'm taking that

shorter Direction and moving a little

farther along it and and sampling

generating text out of those embeddings

again and they're they're even a little

bit shorter but they've they've still

kept the general kind of idea General

topic and uh with that kind of building

block you can build really interesting

interfaces like for example I can I can

plop this piece of text down here and

maybe I want to generate a couple of a

couple of sort of shorter versions um so

this is like a little bit shorter this

is even even more short but uh maybe

maybe I like this version so I'm going

to clone this over here and I'm going to

make this the sentiment of the sentence

a little more

negative and you can start to explore

the lat space of of latent space of this

embedding model this language model by

actually moving around in a kind of spal

canvas interface which is is kind of

interesting another thing you can do

with this kind of end model is now that

we have a vague sense that there are

specific directions in this space that

mean specific things we can start to

more directly look at a text and ask the

model hey where does this piece of text

lie along your length direction or along

your negative sentiment Direction so uh

this is the original text that we've

been playing with it's pretty objective

like a Wikipedia style piece of text

here I've asked CH PT to take the

original text and uh make it sound a lot

more pessimistic so uh things like the

futile quest for meaning and plunging

deeper into the abys of nihilism and uh

and if I embed both of these what I'm

asking the model to do here is embed

both of these things in the embedding

space of the model and then project

those embeddings down onto each of these

directions so one way to read this table

is that um this default piece of text is

at this point in this negative Direction

which by itself doesn't mean anything

but it's clearly less than this so this

piece of text is much further along the

negative sentiment axis inside this

model when you look at other properties

like how how much um of the artistic

kind of topic does it talk about is

roughly the same the length is roughly

the same um the the maybe the the

negative sentiment text is a bit more

elaborate in its its vocabulary and so

you can start to project these things

into these meaningful directions and say

what are the features that the model's

what are the attributes that the model's

um finding in the in the text that we're

feeding it

uh another way you could test out some

of these ideas is by mixing embeddings

and so here I'm going to embed both of

these pieces of text this one's the one

that we've been playing with this one is

a beginning of a short story that I

wrote once it's about this like town in

the Mediterranean uh Coast that's calm

and and a little bit old and um both of

these have been embedded and so I'm

going to say just this is a 2,000

dimensional embedding I'm going to say

give me a new embedding that's just the

first thousand or so Dimensions from the

one embedding and then take the last

thousand dimensions of the second

embedding and just like slam them

together and have this new embedding and

navely you wouldn't really think that

that would amount too much that would be

kind of gibberish but actually if you

generate some samples from it um you can

tell you can see in a bit uh you get a

sentence that's kind of a semantic mix

of both you have structural similarities

to to both of those things like you have

this structure where there's a quoted

kind of title of a book in the beginning

there's topical similarities there

structur there's uh punctuation

similarities tone similarities and so

this is an example of inter

in lat space the last I have I have uh

you may have seen on Twitter is um about

okay I have this un edding model and I

have kind of an un edding model that

works pretty well can I use this un

embedding model and somehow fine-tune it

or otherwise adapt it so you can read

out text from other kinds of embedding

spaces so this is the same sentence

we've been using but now when I hit this

run button it's going to embed this text

not using my embedding model but using

open AI Ada text AA 2 uh and then

there's a linear adapter that I've

trained so that my decoder model can

read out not from my embedding model but

from open ai's embedding space so I'm

going to embed it it's going to try to

decode out the text from given just the

open AI

embedding and

uh you can see okay it's not it's not as

perfect but there's a surprising amount

of detail that we've recovered out of

just the embedding with no reference to

the source text so you can see this

proper noun diaspora it's like

surprisingly still in there um this

feature where there's a a quoted title

of a book is in there it's roughly about

the same topic things like the Rogue AI

um sometimes when I when I rerun this

there's also references to the author

where the name is roughly correct so

even surprising uh things features like

proper nouns punctuation things like the

quotes uh General structure and topic

obviously those are recoverable given

just the embedding because the the kind

of deta amount of detail that these high

high capacity embedding spaces

have but not only can you do this in the

text space you can also do this in image

space

so um here I have a I have a few

prepared files um let's start with

me and um for dumb technical reasons I

have to put two of them in and then

let's try to interpolate in this image

space so this is now using Clip Clips

embedding space I'm going to try to

generate say like six images in between

uh me and the the notion avatar version

of me the cartoon version of me if that

back end will warm up cold starting

models is sometimes difficult

uh there we go so now it's Shing six

images bridging kind of interpolating

between the photographic version of me

and the cartoon version of me uh and uh

again it's not perfect but you can see

here it's on the left it's it's quite

photographic and then as you move

further down this interpolation you're

you're seeing uh more kind of cartoony

features appear here and uh it's

actually quite a surprisingly smooth

transition um another thing you can do

on top of this is you can you can do

text manipul as because aul text image

model and can

say time let's add some text I'm going

to subtract the vector for a photo of a

smiling man and instead I'm going to add

the vector for a photo of a very sad

crying

man uh and then I'll I'll embid these

pieces of text and I empirically I find

that for text I have to be a little more

careful so I'm going to dial down how

much of those vectors I'm adding and

subtracting and then generate six again

and it's taking a

bit okay I'm really

sad and you can do you can do even more

fun things like you can try to add like

here's a photo of a beach I'm going to

try to add some some beach in this this

time maybe just generate four for a sake

of time

um or maybe there's a bug and it won't

let me

generate so in all these demos that I've

done both in the text and image domain

okay the beach didn't quite survive the

uh the lat and space arithmetic but uh

in all these all these demos the only

thing I'm doing is calculating vectors

calculating embeddings for examples and

uh and embedding them and just adding

them together and some with some

normalization and it's surprising that

just by doing that you can try to

manipulate um interesting features in

text and images and with this you can

also do things like at style and subject

at the same time you can uh this is a

cool image that I thought I generated

when I my first demo and then you can

also do some pretty smooth transitions

between landscape

imagery

so that's interesting um in all these

prototypes one principle that I've tried

to reiterate to myself is that often

times when you're studying this very

complex sophisticated um models you

don't really necessarily have the

ability to look inside and say okay

what's what what's happening not even

get an intuitive understand even getting

an intuitive understanding of what is a

model thinking what is a model looking

at can be difficult and I think these

are some of the ways that I've tried to

render the these invisible parts of the

model a little bit more visible to let

you more a little bit more directly

observe exactly what the model is um the

representations the model is operating

in and sometimes you can also take those

and directly interact or let directly

interact with the representations to

explore what these faes

represent um and I think there's there's

a ton of interesting pretty

groundbreaking research that's happening

here um on the left here is the AO World

model paper which is fascinating neurons

on a hyack and then on the right is very

very recent I had to add this in last

minute because it's super relevant um in

a lot of these examples I've calculated

these feature Dimensions by just giving

examples and calculating centroids

between them but here anthropics new

work along with other work from

conjecture and other labs have found

unsupervised ways to try to

automatically discover these Dimensions

inside models so that's super exciting

and in general I'm really excited to see

latent spaces that appear to encode you

know by some definition interpretable

controllable representations of the

models input and output um I want to

talk a little bit in the last few

minutes about the models that I'm using

the text model is a custom model um I

won't talk I won't go into too much

detail but it's fine-tuned from uh T5

checkpoint with as a d using Auto

encoder it's an incoder decoder

Transformer with some modifications that

you can see in the code um so here's a

general Transformer incoder on the left

decoder on the right I have some pooling

layers to get an embedding this is like

a normal T5 embedding model stack and

then on the right I have this special

kind of gated layer that pulls from the

embedding to uh to decode from the

embedding you can look at the code it's

a little more uh easy to understand but

we take this model and we can adapt it

to other models as well as you saw with

the open AI embedding recovery and so on

the left is the normal trading regime

where you have an encoder you get an

embedding and you try to reconstruct the

text on the right we just just train

this linear adapter layer to go from

embedding of a different model to then

reconstruct the text with a normal

decoder um and today I'm excited to

share that these models that I've been

deing with uh that you may have asked

about before are open on hogging face so

you can go download them and try them

out now uh these are the links on the

left is the hug face models and then

there's a collab notebook that lets you

get started really quickly try to do

things like interpolation and

interpretation of these features and so

if you find any interesting results with

these please let me know and and if you

have any questions also reach out and

I'll be able to help you out the image

model that I was using at the end wasow

brain Carlo excited to see Korea um

stepping out there in this model this

model is an uncp model which is trained

kind of like the way that Dolly 2 was

trained as a diffusion model that's

trained to invert clip embeddings so go

from clip embeddings of images back to

text and that lets us do similar things

as the text model that that we

used in all these prototyping I think a

general principle if if you have one

takeaway from this talk it's that when

you're working with with these really

complex models and kind of inscrutable

pieces of data if you can get something

into a thing that feels like it can fit

in your hand that you can play with that

you can concretely see and observe and

interact with can be directly

manipulated visualized all these things

all the tools and prototypes that you

can build around these things I think

help us get a deeper understanding of

how these models work and how we can

improve them

um and in that way I think models

language models and image models

generation generative models are a

really interesting laboratory for

knowledge for studying how these how

these um different kinds of modalities

can be represented and um Brett Victor

said the purpose of a thinking medium is

to bring thought outside the head to

represent these Concepts in a form that

can be seen with the senses and

manipulate with the body in this way the

medium is literally an extension of the

mind and I think that's a great poetic

way to kind of describe the the

philosophy that I've approached a lot of

my prototyping with so if you follow

some of these principles and try to dig

deeper in what the models are actually

looking at build interfaces around them

I think more human interfaces to

knowledge are possible I'm really

excited to see that

future thank

[Applause]

you