Generative Interfaces Beyond Chat // Linus Lee // LLMs in Production Conference

Summary

Linus discusses the evolution of generative interfaces beyond chatbots, emphasizing the importance of context in conversational AI. He explores how interfaces can be improved by integrating more intuitive interactions like point-and-select, and suggests using constraints to balance user-friendliness with the power of language models. Linus also touches on the need for interfaces to provide guidance and feedback loops to enhance user experience.

Takeaways

• 💡 **Chat-style interfaces are useful but not the end-all**: Recognizing that while chat interfaces are valuable, there's room for evolution towards more intuitive and flexible interfaces.
• 🌐 **Importance of context in conversations**: Emphasizing the need for language models to leverage context effectively to interpret user intent accurately.
• 🔍 **Incremental evolution of interfaces**: Discussing how to move from chat-repeat style interactions to more engaging and intuitive interfaces.
• 🛠️ **UI design and interaction**: Linus' background in UI/UX and creative tools informs his perspective on building interfaces that balance flexibility and ease of use.
• 🔗 **Conversation anatomy**: Breaking down the structure of a conversation into context, intent, internal monologue, and action to better understand language model interactions.
• 🎯 **Point and select interfaces**: Introducing the concept of using selections and pointing to clarify context and direct model actions within an interface.
• 📋 **Co-inhabiting workspaces**: The idea that interfaces should allow models to 'live' within the same workspace as the user to enhance interaction.
• 🔑 **Using names to reference objects**: Suggesting that naming elements within an interface can help users interact more naturally with language models.
• 🔄 **Iterative feedback loops**: Encouraging the design of interfaces that allow for quick feedback and iteration to improve user experience.
• 🔑 **Predictive interfaces**: The potential of predictive models to suggest actions and make interfaces more intuitive, reducing the 'blank page' problem.

Q & A

What is the main topic of the discussion led by Linus?

The main topic of the discussion is the evolution of generative interfaces beyond chat, exploring how to create more intuitive and flexible interfaces that balance the capabilities of language models with ease of use.

What does Linus suggest as a problem with current chat-style interfaces?

Linus suggests that current chat-style interfaces often start with zero context, which leads to difficulties in prompting and a need for users to specify every detail of their intent explicitly.

What is the importance of context in conversation according to Linus?

Context is crucial as it determines how everything said in a conversation is interpreted. Linus emphasizes that context is often undervalued in chat interfaces, leading to less effective communication with language models.

What is the role of 'intent' in a conversation as described by Linus?

In a conversation, 'intent' is what the speaker implies, which could be explicit, direct, or implicit. It's a key component that language models need to interpret correctly, often relying heavily on context.

How does Linus propose to improve the interaction with language models?

Linus proposes using point and select interfaces, where users can directly interact with objects in an application, and language models can observe these actions to better understand user intent and provide relevant responses.

What are the different ways Linus discusses for language models to observe user actions?

Linus discusses several ways including an omniscience model where the model has access to all application states, 'call by name' for environments with named objects, direct mention of objects by users, and contextual actions similar to right-click menus.

Why does Linus think that adding constraints to interfaces can be beneficial?

Adding constraints can make interfaces more intuitive and easier to use without sacrificing the power of language models. It provides a structured way for users to interact with models, reducing the 'blank page syndrome' and learning curve.

What is the concept of 'call by name' as mentioned by Linus?

'Call by name' is a method where users can name different parts of an application, like artboards in a design app, and then refer to them directly in their prompts to the language model for more contextual interactions.

What does Linus mean by 'closing feedback loops' in interface design?

Closing feedback loops refers to minimizing the time and effort it takes for users to evaluate system responses and iterate on their inputs. This can be achieved by providing multiple options, interactive components, or predictive actions.

What is the 'Holy Grail' of user interface design according to Linus?

The 'Holy Grail' of user interface design is to balance an intuitive UI that's easy to learn and progressively understand with the flexibility to handle complex tasks, without sacrificing the power of language models.

How does Linus view the future of interfaces that incorporate AI and language models?

Linus views the future of interfaces as incorporating AI and language models that co-inhabit the user's workspace, take advantage of rich shared context, predict happy path actions, and allow for a fast iteration loop, possibly including interactive components.

Outlines

00:00

🌟 Introduction to Generative Interfaces

Linus introduces the topic of generative interfaces beyond chat, acknowledging the usefulness of chat interfaces but also recognizing their limitations. He discusses the need to evolve beyond text-based interactions and integrate language models with more intuitive interfaces. Linus shares his background in UI design and his work as a research engineer at Notion, focusing on building creative and productivity tools. He outlines the key areas of conversation: understanding the parts of a conversation important for language models, the role of context and selection in dialogue, and the benefits of adding constraints to interfaces to balance ease of use with the power of language models.

05:00

🗣️ Anatomy of a Conversation

Linus breaks down the components of a conversation, emphasizing the importance of context in communication. He explains how context shapes the interpretation of language and gives examples of how it varies in different scenarios. He also discusses the role of intent in a conversation, which can be explicit or implicit, and how it's crucial for language models to understand this intent. Linus mentions 'Chain of Thought' as an internal monologue for the model to interpret the user's intent within the given context. He uses examples like Co-pilot in VS Code to illustrate how context is already integrated in some applications, and how intent is clear when the user's environment is known.

10:02

🎯 Point and Select Interfaces

Linus explores the concept of 'point and select' interfaces, suggesting that they can help clarify context and direct the model's actions. He explains how pointing and selecting are natural actions in the physical world that can be translated into digital interactions. Linus proposes several ways for language models to observe user actions, such as the omniscience model where the model has access to all application states, 'call by name' for applications with named objects, direct mention of objects, and contextual actions similar to right-click menus. He argues that these methods can make conversational interfaces feel more integrated and intuitive.

15:03

🔄 Balancing Intuition and Flexibility

Linus discusses the goal of balancing intuitiveness and flexibility in user interface design. He suggests that while chat interfaces offer flexibility, they often lack intuitiveness, leading to a steep learning curve for users. He proposes that by adding constraints and suggestions, interfaces can become more intuitive without sacrificing the power of language models. Linus also talks about the importance of closing feedback loops in creative and productivity applications, allowing users to quickly iterate and refine their requests. He gives examples of how providing a range of options or interactive components can enhance the user experience.

20:03

🚀 Conclusion and Future Outlook

In conclusion, Linus summarizes the key ideas from the discussion, emphasizing the potential for dialogue interfaces built on language models to co-inhabit the user's workspace and take advantage of rich shared context. He suggests that interfaces should start with constrained actions but allow for advanced use through chat. Linus also reiterates the importance of speeding up the user's iteration loop through direct manipulation and interactive feedback. He expresses optimism about the future of intuitive and powerful conversational applications and encourages the exploration of predictive interfaces, given the advancements in language models.

Keywords

💡Generative Interfaces

Generative Interfaces refer to systems that can create content based on user input or interactions. In the video, this concept is explored in the context of evolving beyond traditional chat interfaces to more intuitive and flexible systems that can leverage the power of language models. An example mentioned is how generative interfaces could allow for more natural interactions with AI, similar to how humans interact in the real world.

💡Chat-style Interactions

Chat-style interactions are a form of dialogue where users communicate with a system by exchanging text back and forth, similar to a conversation. The video discusses how these are currently dominant but may not represent the ultimate interface for interacting with AI, suggesting the need for more advanced interfaces that can better utilize contextual information.

💡Context

Context, in the video, refers to the environmental or situational information that influences the interpretation of communication. It is crucial for understanding user intent in conversations with AI. The speaker argues that current chat interfaces often start with little context, which can lead to difficulties in prompting the AI effectively.

💡Intent

Intent in the video represents the purpose or goal behind a user's action or communication with an AI system. It can be explicit or implicit and is important for AI to interpret correctly. The script discusses how additional context can help in accurately interpreting intent, which is vital for the AI to provide the appropriate response or action.

💡Co-pilot

Co-pilot, as used in the script, refers to AI systems that assist users in performing tasks, often by providing suggestions or automating parts of the workflow. An example given is copilot chat inside VS Code, where the AI is aware of the user's current coding context and can generate responses or actions based on that context.

💡Point and Select

Point and Select is a user interaction pattern where users direct the AI's attention to a specific part of the interface by selecting or pointing at elements. The video suggests that this can make interactions more intuitive by allowing users to interact with the AI in a way that mimics natural human behavior, such as pointing at an object to refer to it.

💡Constraints

Constraints in the video refer to limitations or guidelines that are故意 introduced in an interface to make it more intuitive and easier to use. By adding constraints, interfaces can guide users towards common actions or happy paths, reducing the complexity of interactions with AI and making them more accessible.

💡Conversational Interfaces

Conversational Interfaces are systems that allow users to interact with technology using natural language in a conversational manner. The video discusses the evolution of these interfaces, suggesting that while they are powerful, they can also benefit from additional context and constraints to become more intuitive.

💡Intuition

Intuition, in the context of the video, refers to the natural, instinctive understanding that allows users to interact with an interface without needing extensive instruction. The speaker argues for the importance of creating interfaces that are intuitive, suggesting that they should guide users towards actions rather than presenting a blank page.

💡Feedback Loops

Feedback Loops are cycles of user action and system response that allow for continuous improvement and iteration. The video discusses the importance of closing feedback loops in creative and productivity applications, suggesting that interfaces should provide immediate, interactive feedback to facilitate user exploration and refinement of ideas.

💡Interactive Components

Interactive Components are elements of a user interface that respond to user interaction, such as sliders, buttons, or widgets. The video suggests that incorporating these components into conversational interfaces can provide users with more direct ways to explore options and receive feedback from the AI, enhancing the interaction experience.

Highlights

Introduction to generative interfaces beyond chat

The limitations of current chat-style interfaces

The importance of context in language model interactions

The role of intent in conversational interfaces

The concept of 'internal monologue' for language models

The action phase in conversational AI interfaces

Example of context usage in Co-pilot for VS Code

The need for more intuitive interfaces beyond chat

The idea of 'point and select' interfaces

Different ways for models to observe user actions

The concept of 'call by name' for object interaction

The potential of drag-and-drop for interface interaction

The role of contextual actions in interfaces

The importance of balancing intuitiveness and flexibility in UI design

The challenge of 'blank page syndrome' in chat interfaces

The idea of using predictive interfaces to aid user interaction

The concept of interactive components in chat interfaces

The goal of tightening feedback loops in creative applications

Final thoughts on balancing power and intuitiveness in interface design

Transcripts

alrighty uh there we go cool I don't

know or between between uh Harrison and

that song I don't know how to top that

but but I'll make my best attempt

um happy to be here to talk about

generative interfaces Beyond chat

uh I'm I'm Linus and I'll do my intro in

a bit but um where I want to start

today is I think we all generally have a

sense of

catch upd style chat is super useful

super valuable you can do a lot with

them but it's I think at this point we

all kind of accept that it's not the end

of the road right with these interfaces

and I had a I had a a tweet a while ago

where I was like you guys are telling me

that we're gonna invent literal super

intelligence and we're going to interact

with this thing by sending text back and

forth and and it's it's obvious that

it's not the end of the road but Chinese

here today most

um usages of language models and

interfaces that I've seen in production

are built on chat style interactions or

dialogue style turn-by-turn kind of

interactions and there's interesting

experiments on the fringes some of which

I'll hopefully mention later but Chad is

here today and so the leading question

that I want to spend our time talking

about today is given that Chatters where

we are and given the possibilities for

other things to come how do we

incrementally evolve what we have chat

repeat style chat forward to build more

interesting interfaces that balance the

flexibility of language models and the

power of language models with the ease

of use and the intuitiveness of some of

the other kinds of interfaces it can

build

so uh excited to talk about that a

little bit about me I think a lot about

UI design interfaces interaction design

Nai and I've I've spent a bunch of time

thinking about that in the context of

building creative tools and productivity

tools so it makes sense that I am

currently a notion number research

engineer at notion before that I spent a

couple years working independently also

pursuing these ideas building a lot of

prototypes uh it sounds like some of

which are going to be linked somewhere

in the chat

um and have worked on other kind of

productivity tool companies and apps

before

so

if I had to roadmap we're going to talk

about I think there's sort of three big

buckets first I want to just lay the

groundwork for how should we think about

conversations and dialogue what are the

parts of a conversation that we should

think about when we build language

models for conversations and then second

talk about specifically the idea of

context in a conversation and the

selection in a conversation which will

will come up and then third I want to

land on this idea of constraints and

adding constraints and the benefits that

they can have and how we can balance

adding constraints to make interfaces

more intuitive and more easy to use

without sacrifice and Power

so let's talk about conversations

let's say you and your buddy here are

about to talk about something and you

want to say something

even before you say anything at all the

communication channel has already opened

and kind of started because in any

conversation in any kind of like

linguistic act uh do you you start with

a shared context that context might be

uh your friend just pulled up with a

chair next to your office and you're

about to pay a program it might be

you're in a supermarket checking

something out it might be you're

collaborating with a co-worker it might

be your friend uh or a stranger walked

up to on the street that context

determines how everything that you say

and everything that your interlocutor

says back to you is interpreted and so I

think context is important and notably I

think in applications like child PT

context gets very very low billing uh

you basically start with zero context

and you have to embed all the context

that you want the model to use to

interpret your words in the prompt

itself which is where I think a lot of

the difficulties of prompting come from

so you start with some context

and then uh your the speaker will imply

some intent it could be explicit and

direct like hey can you pass me a glass

of water you could be a little more uh

it could be a little more implicit like

if I'm a construction worker I'm

building a house or I'm like assembling

a Lego kid I might say oh the blue brick

and that's not a complete thought but in

the context it can be interpreted to to

figure out exactly what you're looking

for or it could even be just like me

pointing at something right or or like

doing like one of these and then and

then that my partner in in the

conversation can interpret the intent

out of what I'm doing and that's that's

the speaker's role unless you have the

intent um especially important for

language models as giving the model time

to think I'm borrowing some I'm abusing

some reinforcement learning terminology

terminology here and calling this rule

out but uh people call this Chain of

Thought call this a scratch Pad some

internal monologue for the model or for

the recipient of this action to figure

out exactly what you mean and kind of do

the interpretation of your intent what

you said within the context that you

have and then once the model or the

recipient of the message is done

thinking there's some action the action

might be answering the question so maybe

just textual but more exciting and more

often I think we're seeing lots of apps

where the action is some combination of

a response back and

the action that the model is taking

whether in an application or integrating

with an API

so I think this is kind of where the

anatomy of a conversation if you really

break it down in a typical kind of

language model usage Style

let's let's take uh co-pilot does it as

an example this is a screenshot from

co-pilot next or copilot X chat all

these names are insane

um copilot chat inside vs code

this is one of those cases where there's

very clear you're already starting with

some context if you're building

something like this you wouldn't just

build the chat you would want the

language model to be aware of all the

context as much context you can get out

of this application so the context

includes things like what files you have

open uh or do you have a terminal open

is there what's the last a few commands

and the outputs that they got because

maybe the error message in the terminal

can inform what the model can do for the

user it includes it in things like what

line is the cursor on for the user or

what lines do they have selected because

selection is actually a really strong

signal for what the user is thinking

about and looking at it's kind of like

pointing but but on a screen or in a

text editor so you start with some

context

and there's the intent the intent is in

this case is write a set of unit test

functions for the selected code and you

can see that

in interfaces like this you really need

the context to interpret the intent

correctly and the more context you have

usually the better that interpretation

is going to be and then presumably there

are some some internal monologue

thinking for the model

um uh and then after that we we get the

models action back out

so

when we think of prompt these

conversational interfaces I think we

usually focus on the intent The Prompt

and then the completion the output the

action but there's there are these other

elements that I think we should also

think about

um the context in which the users uh

intenses being interpreted and then also

the internal monologue

so uh

this is a screenshot of Microsoft's

copilot for Excel I think this is an

interesting example of a really rich

valuable application where there is a

chat interface but where there's clearly

a lot more we could do

um so so in this case there is so much

context that the model can use to figure

out exactly what the user is doing and

maybe what they're trying to accomplish

um and if you had a chat sidebar like

this the sidebar sort of exists in a

totally different Universe uh than the

chat I mean the model can theoretically

look at everything that's in the

spreadsheet that the user has open but

in this case in this screenshot the user

is having to refer to specific columns

and specific part of this spreadsheet

verbally by naming things saying you

know the in this case it's the column

about uh the sales data or last month's

sales data why can't I just point in and

select and then say what about this

column right in the real world that's

kind of how we work if I want to refer

to something that's in front of me I'll

just point to it I'll just look at it

um

so in this case I think

having the chat agent be co-located or

like inhabited co-inhabiting the same

workspace that the the user is in I

think is a key part of how to make these

interfaces gel a little better

and without that I think these

conversational interfaces start to feel

more like the the command line interface

of generative AI where you're having to

specify increment every possible

information about your intent and your

action explicitly into the prompt rather

than being able to work more fluidly in

a kind of a shared working environment

so uh how do we how do we where do we go

from here well I think

I keep talking about this idea of

pointing and and using the context and

selecting things and so one really

powerful technique that I think we can

look for is using our hands using

selections and pointing

and when you point at things in the

context or when you select things there

are a few different ways I think for the

language model to observe what you're

pointing out or what you're doing

sort of in order from both grounded and

reality so that I think the most

interesting and kind of out there I

think we can start with uh yeah so point

and selection interfaces uh one way to

think of point and select is uh sort of

breaking down your action into nouns and

then verbs so what I mean by that is if

you're in a spreadsheet the noun might

be like the column that I want to

manipulate and you select the noun and

then the vertical bee I'm going to

filter it or I want to aggregate it or I

want to hide it or delete it or

duplicate it

um if you're in a writing app then now

it might be a single block in the

writing app like the title block or it

could be the entire page or it could be

like a new file

um in the real world this point and

select mechanic is sort of built into

every object in every material if I

wanna

take action on some object I have to

first like grab the thing and then do

something with it

um but in in chat style interfaces I

think it's less obvious

but this point in select mechanic is

also what makes the web great for a lot

of applications because there's existing

sort of materiality built into

everything on the web every bit of text

by default on the web is selectable you

can select anything you can copy paste

anything uh you can uh often drag and

drop files into web pages and so there's

all this like noun and Denver based

mechanic built into materials that you

use to build apps on the web

and uh in chat all of that kind of all

of those affordances around selecting

objects and then applying actions to

them are kind of gone and I think we

might we could think about how to bring

that back to uh chat style interfaces

and point and select I think are most

useful for helping clarify context and

focus the model on something you're

looking at or alternatively directing

the kind of action stream or the output

stream of the model so if you're in a

writing app you could say you know take

select something take summarize this bit

and then like put it here at the top of

the page or or you know make a new page

over over here and being able to point

point and select I think are useful for

directing the output as well

so there are a few ways that we can get

the model to observe what you're doing

or what you're pointing out the most

common one currently and I think the

most obvious one is sort of this what

I'm calling like the omniscience model

which is the model can look at

everything everywhere visible all the

time it just kind of knows the entire

state of the application but it's up to

the model to figure out what to query

and what to look at it's so it's

technically the context is technically

fully accessible but the model doesn't

know exactly what you want the model to

look at

next level up from that is what I'm

calling call by name which I think is

kind of interesting for certain types of

application especially kind of pro

applications where there's a lot of

flexibility and customization if you

have an application like like a design

app like figma or a sketch you could

imagine naming different artboards or

different panels and then being able to

app mention and say hey can you clean up

you know panel two or can you clean up

like the like timeline panel

so uh this only really makes sense for

environments where it makes sense to

name objects and refer to them by kind

of handles or names but if that's the

case then I think this is an interesting

way to incorporate context and be able

to directly point to things but with

your words using the names

there's also this kind of really

interesting uh interface that I I don't

think anybody's really seen in

production which is what I'm calling

literally mentioning something

um this in particular is a screenshot

from a paper from a project called

sikulu from I believe an MIT lab where

they uh had a programming language that

interleaved icons and images with a way

to program around the UI and you could

mention you could imagine an interface

where if I wanted to refer to a

paragraph I could start writing and

saying summarize and then I literally

drag and drop the paragraph into the

prompt box or if I wanted to transform

an image I could drag and drop an image

or or if I have to if I want to talk to

a person that's in my contact list I

could grab that person's icon and say

hey can you call this person and then

just drag and drop that image or that

object and so having support for these

rich objects inside the prompt box I

think is a really interesting

possibility

and then the last one is what I'm

calling contextual actions which is a

great example of this is uh like a right

click

so uh an example of the right click is

these context menus right so the left is

notion right is figma you grab an object

sort of metaphorically and then you can

see all the options that are available

to you all the things you might want to

do with that object in a lot of current

applications these are hard-coded in but

you could imagine using a language model

to say okay here's what here's the

object that the user has in their hands

or the most likely actions given the

full context of the application and

given maybe even their like history of

actions and what the title of the file

is and what they want to do what are the

most likely actions they might want to

do you could you could have the model

select from a list you could also have

the model generate possible trajectories

that the user might want to take and so

context menu I think is an interesting

way to reveal actions that you want the

user to take without having to force

them to to type the instruction up fully

another kind of context menu pattern is

this kind of thought driven programming

or autocomplete driven programming which

I think is the the analog of right click

but with text so if I'm typing in a text

editor or code editor and you hit like

dot like document.potty Dot and it'll

show me all the autocomplete options

this is kind of like saying I'm holding

this object in my hand what are all the

things that are accessible to me from

this or what are the actions that I can

take from this or uh in the the other

panel I have tab completion so I'm

working inside a terminal I have this

CLI in my hand what are the things that

I can do with it tell me the

possibilities and this is another way of

grabbing an object and then sort of

showing me what's possible and you can

imagine powering something like this

with a language model as well

and then lastly this is a slightly more

complex pattern but but where you if the

user selects an object you could uh

materialize an entire piece of UI like a

side panel or or a kind of an overlay so

on the left again is notion AI on the

right is is keynote uh which is what I

was using to make the stack and in

either case you select an object and

then you can see a whole host of options

for how you want to control it and this

gives the user a lot of extra power at

the cost of maybe not being obvious

exactly what the user wants to do or

what the user should take action on

so in all these cases we have this sort

of like noun and then verb like choose

the object and then what action you want

to take kind of pattern and that lets

the system constrain the action space

that the user might want to take and

maybe even come up with uh follow-ups or

suggestions and what the best actions

you could take are

given all of this given and given what

we talked about around the anatomy of a

conversation I think

and then when you look back at something

like Challenger PT charger PT is really

just about okay you have this little

tiny prompt box and you have to cram all

of the context all of the intent in

there and also everything that you want

the model to know about where you want

that model to take its action and that

that I think is a good place to start

but it is limiting and there are ways we

can expand out of it

um so one way to summarize the the sort

of ground that we've covered might be

that the Holy Grail or one powerful goal

of user interface design is to balance

um intuition building an intuitive UI

that's uh easy to learn and sort of

progressively understand but flexible

um and intuitive and flexibility I think

is the strength of language models

um

in chat style interfaces with chat cell

interfaces you have the full access to

the full capabilities of a model you can

ask it to do anything that the model

could possibly do including things like

use API and use tools and fully specify

like a programming language that you

want the model to use if you want and

that's the strength of falling back to

chat but by adding these constraints

where you start with something in your

hand and then try to recommend or

suggest or follow up and say given this

is what you're looking at given this is

the locus of your attention right now

here are the things that you can do

um and maybe predict some actions and

add some guardrails add some structure

to the instruction I think that's where

we can add sort of bring back the

intuitiveness of graphical user

interfaces without sacrificing the power

of language models

open-ended natural language interfaces I

think uh trades off too much of that

intuitiveness for for the flexibility

um so in an app like chachu PT you have

this blank page syndrome where the user

doesn't know exactly what they are

supposed to type in maybe they have a a

sense of maybe I want a summary or maybe

I want a conversation of a certain style

but there's no affordances in the UI to

give them hints of okay these are the

things that the model is good at these

are the ways you might not phrase the

answer none of this none of this exists

and so I think it adds a huge learning

curve and as detrimental to the ease of

Discovery and by bringing back some of

these graphical interfaces I think we

can improve that situation a bit

and then lastly since I'm closing in on

time

um I wanted to add one more note around

another goal of interface design often

which I think is closing feedback loops

particularly in Creative applications

and sometimes also in productivity

applications you want to try to tighten

the close the

um

tighten the the feedback loop between

the user attempting something and maybe

having something in their mind they want

to see and then looking at the results

evaluating the result and then figuring

out okay this is how I need to iterate

this is the fix that I need to apply to

get get the model to generate what I

want

and uh there are a few ways to do this

right one is instead of the model

generating one output if you're say

generating images instead of a model

generating one output it could generate

a range of outputs and that allows users

to pick okay these are these are maybe

four different ways of looking at this

answer for four different images that

you could generate this is one that I

like and then iterate on that again this

only really works if the output is easy

to evaluate if I ask the model to write

an essay and it gives me four different

essays it would be it would be pretty

difficult to use

um so going along with that idea I think

uh you want

um

if whenever possible you want to prefer

uh what I've heard referred to as like

people like to shop more than like to

create they like having a range of

options they can choose from and maybe

even like swipe left and right style

kind of do I like this do I not they

want to that kind of interface is easier

to use a more intuitive uh and more

engaging than here's a blank page tell

me exactly what you want and again

powering that kind of thing comes back

to okay what coming up with uh options

and predictions of actions and

suggestions that you is that you sort of

uh plan out for the user in case they

want

and then the user can make that

selection and then lastly

um I've seen some prototypes of this

this thing that I'm calling interactive

components what I'm referring to this

we're referring to by interactive

components is if you are in a chat kind

of interface and you ask a question and

instead of responding with a paragraph

of answer maybe I ask like uh what's the

weather in New York and instead of

responding with a paragraph of answer

the model says okay here's maybe maybe

it says the temperature tomorrow is 85

and then there's a little weather widget

with a slider for time or with with

buttons we're looking at precipitation

and these other things and the model can

synthesize maybe the model will be able

to synthesize little interactive

components little widgets on the Fly and

that again helps me close my feedback

loop by saying okay these are these

other options of information that I can

look at and I can really explore them

really directly without having to

re-prompt and retype my my queries

so uh bringing it all back I wanted to

close out with one of my one of my

favorite quotes from one of my favorite

papers essays when I'm thinking about

creative tools uh by

um

okay content in this essay called casual

creators

um and I I think this quote is great so

I'm just going to quote at length the

possibility space of creative tools and

what you can do the action space should

be narrow enough to exclude broken

artifacts like models that fall over or

break when when you're in a 3D printing

app but it should be broad enough to

contain surprising artifacts as well the

surprising quality of the artifacts

motivates the user to explore the

possibilities base in search of new

discoveries new use cases a motivation

which disappears if that space is too

uniform so again she's talking about

this balance of you want to constrain

just a bit just enough that the user

never gets stuck in that like blank page

state that there's always some option

that they can take or always some

suggested action that seems interesting

uh you want to preserve the power and

the flexibility and the sometimes

surprising quality of these language

models and I think that distracting that

balance is is sort of the primary

challenge of building interfaces for

these models

oh

something's happening

there we go okay uh uh so last slide

just to sum up I think five Big Ideas

that I want uh it'd be great you could

take away from this this conversation I

think good dialogue interface is built

on llms can have agents that co-inhabit

your workspace that are there and can

see what you're doing in its full detail

including where your attention is it

should take full advantage of the rich

shared context that you have with the

model

uh to interpret your actions so that you

don't have to cram everything into a

prompt I think these interfaces can lead

initially with constrained happy path

actions that you can use models and

other language models and other

predictive models to try to predict and

then if the user wants to do something

more advanced or different we can always

fall back to chat as an escape patch

because there's the power and the

flexibility in language models and then

lastly

whether you're building a chat interface

or something a little more direct

manipulation graphical uh it's I think

it's always good to think about how we

can speed up that iteration Loop

especially by forcing the user not not

forcing the user to type text but by

responding uh more directly with a with

a mouse or with a touchscreen for

closing that feedback loop

so with that I hope that was interesting

and useful and hope you can build some

some great conversational applications

uh given there's a extrude

wow wow I mean so many questions there's

so much stuff that's going through my

mind and I love the idea of how it's

like

you

you're helping guide people that is so

nice to think about instead of just

leaving this open space And then trying

to figure it out it's like hey can we

can we suggest things so that people can

figure it out with us as opposed to just

letting their imagination go wild and

then it may or may not turn out okay

yeah exactly I think

some there's some history of predictive

interfaces like this and I think uh in

the design World collectively our tastes

have been soured a bit on predictive

interfaces because the model that we've

used models that we've used in the past

have not been that good and so we

couldn't really predict that far and we

could really predict only simple actions

but I've seen prototypes of like

programming interfaces where given the

full file context uh you can predict not

only code but you can predict hey do you

want to refactor this function or do you

want to like rewrite this type into this

other type or if you're in a creative

app you could predict fairly complex

trajectories for the user like hey do

you want to take this drawing and

recolor it in this way or do you want to

apply this filter and then this other

filter and given the power of these

models I think we should I think it's

worth taking another look at um these

predictive interfaces as well obviously

leaving the escape hatch that is just

normal chat

yes so I'm excited for the day that

notion automatically knows I want to

create a table with something and it

will populate it with things exactly

what I want and uh I'm guessing that

you're going to be one of the people

that's making that the reality of the

future perhaps one day

sweet man well this was awesome there's

so many incredible questions for you

that are happening in the chat so if you

all want to continue the conversation I

am pretty sure that is is on slack and

it's not at Linus it is at the that is

yeah it's my there it is my internet

name I guess so

[Music]

thank you

[Music]