Generative Interfaces Beyond Chat // Linus Lee // LLMs in Production Conference
Summary
TLDRLinus discusses the evolution of generative interfaces beyond chatbots, emphasizing the importance of context in conversational AI. He explores how interfaces can be improved by integrating more intuitive interactions like point-and-select, and suggests using constraints to balance user-friendliness with the power of language models. Linus also touches on the need for interfaces to provide guidance and feedback loops to enhance user experience.
Takeaways
- ๐ก **Chat-style interfaces are useful but not the end-all**: Recognizing that while chat interfaces are valuable, there's room for evolution towards more intuitive and flexible interfaces.
- ๐ **Importance of context in conversations**: Emphasizing the need for language models to leverage context effectively to interpret user intent accurately.
- ๐ **Incremental evolution of interfaces**: Discussing how to move from chat-repeat style interactions to more engaging and intuitive interfaces.
- ๐ ๏ธ **UI design and interaction**: Linus' background in UI/UX and creative tools informs his perspective on building interfaces that balance flexibility and ease of use.
- ๐ **Conversation anatomy**: Breaking down the structure of a conversation into context, intent, internal monologue, and action to better understand language model interactions.
- ๐ฏ **Point and select interfaces**: Introducing the concept of using selections and pointing to clarify context and direct model actions within an interface.
- ๐ **Co-inhabiting workspaces**: The idea that interfaces should allow models to 'live' within the same workspace as the user to enhance interaction.
- ๐ **Using names to reference objects**: Suggesting that naming elements within an interface can help users interact more naturally with language models.
- ๐ **Iterative feedback loops**: Encouraging the design of interfaces that allow for quick feedback and iteration to improve user experience.
- ๐ **Predictive interfaces**: The potential of predictive models to suggest actions and make interfaces more intuitive, reducing the 'blank page' problem.
Q & A
What is the main topic of the discussion led by Linus?
-The main topic of the discussion is the evolution of generative interfaces beyond chat, exploring how to create more intuitive and flexible interfaces that balance the capabilities of language models with ease of use.
What does Linus suggest as a problem with current chat-style interfaces?
-Linus suggests that current chat-style interfaces often start with zero context, which leads to difficulties in prompting and a need for users to specify every detail of their intent explicitly.
What is the importance of context in conversation according to Linus?
-Context is crucial as it determines how everything said in a conversation is interpreted. Linus emphasizes that context is often undervalued in chat interfaces, leading to less effective communication with language models.
What is the role of 'intent' in a conversation as described by Linus?
-In a conversation, 'intent' is what the speaker implies, which could be explicit, direct, or implicit. It's a key component that language models need to interpret correctly, often relying heavily on context.
How does Linus propose to improve the interaction with language models?
-Linus proposes using point and select interfaces, where users can directly interact with objects in an application, and language models can observe these actions to better understand user intent and provide relevant responses.
What are the different ways Linus discusses for language models to observe user actions?
-Linus discusses several ways including an omniscience model where the model has access to all application states, 'call by name' for environments with named objects, direct mention of objects by users, and contextual actions similar to right-click menus.
Why does Linus think that adding constraints to interfaces can be beneficial?
-Adding constraints can make interfaces more intuitive and easier to use without sacrificing the power of language models. It provides a structured way for users to interact with models, reducing the 'blank page syndrome' and learning curve.
What is the concept of 'call by name' as mentioned by Linus?
-'Call by name' is a method where users can name different parts of an application, like artboards in a design app, and then refer to them directly in their prompts to the language model for more contextual interactions.
What does Linus mean by 'closing feedback loops' in interface design?
-Closing feedback loops refers to minimizing the time and effort it takes for users to evaluate system responses and iterate on their inputs. This can be achieved by providing multiple options, interactive components, or predictive actions.
What is the 'Holy Grail' of user interface design according to Linus?
-The 'Holy Grail' of user interface design is to balance an intuitive UI that's easy to learn and progressively understand with the flexibility to handle complex tasks, without sacrificing the power of language models.
How does Linus view the future of interfaces that incorporate AI and language models?
-Linus views the future of interfaces as incorporating AI and language models that co-inhabit the user's workspace, take advantage of rich shared context, predict happy path actions, and allow for a fast iteration loop, possibly including interactive components.
Outlines
๐ Introduction to Generative Interfaces
Linus introduces the topic of generative interfaces beyond chat, acknowledging the usefulness of chat interfaces but also recognizing their limitations. He discusses the need to evolve beyond text-based interactions and integrate language models with more intuitive interfaces. Linus shares his background in UI design and his work as a research engineer at Notion, focusing on building creative and productivity tools. He outlines the key areas of conversation: understanding the parts of a conversation important for language models, the role of context and selection in dialogue, and the benefits of adding constraints to interfaces to balance ease of use with the power of language models.
๐ฃ๏ธ Anatomy of a Conversation
Linus breaks down the components of a conversation, emphasizing the importance of context in communication. He explains how context shapes the interpretation of language and gives examples of how it varies in different scenarios. He also discusses the role of intent in a conversation, which can be explicit or implicit, and how it's crucial for language models to understand this intent. Linus mentions 'Chain of Thought' as an internal monologue for the model to interpret the user's intent within the given context. He uses examples like Co-pilot in VS Code to illustrate how context is already integrated in some applications, and how intent is clear when the user's environment is known.
๐ฏ Point and Select Interfaces
Linus explores the concept of 'point and select' interfaces, suggesting that they can help clarify context and direct the model's actions. He explains how pointing and selecting are natural actions in the physical world that can be translated into digital interactions. Linus proposes several ways for language models to observe user actions, such as the omniscience model where the model has access to all application states, 'call by name' for applications with named objects, direct mention of objects, and contextual actions similar to right-click menus. He argues that these methods can make conversational interfaces feel more integrated and intuitive.
๐ Balancing Intuition and Flexibility
Linus discusses the goal of balancing intuitiveness and flexibility in user interface design. He suggests that while chat interfaces offer flexibility, they often lack intuitiveness, leading to a steep learning curve for users. He proposes that by adding constraints and suggestions, interfaces can become more intuitive without sacrificing the power of language models. Linus also talks about the importance of closing feedback loops in creative and productivity applications, allowing users to quickly iterate and refine their requests. He gives examples of how providing a range of options or interactive components can enhance the user experience.
๐ Conclusion and Future Outlook
In conclusion, Linus summarizes the key ideas from the discussion, emphasizing the potential for dialogue interfaces built on language models to co-inhabit the user's workspace and take advantage of rich shared context. He suggests that interfaces should start with constrained actions but allow for advanced use through chat. Linus also reiterates the importance of speeding up the user's iteration loop through direct manipulation and interactive feedback. He expresses optimism about the future of intuitive and powerful conversational applications and encourages the exploration of predictive interfaces, given the advancements in language models.
Mindmap
Keywords
๐กGenerative Interfaces
๐กChat-style Interactions
๐กContext
๐กIntent
๐กCo-pilot
๐กPoint and Select
๐กConstraints
๐กConversational Interfaces
๐กIntuition
๐กFeedback Loops
๐กInteractive Components
Highlights
Introduction to generative interfaces beyond chat
The limitations of current chat-style interfaces
The importance of context in language model interactions
The role of intent in conversational interfaces
The concept of 'internal monologue' for language models
The action phase in conversational AI interfaces
Example of context usage in Co-pilot for VS Code
The need for more intuitive interfaces beyond chat
The idea of 'point and select' interfaces
Different ways for models to observe user actions
The concept of 'call by name' for object interaction
The potential of drag-and-drop for interface interaction
The role of contextual actions in interfaces
The importance of balancing intuitiveness and flexibility in UI design
The challenge of 'blank page syndrome' in chat interfaces
The idea of using predictive interfaces to aid user interaction
The concept of interactive components in chat interfaces
The goal of tightening feedback loops in creative applications
Final thoughts on balancing power and intuitiveness in interface design
Transcripts
alrighty uh there we go cool I don't
know or between between uh Harrison and
that song I don't know how to top that
but but I'll make my best attempt
um happy to be here to talk about
generative interfaces Beyond chat
uh I'm I'm Linus and I'll do my intro in
a bit but um where I want to start
today is I think we all generally have a
sense of
catch upd style chat is super useful
super valuable you can do a lot with
them but it's I think at this point we
all kind of accept that it's not the end
of the road right with these interfaces
and I had a I had a a tweet a while ago
where I was like you guys are telling me
that we're gonna invent literal super
intelligence and we're going to interact
with this thing by sending text back and
forth and and it's it's obvious that
it's not the end of the road but Chinese
here today most
um usages of language models and
interfaces that I've seen in production
are built on chat style interactions or
dialogue style turn-by-turn kind of
interactions and there's interesting
experiments on the fringes some of which
I'll hopefully mention later but Chad is
here today and so the leading question
that I want to spend our time talking
about today is given that Chatters where
we are and given the possibilities for
other things to come how do we
incrementally evolve what we have chat
repeat style chat forward to build more
interesting interfaces that balance the
flexibility of language models and the
power of language models with the ease
of use and the intuitiveness of some of
the other kinds of interfaces it can
build
so uh excited to talk about that a
little bit about me I think a lot about
UI design interfaces interaction design
Nai and I've I've spent a bunch of time
thinking about that in the context of
building creative tools and productivity
tools so it makes sense that I am
currently a notion number research
engineer at notion before that I spent a
couple years working independently also
pursuing these ideas building a lot of
prototypes uh it sounds like some of
which are going to be linked somewhere
in the chat
um and have worked on other kind of
productivity tool companies and apps
before
so
if I had to roadmap we're going to talk
about I think there's sort of three big
buckets first I want to just lay the
groundwork for how should we think about
conversations and dialogue what are the
parts of a conversation that we should
think about when we build language
models for conversations and then second
talk about specifically the idea of
context in a conversation and the
selection in a conversation which will
will come up and then third I want to
land on this idea of constraints and
adding constraints and the benefits that
they can have and how we can balance
adding constraints to make interfaces
more intuitive and more easy to use
without sacrifice and Power
so let's talk about conversations
let's say you and your buddy here are
about to talk about something and you
want to say something
even before you say anything at all the
communication channel has already opened
and kind of started because in any
conversation in any kind of like
linguistic act uh do you you start with
a shared context that context might be
uh your friend just pulled up with a
chair next to your office and you're
about to pay a program it might be
you're in a supermarket checking
something out it might be you're
collaborating with a co-worker it might
be your friend uh or a stranger walked
up to on the street that context
determines how everything that you say
and everything that your interlocutor
says back to you is interpreted and so I
think context is important and notably I
think in applications like child PT
context gets very very low billing uh
you basically start with zero context
and you have to embed all the context
that you want the model to use to
interpret your words in the prompt
itself which is where I think a lot of
the difficulties of prompting come from
so you start with some context
and then uh your the speaker will imply
some intent it could be explicit and
direct like hey can you pass me a glass
of water you could be a little more uh
it could be a little more implicit like
if I'm a construction worker I'm
building a house or I'm like assembling
a Lego kid I might say oh the blue brick
and that's not a complete thought but in
the context it can be interpreted to to
figure out exactly what you're looking
for or it could even be just like me
pointing at something right or or like
doing like one of these and then and
then that my partner in in the
conversation can interpret the intent
out of what I'm doing and that's that's
the speaker's role unless you have the
intent um especially important for
language models as giving the model time
to think I'm borrowing some I'm abusing
some reinforcement learning terminology
terminology here and calling this rule
out but uh people call this Chain of
Thought call this a scratch Pad some
internal monologue for the model or for
the recipient of this action to figure
out exactly what you mean and kind of do
the interpretation of your intent what
you said within the context that you
have and then once the model or the
recipient of the message is done
thinking there's some action the action
might be answering the question so maybe
just textual but more exciting and more
often I think we're seeing lots of apps
where the action is some combination of
a response back and
the action that the model is taking
whether in an application or integrating
with an API
so I think this is kind of where the
anatomy of a conversation if you really
break it down in a typical kind of
language model usage Style
let's let's take uh co-pilot does it as
an example this is a screenshot from
co-pilot next or copilot X chat all
these names are insane
um copilot chat inside vs code
this is one of those cases where there's
very clear you're already starting with
some context if you're building
something like this you wouldn't just
build the chat you would want the
language model to be aware of all the
context as much context you can get out
of this application so the context
includes things like what files you have
open uh or do you have a terminal open
is there what's the last a few commands
and the outputs that they got because
maybe the error message in the terminal
can inform what the model can do for the
user it includes it in things like what
line is the cursor on for the user or
what lines do they have selected because
selection is actually a really strong
signal for what the user is thinking
about and looking at it's kind of like
pointing but but on a screen or in a
text editor so you start with some
context
and there's the intent the intent is in
this case is write a set of unit test
functions for the selected code and you
can see that
in interfaces like this you really need
the context to interpret the intent
correctly and the more context you have
usually the better that interpretation
is going to be and then presumably there
are some some internal monologue
thinking for the model
um uh and then after that we we get the
models action back out
so
when we think of prompt these
conversational interfaces I think we
usually focus on the intent The Prompt
and then the completion the output the
action but there's there are these other
elements that I think we should also
think about
um the context in which the users uh
intenses being interpreted and then also
the internal monologue
so uh
this is a screenshot of Microsoft's
copilot for Excel I think this is an
interesting example of a really rich
valuable application where there is a
chat interface but where there's clearly
a lot more we could do
um so so in this case there is so much
context that the model can use to figure
out exactly what the user is doing and
maybe what they're trying to accomplish
um and if you had a chat sidebar like
this the sidebar sort of exists in a
totally different Universe uh than the
chat I mean the model can theoretically
look at everything that's in the
spreadsheet that the user has open but
in this case in this screenshot the user
is having to refer to specific columns
and specific part of this spreadsheet
verbally by naming things saying you
know the in this case it's the column
about uh the sales data or last month's
sales data why can't I just point in and
select and then say what about this
column right in the real world that's
kind of how we work if I want to refer
to something that's in front of me I'll
just point to it I'll just look at it
um
so in this case I think
having the chat agent be co-located or
like inhabited co-inhabiting the same
workspace that the the user is in I
think is a key part of how to make these
interfaces gel a little better
and without that I think these
conversational interfaces start to feel
more like the the command line interface
of generative AI where you're having to
specify increment every possible
information about your intent and your
action explicitly into the prompt rather
than being able to work more fluidly in
a kind of a shared working environment
so uh how do we how do we where do we go
from here well I think
I keep talking about this idea of
pointing and and using the context and
selecting things and so one really
powerful technique that I think we can
look for is using our hands using
selections and pointing
and when you point at things in the
context or when you select things there
are a few different ways I think for the
language model to observe what you're
pointing out or what you're doing
sort of in order from both grounded and
reality so that I think the most
interesting and kind of out there I
think we can start with uh yeah so point
and selection interfaces uh one way to
think of point and select is uh sort of
breaking down your action into nouns and
then verbs so what I mean by that is if
you're in a spreadsheet the noun might
be like the column that I want to
manipulate and you select the noun and
then the vertical bee I'm going to
filter it or I want to aggregate it or I
want to hide it or delete it or
duplicate it
um if you're in a writing app then now
it might be a single block in the
writing app like the title block or it
could be the entire page or it could be
like a new file
um in the real world this point and
select mechanic is sort of built into
every object in every material if I
wanna
take action on some object I have to
first like grab the thing and then do
something with it
um but in in chat style interfaces I
think it's less obvious
but this point in select mechanic is
also what makes the web great for a lot
of applications because there's existing
sort of materiality built into
everything on the web every bit of text
by default on the web is selectable you
can select anything you can copy paste
anything uh you can uh often drag and
drop files into web pages and so there's
all this like noun and Denver based
mechanic built into materials that you
use to build apps on the web
and uh in chat all of that kind of all
of those affordances around selecting
objects and then applying actions to
them are kind of gone and I think we
might we could think about how to bring
that back to uh chat style interfaces
and point and select I think are most
useful for helping clarify context and
focus the model on something you're
looking at or alternatively directing
the kind of action stream or the output
stream of the model so if you're in a
writing app you could say you know take
select something take summarize this bit
and then like put it here at the top of
the page or or you know make a new page
over over here and being able to point
point and select I think are useful for
directing the output as well
so there are a few ways that we can get
the model to observe what you're doing
or what you're pointing out the most
common one currently and I think the
most obvious one is sort of this what
I'm calling like the omniscience model
which is the model can look at
everything everywhere visible all the
time it just kind of knows the entire
state of the application but it's up to
the model to figure out what to query
and what to look at it's so it's
technically the context is technically
fully accessible but the model doesn't
know exactly what you want the model to
look at
next level up from that is what I'm
calling call by name which I think is
kind of interesting for certain types of
application especially kind of pro
applications where there's a lot of
flexibility and customization if you
have an application like like a design
app like figma or a sketch you could
imagine naming different artboards or
different panels and then being able to
app mention and say hey can you clean up
you know panel two or can you clean up
like the like timeline panel
so uh this only really makes sense for
environments where it makes sense to
name objects and refer to them by kind
of handles or names but if that's the
case then I think this is an interesting
way to incorporate context and be able
to directly point to things but with
your words using the names
there's also this kind of really
interesting uh interface that I I don't
think anybody's really seen in
production which is what I'm calling
literally mentioning something
um this in particular is a screenshot
from a paper from a project called
sikulu from I believe an MIT lab where
they uh had a programming language that
interleaved icons and images with a way
to program around the UI and you could
mention you could imagine an interface
where if I wanted to refer to a
paragraph I could start writing and
saying summarize and then I literally
drag and drop the paragraph into the
prompt box or if I wanted to transform
an image I could drag and drop an image
or or if I have to if I want to talk to
a person that's in my contact list I
could grab that person's icon and say
hey can you call this person and then
just drag and drop that image or that
object and so having support for these
rich objects inside the prompt box I
think is a really interesting
possibility
and then the last one is what I'm
calling contextual actions which is a
great example of this is uh like a right
click
so uh an example of the right click is
these context menus right so the left is
notion right is figma you grab an object
sort of metaphorically and then you can
see all the options that are available
to you all the things you might want to
do with that object in a lot of current
applications these are hard-coded in but
you could imagine using a language model
to say okay here's what here's the
object that the user has in their hands
or the most likely actions given the
full context of the application and
given maybe even their like history of
actions and what the title of the file
is and what they want to do what are the
most likely actions they might want to
do you could you could have the model
select from a list you could also have
the model generate possible trajectories
that the user might want to take and so
context menu I think is an interesting
way to reveal actions that you want the
user to take without having to force
them to to type the instruction up fully
another kind of context menu pattern is
this kind of thought driven programming
or autocomplete driven programming which
I think is the the analog of right click
but with text so if I'm typing in a text
editor or code editor and you hit like
dot like document.potty Dot and it'll
show me all the autocomplete options
this is kind of like saying I'm holding
this object in my hand what are all the
things that are accessible to me from
this or what are the actions that I can
take from this or uh in the the other
panel I have tab completion so I'm
working inside a terminal I have this
CLI in my hand what are the things that
I can do with it tell me the
possibilities and this is another way of
grabbing an object and then sort of
showing me what's possible and you can
imagine powering something like this
with a language model as well
and then lastly this is a slightly more
complex pattern but but where you if the
user selects an object you could uh
materialize an entire piece of UI like a
side panel or or a kind of an overlay so
on the left again is notion AI on the
right is is keynote uh which is what I
was using to make the stack and in
either case you select an object and
then you can see a whole host of options
for how you want to control it and this
gives the user a lot of extra power at
the cost of maybe not being obvious
exactly what the user wants to do or
what the user should take action on
so in all these cases we have this sort
of like noun and then verb like choose
the object and then what action you want
to take kind of pattern and that lets
the system constrain the action space
that the user might want to take and
maybe even come up with uh follow-ups or
suggestions and what the best actions
you could take are
given all of this given and given what
we talked about around the anatomy of a
conversation I think
and then when you look back at something
like Challenger PT charger PT is really
just about okay you have this little
tiny prompt box and you have to cram all
of the context all of the intent in
there and also everything that you want
the model to know about where you want
that model to take its action and that
that I think is a good place to start
but it is limiting and there are ways we
can expand out of it
um so one way to summarize the the sort
of ground that we've covered might be
that the Holy Grail or one powerful goal
of user interface design is to balance
um intuition building an intuitive UI
that's uh easy to learn and sort of
progressively understand but flexible
um and intuitive and flexibility I think
is the strength of language models
um
in chat style interfaces with chat cell
interfaces you have the full access to
the full capabilities of a model you can
ask it to do anything that the model
could possibly do including things like
use API and use tools and fully specify
like a programming language that you
want the model to use if you want and
that's the strength of falling back to
chat but by adding these constraints
where you start with something in your
hand and then try to recommend or
suggest or follow up and say given this
is what you're looking at given this is
the locus of your attention right now
here are the things that you can do
um and maybe predict some actions and
add some guardrails add some structure
to the instruction I think that's where
we can add sort of bring back the
intuitiveness of graphical user
interfaces without sacrificing the power
of language models
open-ended natural language interfaces I
think uh trades off too much of that
intuitiveness for for the flexibility
um so in an app like chachu PT you have
this blank page syndrome where the user
doesn't know exactly what they are
supposed to type in maybe they have a a
sense of maybe I want a summary or maybe
I want a conversation of a certain style
but there's no affordances in the UI to
give them hints of okay these are the
things that the model is good at these
are the ways you might not phrase the
answer none of this none of this exists
and so I think it adds a huge learning
curve and as detrimental to the ease of
Discovery and by bringing back some of
these graphical interfaces I think we
can improve that situation a bit
and then lastly since I'm closing in on
time
um I wanted to add one more note around
another goal of interface design often
which I think is closing feedback loops
particularly in Creative applications
and sometimes also in productivity
applications you want to try to tighten
the close the
um
tighten the the feedback loop between
the user attempting something and maybe
having something in their mind they want
to see and then looking at the results
evaluating the result and then figuring
out okay this is how I need to iterate
this is the fix that I need to apply to
get get the model to generate what I
want
and uh there are a few ways to do this
right one is instead of the model
generating one output if you're say
generating images instead of a model
generating one output it could generate
a range of outputs and that allows users
to pick okay these are these are maybe
four different ways of looking at this
answer for four different images that
you could generate this is one that I
like and then iterate on that again this
only really works if the output is easy
to evaluate if I ask the model to write
an essay and it gives me four different
essays it would be it would be pretty
difficult to use
um so going along with that idea I think
uh you want
um
if whenever possible you want to prefer
uh what I've heard referred to as like
people like to shop more than like to
create they like having a range of
options they can choose from and maybe
even like swipe left and right style
kind of do I like this do I not they
want to that kind of interface is easier
to use a more intuitive uh and more
engaging than here's a blank page tell
me exactly what you want and again
powering that kind of thing comes back
to okay what coming up with uh options
and predictions of actions and
suggestions that you is that you sort of
uh plan out for the user in case they
want
and then the user can make that
selection and then lastly
um I've seen some prototypes of this
this thing that I'm calling interactive
components what I'm referring to this
we're referring to by interactive
components is if you are in a chat kind
of interface and you ask a question and
instead of responding with a paragraph
of answer maybe I ask like uh what's the
weather in New York and instead of
responding with a paragraph of answer
the model says okay here's maybe maybe
it says the temperature tomorrow is 85
and then there's a little weather widget
with a slider for time or with with
buttons we're looking at precipitation
and these other things and the model can
synthesize maybe the model will be able
to synthesize little interactive
components little widgets on the Fly and
that again helps me close my feedback
loop by saying okay these are these
other options of information that I can
look at and I can really explore them
really directly without having to
re-prompt and retype my my queries
so uh bringing it all back I wanted to
close out with one of my one of my
favorite quotes from one of my favorite
papers essays when I'm thinking about
creative tools uh by
um
okay content in this essay called casual
creators
um and I I think this quote is great so
I'm just going to quote at length the
possibility space of creative tools and
what you can do the action space should
be narrow enough to exclude broken
artifacts like models that fall over or
break when when you're in a 3D printing
app but it should be broad enough to
contain surprising artifacts as well the
surprising quality of the artifacts
motivates the user to explore the
possibilities base in search of new
discoveries new use cases a motivation
which disappears if that space is too
uniform so again she's talking about
this balance of you want to constrain
just a bit just enough that the user
never gets stuck in that like blank page
state that there's always some option
that they can take or always some
suggested action that seems interesting
uh you want to preserve the power and
the flexibility and the sometimes
surprising quality of these language
models and I think that distracting that
balance is is sort of the primary
challenge of building interfaces for
these models
oh
something's happening
there we go okay uh uh so last slide
just to sum up I think five Big Ideas
that I want uh it'd be great you could
take away from this this conversation I
think good dialogue interface is built
on llms can have agents that co-inhabit
your workspace that are there and can
see what you're doing in its full detail
including where your attention is it
should take full advantage of the rich
shared context that you have with the
model
uh to interpret your actions so that you
don't have to cram everything into a
prompt I think these interfaces can lead
initially with constrained happy path
actions that you can use models and
other language models and other
predictive models to try to predict and
then if the user wants to do something
more advanced or different we can always
fall back to chat as an escape patch
because there's the power and the
flexibility in language models and then
lastly
whether you're building a chat interface
or something a little more direct
manipulation graphical uh it's I think
it's always good to think about how we
can speed up that iteration Loop
especially by forcing the user not not
forcing the user to type text but by
responding uh more directly with a with
a mouse or with a touchscreen for
closing that feedback loop
so with that I hope that was interesting
and useful and hope you can build some
some great conversational applications
uh given there's a extrude
wow wow I mean so many questions there's
so much stuff that's going through my
mind and I love the idea of how it's
like
you
you're helping guide people that is so
nice to think about instead of just
leaving this open space And then trying
to figure it out it's like hey can we
can we suggest things so that people can
figure it out with us as opposed to just
letting their imagination go wild and
then it may or may not turn out okay
yeah exactly I think
some there's some history of predictive
interfaces like this and I think uh in
the design World collectively our tastes
have been soured a bit on predictive
interfaces because the model that we've
used models that we've used in the past
have not been that good and so we
couldn't really predict that far and we
could really predict only simple actions
but I've seen prototypes of like
programming interfaces where given the
full file context uh you can predict not
only code but you can predict hey do you
want to refactor this function or do you
want to like rewrite this type into this
other type or if you're in a creative
app you could predict fairly complex
trajectories for the user like hey do
you want to take this drawing and
recolor it in this way or do you want to
apply this filter and then this other
filter and given the power of these
models I think we should I think it's
worth taking another look at um these
predictive interfaces as well obviously
leaving the escape hatch that is just
normal chat
yes so I'm excited for the day that
notion automatically knows I want to
create a table with something and it
will populate it with things exactly
what I want and uh I'm guessing that
you're going to be one of the people
that's making that the reality of the
future perhaps one day
sweet man well this was awesome there's
so many incredible questions for you
that are happening in the chat so if you
all want to continue the conversation I
am pretty sure that is is on slack and
it's not at Linus it is at the that is
yeah it's my there it is my internet
name I guess so
[Music]
thank you
[Music]
Browse More Related Video
5.0 / 5 (0 votes)