Can we create new senses for humans? | David Eagleman

Summary

The transcript explores the concept of 'umwelt,' or the subjective reality experienced by different species, highlighting how human perception is limited by biology. Neuroscientist David Eagleman discusses how technology can expand our sensory experiences beyond natural limits, enabling sensory substitution and even sensory addition. He demonstrates how devices like vests can convert sound into vibrations, allowing the deaf to perceive speech, and suggests future possibilities, such as astronauts monitoring space station health through sensory input. Eagleman envisions a future where humans can customize their sensory experience, unlocking new ways to interact with the world.

Takeaways

• 🌌 Humans are limited in their perception of reality, only perceiving a small fraction of the vast cosmos around them.
• 👁 Our experience of the world is constrained by biology, with our senses only capturing a tiny portion of available information.
• 🐍 Different animals perceive reality in unique ways, such as snakes detecting infrared or bats using echolocation.
• 🧠 The brain is highly adaptable and can process sensory information from various sources, regardless of how it is received.
• 🔌 Sensory substitution technologies, like those that allow blind people to 'see' through their tongues or backs, demonstrate the brain's flexibility.
• 🔊 The vest that translates sound into vibrations on the skin shows potential for expanding sensory experiences for the deaf.
• 💡 Sensory addition could allow humans to experience new kinds of data, like real-time stock market information or the emotional state of a crowd.
• 🚀 This technology could be applied in various fields, such as allowing astronauts to feel the health of their spacecraft or pilots to feel the status of their aircraft.
• 📊 The brain's ability to handle multidimensional data could revolutionize how we interact with complex information, making it more intuitive.
• 🔄 The future may involve humans choosing their sensory experiences, enhancing and expanding their perception beyond natural limitations.

Q & A

What is the 'umwelt' and how does it relate to our perception of reality?

The 'umwelt' is a term used to describe the surrounding world as experienced by an organism. It relates to our perception of reality by highlighting that each species perceives the world differently based on their sensory capabilities. Humans, for example, perceive only a small fraction of reality due to the limitations of our sensory organs.

Why do humans only perceive a small fraction of the electromagnetic spectrum?

Humans only perceive a small fraction of the electromagnetic spectrum because our biological receptors, specifically our eyes, are only sensitive to certain wavelengths of light, known as the visible spectrum. We are unaware of other wavelengths, such as radio waves or X-rays, because we lack the necessary biological sensors to detect them.

How do animals like snakes or honeybees perceive reality differently from humans?

Snakes and honeybees perceive reality differently because they have sensory capabilities that humans do not. Snakes can detect infrared radiation, allowing them to sense heat, while honeybees can see ultraviolet light, giving them a different visual experience of the world.

What is sensory substitution, and how has it been demonstrated?

Sensory substitution is the process of feeding information into the brain through unconventional sensory channels, allowing the brain to interpret this data in a meaningful way. It has been demonstrated with blind individuals who could 'see' objects by feeling patterns on their back or tongue, showing that the brain can adapt to different types of input.

What is the significance of the brain's ability to process different types of sensory input?

The significance of the brain's ability to process different types of sensory input is that it shows the brain's flexibility and capacity to adapt. It can interpret data from various sources and construct a coherent understanding of the environment, even when the data comes from unconventional sources, such as vibrations on the skin.

How does the vest used in the research translate sound into a sensory experience for deaf individuals?

The vest used in the research translates sound into a sensory experience by converting sound waves into patterns of vibration that can be felt on the skin. These vibrations correspond to the sounds, allowing deaf individuals to 'feel' speech and other auditory information, which their brains can then learn to interpret.

What potential does the vest technology have for sensory addition?

The vest technology has the potential for sensory addition by allowing humans to experience new kinds of data in real-time, such as stock market trends or environmental conditions, directly through their bodies. This could expand human perception and interaction with the world in unprecedented ways.

Why does the brain not care about the origin of sensory data?

The brain does not care about the origin of sensory data because it is designed to process and make sense of any electrochemical signals it receives, regardless of where they come from. This allows the brain to interpret data from various sources, whether they are natural senses or artificial inputs, and construct a coherent perception of reality.

What are the implications of the brain's 'general-purpose' computing ability?

The implications of the brain's 'general-purpose' computing ability are vast. It means that the brain can potentially adapt to new forms of sensory input, allowing for the development of technologies that can enhance or even expand our sensory experience. This could lead to new ways of interacting with the world, beyond our natural capabilities.

What are some possible future applications of sensory expansion technologies?

Possible future applications of sensory expansion technologies include allowing astronauts to feel the health of a space station, enabling people to monitor their own biological states like blood sugar levels, or providing pilots with a direct sense of their aircraft's condition. These technologies could revolutionize how we interact with and understand complex data in real-time.

Outlines

00:00

🔬 Understanding the Limits of Human Perception

This paragraph explores the limitations of human perception, emphasizing that we are only able to perceive a tiny fraction of the electromagnetic spectrum. While our biology allows us to see colors and light, it leaves us blind to other wavelengths like radio waves and infrared. The paragraph introduces the concept of 'umwelt,' a term used to describe the limited slice of reality each species can perceive, highlighting how different animals sense the world differently based on their biological makeup.

05:03

🧠 The Brain's Ability to Interpret Signals

This section delves into how the brain processes information. Despite being isolated in the darkness of the skull, the brain interprets electrochemical signals to create our subjective experience of the world. The brain's adaptability is highlighted, as it can make sense of signals from artificial devices like cochlear implants or retinal implants. This ability to interpret different data sources suggests that the brain is a general-purpose computing device, capable of making sense of any sensory input it receives.

10:04

🔧 Sensory Substitution and Human Potential

The focus here is on sensory substitution, where the brain is fed information through unconventional channels and learns to interpret it. The paragraph discusses how blind individuals have been able to 'see' through their backs, tongues, and other body parts using various devices. This demonstrates the brain's incredible flexibility in processing new types of sensory data. The section ends by introducing a project aimed at enabling deaf people to 'hear' through vibrations on a vest, showing early success in this innovative approach.

15:05

📡 Expanding Human Senses with Technology

This paragraph explores the potential for sensory addition, where technology could introduce entirely new senses to humans. Examples include real-time data feeds from the internet being converted into sensory input, such as economic data or emotional sentiment analysis from Twitter. The possibilities for enhancing human experience are vast, ranging from monitoring the International Space Station's health to feeling your own blood sugar levels. The concept suggests that we can transcend our biological limitations by integrating new sensory devices into our experience.

20:06

🚀 The Future of Human Sensory Expansion

In the final paragraph, the potential applications of sensory expansion technology are discussed. The speaker envisions a future where humans can feel complex data streams, such as the state of machinery or health, through sensory input rather than visual or auditory cues. The talk concludes with the idea that humans will soon be able to choose their sensory experiences, no longer confined by the limitations of our natural biology. This marks a significant leap in human evolution, enabling us to experience the universe in entirely new ways.

Keywords

💡Umwelt

The concept of 'Umwelt' refers to the surrounding sensory world that an organism perceives, based on its biological sensory apparatus. In the video, it's used to explain how different species experience reality differently. For example, a tick's umwelt revolves around temperature and butyric acid, while a human's umwelt is centered on the senses available to them, such as sight and sound.

💡Sensory Substitution

Sensory substitution involves feeding information into the brain through unusual sensory channels, allowing the brain to interpret this information as if it were coming from a traditional sense. The video discusses how blind people can 'see' using tactile feedback on their back or tongue, showing that the brain can adapt to new forms of input and generate meaningful perceptions from them.

💡Sensory Addition

Sensory addition refers to the idea of introducing completely new sensory experiences beyond the ones evolution has provided. The video explores this concept by discussing the potential to feed real-time data from sources like the internet or stock market directly into the brain, allowing humans to experience new dimensions of reality.

💡Peripheral Devices

'Peripheral devices' in the video are compared to sensory organs like eyes and ears, which provide data to the brain. The brain, like a general-purpose computing device, interprets this data without concern for its source. This analogy highlights the flexibility of the brain to integrate new sensory inputs through technological advancements.

💡General Purpose Computing Device

This term describes the brain's ability to process and interpret various types of data, regardless of its origin. The brain does not inherently distinguish between traditional sensory inputs and those provided by technology, as demonstrated by sensory substitution devices. This concept is crucial to understanding how the brain adapts to new sensory information.

💡Electrochemical Signals

Electrochemical signals are the primary language of the brain, representing the data it receives from sensory organs. The video explains that regardless of the sensory organ or technological device providing the input, the brain translates these signals into our subjective experience of reality. This underpins the brain's adaptability and the potential for sensory substitution.

💡Sensory Technology

Sensory technology refers to devices that can enhance or replace biological senses. Examples from the video include cochlear implants for hearing and retinal implants for vision, which demonstrate that technology can interface with the brain to provide sensory experiences, expanding or restoring the umwelt for individuals with sensory impairments.

💡Artificial Hearing/Vision

These are examples of sensory substitution where technology, such as cochlear implants or retinal implants, is used to replicate the function of biological senses. The video uses these examples to illustrate how technology can bridge the gap between sensory deficits and the brain's ability to interpret new forms of sensory data.

💡Pattern Recognition

Pattern recognition is the brain's ability to detect and interpret regularities in the data it receives. The video emphasizes how the brain excels at this task, allowing it to make sense of complex sensory inputs, whether they come from traditional senses or new technological channels like the vest that translates sound into vibrations.

💡Human Expansion

Human expansion refers to the idea of extending the sensory and perceptual capabilities of humans beyond natural limitations using technology. The video discusses potential applications, such as astronauts feeling the health of a space station or people sensing data from the internet, which could lead to new ways of experiencing and interacting with the world.

Highlights

Humans are not naturally equipped to perceive the full spectrum of reality, as our senses are limited to a narrow band of perception.

Most of the electromagnetic spectrum, such as radio waves and X-rays, is invisible to us, though other animals and machines can detect them.

The concept of 'umwelt' describes the limited sensory world of each species, emphasizing that humans are confined by their biological senses.

The brain is a general-purpose computing device that interprets signals from various sensory inputs, regardless of their source.

Technology can expand human perception by creating new sensory inputs, such as artificial hearing and vision devices.

Sensory substitution allows the brain to process information from unconventional sources, such as converting visual data into tactile signals for the blind.

The 'brainport' device enables blind people to perceive their surroundings through electrotactile signals on their tongue, effectively allowing them to 'see' through their tongue.

The brain does not differentiate between sensory inputs, as long as the signals are interpretable, making it possible to experience new types of perception.

A vest equipped with vibratory motors can translate sound into patterns of vibration, enabling deaf individuals to understand spoken language through touch.

The concept of sensory addition involves using technology to introduce entirely new senses, potentially allowing humans to experience data like real-time stock market movements.

Experiments are being conducted to determine if people can develop a direct perceptual experience of abstract data, such as economic trends, through sensory addition.

The potential applications of sensory addition are vast, including enhancing astronauts' perception of spacecraft health or providing 360-degree vision.

The brain's flexibility in adapting to new sensory inputs opens up possibilities for redefining human perception and experience through technology.

The project demonstrates that wearable technology could be a cost-effective alternative to invasive medical procedures, like cochlear implants.

The future of human experience may involve choosing and customizing our sensory peripherals, enabling us to interact with and understand the universe in entirely new ways.

Transcripts

We are built out of very small stuff,

and we are embedded in a very large cosmos,

and the fact is that we are not very good at understanding reality

at either of those scales,

and that's because our brains

haven't evolved to understand the world at that scale.

Instead, we're trapped on this very thin slice of perception

right in the middle.

But it gets strange, because even at that slice of reality that we call home,

we're not seeing most of the action that's going on.

So take the colors of our world.

This is light waves, electromagnetic radiation that bounces off objects

and it hits specialized receptors in the back of our eyes.

But we're not seeing all the waves out there.

In fact, what we see

is less than a 10 trillionth of what's out there.

So you have radio waves and microwaves

and X-rays and gamma rays passing through your body right now

and you're completely unaware of it,

because you don't come with the proper biological receptors

for picking it up.

There are thousands of cell phone conversations

passing through you right now,

and you're utterly blind to it.

Now, it's not that these things are inherently unseeable.

Snakes include some infrared in their reality,

and honeybees include ultraviolet in their view of the world,

and of course we build machines in the dashboards of our cars

to pick up on signals in the radio frequency range,

and we built machines in hospitals to pick up on the X-ray range.

But you can't sense any of those by yourself,

at least not yet,

because you don't come equipped with the proper sensors.

Now, what this means is that our experience of reality

is constrained by our biology,

and that goes against the common sense notion

that our eyes and our ears and our fingertips

are just picking up the objective reality that's out there.

Instead, our brains are sampling just a little bit of the world.

Now, across the animal kingdom,

different animals pick up on different parts of reality.

So in the blind and deaf world of the tick,

the important signals are temperature and butyric acid;

in the world of the black ghost knifefish,

its sensory world is lavishly colored by electrical fields;

and for the echolocating bat,

its reality is constructed out of air compression waves.

That's the slice of their ecosystem that they can pick up on,

and we have a word for this in science.

It's called the umwelt,

which is the German word for the surrounding world.

Now, presumably, every animal assumes

that its umwelt is the entire objective reality out there,

because why would you ever stop to imagine

that there's something beyond what we can sense.

Instead, what we all do is we accept reality

as it's presented to us.

Let's do a consciousness-raiser on this.

Imagine that you are a bloodhound dog.

Your whole world is about smelling.

You've got a long snout that has 200 million scent receptors in it,

and you have wet nostrils that attract and trap scent molecules,

and your nostrils even have slits so you can take big nosefuls of air.

Everything is about smell for you.

So one day, you stop in your tracks with a revelation.

You look at your human owner and you think,

"What is it like to have the pitiful, impoverished nose of a human?

(Laughter)

What is it like when you take a feeble little noseful of air?

How can you not know that there's a cat 100 yards away,

or that your neighbor was on this very spot six hours ago?"

(Laughter)

So because we're humans,

we've never experienced that world of smell,

so we don't miss it,

because we are firmly settled into our umwelt.

But the question is, do we have to be stuck there?

So as a neuroscientist, I'm interested in the way that technology

might expand our umwelt,

and how that's going to change the experience of being human.

So we already know that we can marry our technology to our biology,

because there are hundreds of thousands of people walking around

with artificial hearing and artificial vision.

So the way this works is, you take a microphone and you digitize the signal,

and you put an electrode strip directly into the inner ear.

Or, with the retinal implant, you take a camera

and you digitize the signal, and then you plug an electrode grid

directly into the optic nerve.

And as recently as 15 years ago,

there were a lot of scientists who thought these technologies wouldn't work.

Why? It's because these technologies speak the language of Silicon Valley,

and it's not exactly the same dialect as our natural biological sense organs.

But the fact is that it works;

the brain figures out how to use the signals just fine.

Now, how do we understand that?

Well, here's the big secret:

Your brain is not hearing or seeing any of this.

Your brain is locked in a vault of silence and darkness inside your skull.

All it ever sees are electrochemical signals

that come in along different data cables,

and this is all it has to work with, and nothing more.

Now, amazingly,

the brain is really good at taking in these signals

and extracting patterns and assigning meaning,

so that it takes this inner cosmos and puts together a story

of this, your subjective world.

But here's the key point:

Your brain doesn't know, and it doesn't care,

where it gets the data from.

Whatever information comes in, it just figures out what to do with it.

And this is a very efficient kind of machine.

It's essentially a general purpose computing device,

and it just takes in everything

and figures out what it's going to do with it,

and that, I think, frees up Mother Nature

to tinker around with different sorts of input channels.

So I call this the P.H. model of evolution,

and I don't want to get too technical here,

but P.H. stands for Potato Head,

and I use this name to emphasize that all these sensors

that we know and love, like our eyes and our ears and our fingertips,

these are merely peripheral plug-and-play devices:

You stick them in, and you're good to go.

The brain figures out what to do with the data that comes in.

And when you look across the animal kingdom,

you find lots of peripheral devices.

So snakes have heat pits with which to detect infrared,

and the ghost knifefish has electroreceptors,

and the star-nosed mole has this appendage

with 22 fingers on it

with which it feels around and constructs a 3D model of the world,

and many birds have magnetite so they can orient

to the magnetic field of the planet.

So what this means is that nature doesn't have to continually

redesign the brain.

Instead, with the principles of brain operation established,

all nature has to worry about is designing new peripherals.

Okay. So what this means is this:

The lesson that surfaces

is that there's nothing really special or fundamental

about the biology that we come to the table with.

It's just what we have inherited

from a complex road of evolution.

But it's not what we have to stick with,

and our best proof of principle of this

comes from what's called sensory substitution.

And that refers to feeding information into the brain

via unusual sensory channels,

and the brain just figures out what to do with it.

Now, that might sound speculative,

but the first paper demonstrating this was published in the journal Nature in 1969.

So a scientist named Paul Bach-y-Rita

put blind people in a modified dental chair,

and he set up a video feed,

and he put something in front of the camera,

and then you would feel that

poked into your back with a grid of solenoids.

So if you wiggle a coffee cup in front of the camera,

you're feeling that in your back,

and amazingly, blind people got pretty good

at being able to determine what was in front of the camera

just by feeling it in the small of their back.

Now, there have been many modern incarnations of this.

The sonic glasses take a video feed right in front of you

and turn that into a sonic landscape,

so as things move around, and get closer and farther,

it sounds like "Bzz, bzz, bzz."

It sounds like a cacophony,

but after several weeks, blind people start getting pretty good

at understanding what's in front of them

just based on what they're hearing.

And it doesn't have to be through the ears:

this system uses an electrotactile grid on the forehead,

so whatever's in front of the video feed, you're feeling it on your forehead.

Why the forehead? Because you're not using it for much else.

The most modern incarnation is called the brainport,

and this is a little electrogrid that sits on your tongue,

and the video feed gets turned into these little electrotactile signals,

and blind people get so good at using this that they can throw a ball into a basket,

or they can navigate complex obstacle courses.

They can come to see through their tongue.

Now, that sounds completely insane, right?

But remember, all vision ever is

is electrochemical signals coursing around in your brain.

Your brain doesn't know where the signals come from.

It just figures out what to do with them.

So my interest in my lab is sensory substitution for the deaf,

and this is a project I've undertaken

with a graduate student in my lab, Scott Novich,

who is spearheading this for his thesis.

And here is what we wanted to do:

we wanted to make it so that sound from the world gets converted

in some way so that a deaf person can understand what is being said.

And we wanted to do this, given the power and ubiquity of portable computing,

we wanted to make sure that this would run on cell phones and tablets,

and also we wanted to make this a wearable,

something that you could wear under your clothing.

So here's the concept.

So as I'm speaking, my sound is getting captured by the tablet,

and then it's getting mapped onto a vest that's covered in vibratory motors,

just like the motors in your cell phone.

So as I'm speaking,

the sound is getting translated to a pattern of vibration on the vest.

Now, this is not just conceptual:

this tablet is transmitting Bluetooth, and I'm wearing the vest right now.

So as I'm speaking -- (Applause) --

the sound is getting translated into dynamic patterns of vibration.

I'm feeling the sonic world around me.

So, we've been testing this with deaf people now,

and it turns out that after just a little bit of time,

people can start feeling, they can start understanding

the language of the vest.

So this is Jonathan. He's 37 years old. He has a master's degree.

He was born profoundly deaf,

which means that there's a part of his umwelt that's unavailable to him.

So we had Jonathan train with the vest for four days, two hours a day,

and here he is on the fifth day.

Scott Novich: You.

David Eagleman: So Scott says a word, Jonathan feels it on the vest,

and he writes it on the board.

SN: Where. Where.

DE: Jonathan is able to translate this complicated pattern of vibrations

into an understanding of what's being said.

SN: Touch. Touch.

DE: Now, he's not doing this --

(Applause) --

Jonathan is not doing this consciously, because the patterns are too complicated,

but his brain is starting to unlock the pattern that allows it to figure out

what the data mean,

and our expectation is that, after wearing this for about three months,

he will have a direct perceptual experience of hearing

in the same way that when a blind person passes a finger over braille,

the meaning comes directly off the page without any conscious intervention at all.

Now, this technology has the potential to be a game-changer,

because the only other solution for deafness is a cochlear implant,

and that requires an invasive surgery.

And this can be built for 40 times cheaper than a cochlear implant,

which opens up this technology globally, even for the poorest countries.

Now, we've been very encouraged by our results with sensory substitution,

but what we've been thinking a lot about is sensory addition.

How could we use a technology like this to add a completely new kind of sense,

to expand the human umvelt?

For example, could we feed real-time data from the Internet

directly into somebody's brain,

and can they develop a direct perceptual experience?

So here's an experiment we're doing in the lab.

A subject is feeling a real-time streaming feed from the Net of data

for five seconds.

Then, two buttons appear, and he has to make a choice.

He doesn't know what's going on.

He makes a choice, and he gets feedback after one second.

Now, here's the thing:

The subject has no idea what all the patterns mean,

but we're seeing if he gets better at figuring out which button to press.

He doesn't know that what we're feeding

is real-time data from the stock market,

and he's making buy and sell decisions.

(Laughter)

And the feedback is telling him whether he did the right thing or not.

And what we're seeing is, can we expand the human umvelt

so that he comes to have, after several weeks,

a direct perceptual experience of the economic movements of the planet.

So we'll report on that later to see how well this goes.

(Laughter)

Here's another thing we're doing:

During the talks this morning, we've been automatically scraping Twitter

for the TED2015 hashtag,

and we've been doing an automated sentiment analysis,

which means, are people using positive words or negative words or neutral?

And while this has been going on,

I have been feeling this,

and so I am plugged in to the aggregate emotion

of thousands of people in real time,

and that's a new kind of human experience, because now I can know

how everyone's doing and how much you're loving this.

(Laughter) (Applause)

It's a bigger experience than a human can normally have.

We're also expanding the umvelt of pilots.

So in this case, the vest is streaming nine different measures

from this quadcopter,

so pitch and yaw and roll and orientation and heading,

and that improves this pilot's ability to fly it.

It's essentially like he's extending his skin up there, far away.

And that's just the beginning.

What we're envisioning is taking a modern cockpit full of gauges

and instead of trying to read the whole thing, you feel it.

We live in a world of information now,

and there is a difference between accessing big data

and experiencing it.

So I think there's really no end to the possibilities

on the horizon for human expansion.

Just imagine an astronaut being able to feel

the overall health of the International Space Station,

or, for that matter, having you feel the invisible states of your own health,

like your blood sugar and the state of your microbiome,

or having 360-degree vision or seeing in infrared or ultraviolet.

So the key is this: As we move into the future,

we're going to increasingly be able to choose our own peripheral devices.

We no longer have to wait for Mother Nature's sensory gifts

on her timescales,

but instead, like any good parent, she's given us the tools that we need

to go out and define our own trajectory.

So the question now is,

how do you want to go out and experience your universe?

Thank you.

(Applause)

Chris Anderson: Can you feel it? DE: Yeah.

Actually, this was the first time I felt applause on the vest.

It's nice. It's like a massage. (Laughter)

CA: Twitter's going crazy. Twitter's going mad.

So that stock market experiment.

This could be the first experiment that secures its funding forevermore,

right, if successful?

DE: Well, that's right, I wouldn't have to write to NIH anymore.

CA: Well look, just to be skeptical for a minute,

I mean, this is amazing, but isn't most of the evidence so far

that sensory substitution works,

not necessarily that sensory addition works?

I mean, isn't it possible that the blind person can see through their tongue

because the visual cortex is still there, ready to process,

and that that is needed as part of it?

DE: That's a great question. We actually have no idea

what the theoretical limits are of what kind of data the brain can take in.

The general story, though, is that it's extraordinarily flexible.

So when a person goes blind, what we used to call their visual cortex

gets taken over by other things, by touch, by hearing, by vocabulary.

So what that tells us is that the cortex is kind of a one-trick pony.

It just runs certain kinds of computations on things.

And when we look around at things like braille, for example,

people are getting information through bumps on their fingers.

So I don't think we have any reason to think there's a theoretical limit

that we know the edge of.

CA: If this checks out, you're going to be deluged.

There are so many possible applications for this.

Are you ready for this? What are you most excited about, the direction it might go?

DE: I mean, I think there's a lot of applications here.

In terms of beyond sensory substitution, the things I started mentioning

about astronauts on the space station, they spend a lot of their time

monitoring things, and they could instead just get what's going on,

because what this is really good for is multidimensional data.

The key is this: Our visual systems are good at detecting blobs and edges,

but they're really bad at what our world has become,

which is screens with lots and lots of data.

We have to crawl that with our attentional systems.

So this is a way of just feeling the state of something,

just like the way you know the state of your body as you're standing around.

So I think heavy machinery, safety, feeling the state of a factory,

of your equipment, that's one place it'll go right away.

CA: David Eagleman, that was one mind-blowing talk. Thank you very much.

DE: Thank you, Chris. (Applause)