I Made a Mind-Controlled Game Controller

Summary

The script details an individual's innovative journey to develop a mind-controlled gaming experience using EEG technology and eye tracking. After overcoming numerous challenges, including technical setbacks and the limitations of thought recognition, the creator successfully builds a hands-free game controller. The project not only showcases the potential of brain-computer interfaces but also the creator's resilience and the profound appreciation for those who cannot game conventionally.

Takeaways

• 🧠 The creator has a background in Psychology and experience with EEG for medical and research purposes.
• 🎮 The project aimed to merge gaming with EEG technology to control games using brain activity.
• 🛍️ The discovery of a user-friendly EEG device was a significant enabler for the project.
• 🔬 The initial plan involved using EEG to record brain signals during different emotional states in games.
• 📈 The creator utilized software that came with the EEG to record and recognize different thinking patterns.
• 🔄 After many attempts, the project faced challenges in getting the EEG to consistently recognize thoughts for in-game actions.
• 🎉 A breakthrough came with the recognition of imagining lifting a cube, which boosted confidence in the project.
• 🛠️ The project evolved to code the EEG signals to perform in-game actions like left-clicking in Minecraft.
• 👀 The creator encountered issues with maintaining focus and eye movement while trying to control the game.
• 🤔 A shift in strategy led to considering more universal visualizations for game actions, like pushing something forward.
• 🖱️ The introduction of an eye tracker allowed for hands-free movement and camera control in 3D spaces.
• 🔄 Despite successes, the project hit a low point with difficulties in distinguishing between different mental commands.
• 📞 A consultation with the EEG manufacturer and a team from Ear Switch provided insights into multisensory brain activation.
• 🆕 Starting anew with mental command training, the creator implemented a system with virtual joysticks and buttons.
• 🎲 The final setup included a combination of eye tracking and four mental commands for a hands-free gaming experience.
• 🎉 The project concluded with a successful demonstration of playing a game using the mind-controlled setup.
• 💡 The creator gained a deep appreciation for the complexity of controlling thoughts and the potential of brain-computer interfaces.

Q & A

What is the main concept of the video?

The video discusses the creator's experiment to develop a hands-free game controller using EEG and eye-tracking technology to control games with thoughts and eye movements.

What background does the creator have that inspired this project?

The creator has a background in Psychology and experience with using EEG for medical and research purposes, which inspired the idea to combine gaming with brainwave technology.

What was the initial plan for using EEG in the project?

The initial plan was to use EEG to record the brain's electrical signals to analyze the creator's brain activity during different gaming experiences, such as getting jump scared or being relaxed.

How did the creator find out about user-friendly EEG devices?

The creator discovered user-friendly EEG devices while browsing the internet, which was a significant revelation that excited them about the potential applications of this technology.

What software feature of the EEG device did the creator utilize to recognize brain activity?

The creator used the EEG's software that allows recording of brain activity while thinking about different things, which helped in recognizing specific thoughts for controlling the game.

What game was chosen to test the mind-controlled gaming concept?

The creator chose Minecraft as the game to test the initial concept of mind-controlled gaming due to its simplicity and familiarity.

What was the main challenge faced when trying to control the game with thoughts alone?

The main challenge was maintaining focus and not losing the mental image needed to control the game, as the eyes would drift and the creator couldn't hold the image long enough for continuous control.

How did the creator address the issue of needing more commands for a more universal gaming experience?

The creator decided to find a universal visualization that could be applied to different games, such as imagining pushing or moving something forward, which was easier to maintain and less distracting.

What additional technology was integrated to enhance the mind-controlled game controller?

An eye tracker was integrated to create virtual joysticks on an Xbox controller, allowing the computer to know where the creator was looking and the direction of head tilt for camera or character movement.

What was the final outcome of the project after overcoming various challenges?

The final outcome was a hands-free mind-controlled game controller that included virtual joysticks and four mental commands for virtual buttons, making it a reliable and functional gaming tool.

What personal realization did the creator have after completing this project?

The creator realized the difficulty of controlling thoughts and emotions and gained a deep appreciation for those who are unable to game conventionally and the challenges they face daily.

Outlines

00:00

🧠 Mind-Powered Gaming Experiment

The script introduces a personal journey of combining gaming with psychology and EEG technology. The creator, with a background in Psychology, explores the possibility of using brainwave data from an EEG device to control in-game actions. Initially, the plan was to record brain activity during different emotional states, like fear or relaxation, but evolved into an attempt to control a game using thoughts alone. The creator leverages pre-existing software to recognize different mental commands, such as imagining actions like dropping something or turning directions, to manipulate an in-game cube. Despite initial failures, persistence leads to a breakthrough, enabling the lifting of the cube through thought alone, which then translates into in-game actions like left-clicking in Minecraft.

05:01

🎮 Advancing Mind-Controlled Gaming

Building upon the initial success, the creator aims to refine the mind-controlled gaming experience, facing challenges with maintaining focus and expanding the range of controllable actions. The script details the experimentation with different visualizations to improve control and the integration of an eye tracker to create virtual joysticks for camera movement. Despite setbacks, including technical issues with the EEG device, the creator remains committed, investing personal funds to continue development. The narrative emphasizes the complexity of distinguishing between similar brainwave patterns for different commands and the innovative approach of using chaos in visualization to create unique mental commands. The culmination of these efforts results in a more advanced mind-controlled game controller capable of basic in-game interactions.

10:02

🚀 Finalizing the Mind-Controlled Game Controller

The final paragraph describes the culmination of the creator's efforts to develop a hands-free, mind-controlled game controller. After a year of trials, errors, and learning, the creator incorporates four distinct mental commands and an eye tracker for a more comprehensive gaming experience. The journey highlights the difficulty of differentiating between complex thought patterns and the realization that incorporating various sensory inputs could enhance the control mechanism. The creator's collaboration with experts and the integration of these insights lead to a functional prototype that is tested in real-time with an audience. The successful demonstration of the controller in a game environment signifies not only a personal triumph but also a step forward in accessible gaming technology.

Keywords

💡EEG

EEG stands for Electroencephalography, a medical procedure that measures the electrical activity of the brain. In the context of this video, the creator uses an EEG device to record brain signals with the aim of controlling a game through thought. The script mentions 'using EEG for medical and research purposes' and the discovery of a 'userfriendly EEG' that spurred the project.

💡Psychology

Psychology is the scientific study of the human mind and its functions, particularly those affecting behavior in a given context. The creator of the video has a background in psychology, which influenced their interest in exploring the mind's capabilities in conjunction with gaming technology. The script references this background when discussing the initial idea for the project.

💡Gaming

Gaming refers to playing video games or participating in interactive electronic entertainment. The video's theme revolves around gaming, as the creator seeks to enhance the gaming experience through mind-controlled interfaces. The script illustrates this with phrases like 'a lot of people I love gaming' and the use of games like Minecraft and Elden Ring as examples.

💡Mind Control

Mind control, in this context, refers to the ability to manipulate technology or software using thoughts alone. The video's main theme is the development of a mind-controlled game controller. The script describes the process of 'using my thoughts' to control in-game actions and the challenges faced in distinguishing different thought patterns for control.

💡Visualization

Visualization is the process of forming a mental image or concept of something. In the script, the creator uses visualization techniques to train the EEG software to recognize different thoughts for controlling in-game actions, such as 'imagining lifting the cube' or 'pushing something forward'.

💡Mental Commands

Mental commands are thought patterns or images that the creator uses to issue instructions to the game. The video discusses the development of mental commands like 'pushing a cube' or 'spinning a plate' to control in-game actions. The script mentions 'training mental commands from scratch' to improve the mind-controlled interface.

💡Eye Tracker

An eye tracker is a device that tracks the movement of the eyes and interprets where a person is looking on a screen. In the video, the creator uses an eye tracker to create virtual joysticks for a hands-free gaming experience. The script describes using 'an eye tracker' to make 'virtual joysticks on an Xbox controller'.

💡Controller

A controller is a device used to manipulate the actions of a character or object within a video game. The video's goal is to create a mind-controlled game controller. The script discusses the limitations of using a traditional controller and the transition to a hands-free, mind-controlled system.

💡Programming

Programming involves writing code to create software or instruct a computer to perform specific tasks. The creator mentions programming their own AI model initially but later opts for pre-existing software. The script refers to 'coding' the system so that brain activity translates into in-game actions.

💡Innovation

Innovation refers to the process of creating new methods, ideas, or devices. The video is a narrative of innovation, where the creator experiments with combining EEG technology and gaming. The script captures this through phrases like 'a cool experiment to try and combine the two' and 'I was on to something'.

💡Accessibility

Accessibility in the context of this video pertains to making gaming technology usable for people who cannot use traditional controllers. The creator reflects on the potential of their mind-controlled controller to make gaming more accessible. The script concludes with thoughts on 'making it more accessible and available' for those who struggle with conventional gaming methods.

Highlights

Combining gaming with EEG technology to control games using brain signals.

Using a user-friendly EEG device to record brain activity for gaming purposes.

Initial challenges in getting the EEG to recognize specific thought patterns for game control.

Concept of proof achieved with the ability to lift a virtual cube using thoughts.

Coding the EEG data to translate thoughts into in-game actions, like a left mouse click.

The idea of using a universal visualization for different gaming actions.

Facing difficulties with maintaining focus and consistency in thought recognition.

Incorporating an eye tracker to create virtual joysticks for camera and character control.

Overcoming sensitivity issues with the eye tracker to improve control.

Creating a hands-free mind-controlled game controller with multiple commands.

The struggle with expanding the number of mental commands for more complex gaming.

Experimenting with non-visual mental commands to diversify brain activity recognition.

Collaboration with 'ear switch' for insights on different sensory inputs affecting brain activity.

Re-training mental commands from scratch to improve the usability of the game controller.

Integrating a variety of mental commands for a more functional gaming experience.

Successfully using the mind-controlled game controller in actual gameplay.

The personal realization of the difficulty in controlling thoughts and emotions.

Appreciation for those who cannot game conventionally and the potential of advanced tech.

The limitless possibilities of what can be achieved with the human brain and technology.

Transcripts

this is me playing games with my mind

after I developed a way to do this using

only my

thoughts I like

you okay

[Music]

stop and I'm going to show you how like

a lot of people I love gaming and since

I have a background in Psychology and

using EEG for medical and research

purposes I thought it might be a cool

experiment to try and combine the two at

the time I didn't record or make notes

on anything so there'll be a lot of

dramatic reenactments coming up my

original plan was to use EEG which

records the electrical signals your

brain produces to show what my brain was

doing when getting jump scared in a

horror for

example okay or when I was super relaxed

in a horror for example this was only

possible because one day when I was just

scrolling through the internet I

discovered that you could just buy a

userfriendly

EEG which already was mind-blowing

enough for me to get really excited

about all the things I could do with

this but I wanted to take it a step

further and see if I could use the data

the EEG was getting to actually control

the game so my plan was to use this data

in a program that would send a signal to

press keys on my keyboard but first I

needed to decide what data what part of

my brain activity would press each key

and I needed to teach it to detect and

recognize this so I could actually

control it but I was lucky enough that a

lot of this work had already been done

for me this EEG has a software that lets

you record your brain activity when

you're thinking about different things

this could not be more perfect and it's

such a better idea than me trying to

program my own AI model like I was

originally planning so I started

recording attempt after attempt at

trying to get it to recognize me

thinking about dropping something or

turning left or turning right or

spinning or dangling from a string if it

does recognize it this Cube will move

but nothing no matter what I tried I

couldn't get it to recognize my thoughts

until

[Applause]

that's a good start we have proof of

concept with this confidence boost I was

now more committed than ever and soon I

was able to consistently lift the cube

into the air by just imagining that so

the next thing to do was to code this so

that every time I was able to push the

cube it would push a button to do

something in game I thought the easiest

thing to do was a left click and

naturally when you think game you think

Minecraft

so it wasn't perfect but I was more

convinced than ever that I was on to

something the problem was that as I was

imagining lifting the cube my eyes would

drift upwards and I'd lose focus I just

couldn't hold on to the image for long

enough to mine an entire block so back

to the drawing board I started thinking

about what I actually wanted it to do

just thinking about mining would take a

lot of practice to get the same every

single time and I wanted a more un

Universal visualization so that I could

use it with different games but what did

mining have in common with hitting a

tree or casting a spell or even shooting

a gun they all involv moving or pushing

something forward this worked amazingly

it was easy to maintain and I didn't get

distracted as I wouldn't have to move my

eyes like this soon we had a mouse click

that was easy to control most of the

time so naturally it was time to take

this live and use it to play elen Ring y

alen ring why not I was so excited to

show everyone this insanely cool project

I had spent 3 months of hard work done

long nights and early mornings and blown

through my entire cool project savings

fund I was just about to go live

when it's a ribbon cable can we fix

it no but I wasn't going to give up

after a moment to grief I spent my

entire food and shelter fund and ordered

another

EEG this time with insurance and got

back to

work okay we're ready this was my second

and final shot of making this work if I

blew it I knew I couldn't afford to try

it

again somewhere to hide behind baby

no spurred on I instantly got started on

getting another visualization and

another button to work but when I I

trained one command the software would

get confused and forget the other one it

couldn't hold on to both simultaneously

what my brain was doing was just too

similar so to make the visualizations

different I tried to make the second

command as chaotic as possible imagining

music playing in the cube spinning far

so much it became more of a plate in my

mind and to my surprise this actually

worked one thing that always bothered me

about my mind control game controller

was the fact that I was still using my

hands and a literal controller the

reason for this was because there was no

way to fluidly move a camera or

character in a 3D space when I was just

using yes no thought recognition so I

thought if hands were the problem why

not try and use some other Tech this is

an ey tracker using this the computer

knows where I'm looking at on the screen

and also which direction my head is

tilting I used these two things to make

virtual joysticks on an Xbox controller

and created a Nifty little visual to see

where they were it worked but I I did

have to fix some sensitivity issues

and counteract the dead zones created

because of just the nature of using

joysticks now the only thing left to do

was combine them ladies and gentlemen we

have a handsfree mind controlled game

controller I don't know about you though

but I don't know many games where you

can play them with only one or two

buttons sure you could probably get away

with a Dark Souls boss or two if you

never healed or you can play a shooter

if you don't use grenades or mie or need

to switch which weapon ever or use any

sort of menu but that's not gaming I

needed more commands but no matter what

I did what I tried I just couldn't get

any it can't tell the difference between

my thoughts even if I try to imagine

something that I think is completely

different to anything else that I

trained before it will still get all

mushed together I toyed around with the

idea of using voice commands to fill in

any more buttons I really needed it does

mean though that when we go into the

menu

we don't have the ability to move this

way but this felt wrong it wasn't mind

control without more commands though

this would never be a game controller

and everything I had done felt all for

nothing more long nights and early

mornings desperate to come up with

something and nothing along the journey

this was probably my lowest point I just

can't make my brain do what I want it to

do and usually it's the tech or usually

it's the code but this time it's just me

which somehow feels a lot

worse I still had my phone a friend it

was time to reach out I'm about to hop

on with a motive the maker of the EEG

and software that I'm using but it's all

very NDA so you got to get out of

here after a lot of cool stuff the main

thing that I took from our discussion

was how much different sensors can

actually affect different parts of your

brain a visual idea is going to awaken

different areas than a smell for example

and all all of my commands had been

visual there was a whole world of

possibilities that I was just ignoring I

also spent a lot of time with a team

down at ear switch who are making an

inear gaming input device which got me

thinking about how I could use my head

in different ways so I decided to wipe

the whole slate clean and start training

mental commands from scratch to see how

good I could make this could it be a

usable game controller so here it was

the final plan I tring and dryer for

virtual joystick and four mental

commands for virtual buttons pushing a

cube spinning a plate imagining my fist

clenching while angry and feeling hot

and a little Cricket hopping while I

pulse muscles in my face I also added a

cute little drop- down menu for each of

these so I could make them press any

button I wanted at any time add an

option to toggle to spam or hold the

buttons and some sensitivity sliders for

the eye tracker and she was ready it's

ready to test and if this doesn't work

then I think I'm genuinely out of ideas

I might have to actually accept that I

just don't have the level of technology

or knowledge on how to make this work

right now which would be Soul crushing

after an entire year and hundreds of

hours of just training let alone

everything else and I would have to

break it to all of the really supportive

people that have been following this

journey and the people who are watching

it unfold live with a couple hundred

onlookers I started training I managed

to get a third input and then a

fourth but it's one thing to be able to

move a cube on a white screen could I

use this to actually game pick up this

[Music]

Rock I'm going to hit this

rock great I'm going to

jump and I'm going to try to throw a pal

but I don't have

any mind control and the final

[Music]

Test passing

[Music]

skin

[Music]

yeah

[Music]

so there it is I made a handsfree game

[Music]

[Music]

controller that's a reliable weapon

[Applause]

yeah this experience really taught me

how sometimes we can be so caught up in

our day-to-day lives that we don't

actually appreciate how hard it is to

really control our thoughts and our

emotions and even our Focus the amount

of appreciation that I gained for those

who are unable to game conventionally

and have probably struggled every single

day to fit in was immense since this

Tech is becoming more and more advanced

I feel so lucky to have been right in

the middle of its emergence and I can't

wait for it to become more accessible

and available we take all this for

granted but our brains are amazing and

the things we can do with them are

[Music]

Limitless