Future Interfaces Group: The next phase of computer-human interaction

Engadget

17 Dec 201806:53

Summary

TLDRThe Future Interfaces Group at Carnegie Mellon University develops innovative human-computer interaction technologies, exploring new ways to communicate beyond traditional inputs like keyboards and touchscreens. Their projects include transforming smartwatches into advanced input devices through high-speed accelerometers and leveraging environmental sound recognition for contextual awareness. They also experiment with camera-based systems for real-time monitoring in smart environments. These cutting-edge innovations aim to create more intuitive, assistive technology. However, the team acknowledges the challenges of security, privacy, and practicality in the adoption of these technologies at scale.

Takeaways

📱 Over 100 million devices can now distinguish between knuckle and finger touches and detect when they are lifted to the ear.
🤖 The Future Interfaces Group (FIG) at Carnegie Mellon University, established in 2014, explores new modes of human-computer interaction.
💡 FIG is sponsored by major tech companies like Google, Intel, and Qualcomm, focusing on speculative and experimental technologies.
🖥️ The lab's vision includes creating intelligent environments where smart devices have contextual awareness and can assist users more naturally.
👂 One project, Ubiquoustics, enables devices to listen to ambient sounds, like chopping vegetables or blending, to understand their context.
⌚ Another innovation involves transforming smartwatches into versatile input devices using high-speed accelerometers to detect micro-vibrations.
🖐️ Gesture-based interaction is being explored, such as snapping fingers to control lights or clapping to activate devices.
📸 The lab also explores turning cameras into sensors, enabling smart environments to recognize objects or people without active human monitoring.
🚗 FIG is testing real-time parking solutions using camera-based technology to reduce congestion and pollution in cities.
⚖️ FIG balances technological innovation with privacy concerns, recognizing that no system is 100% secure and that trust hinges on the perceived value of new technologies.

Q & A

What is the Future Interfaces Group (FIG) and where is it located?
-The Future Interfaces Group (FIG) is a research lab at Carnegie Mellon University in Pittsburgh, Pennsylvania, that focuses on human-computer interaction. It was founded in 2014 and works on speculative projects to improve communication between humans and machines beyond traditional methods like keyboards and touchscreens.
What are some examples of projects developed by FIG?
-Examples of FIG projects include touchscreens that can detect if you're using a knuckle or finger, and devices that can recognize if you're lifting the phone to your ear. The lab works on ideas that expand how machines interact with humans, using sensors, sound recognition, and other contextual cues.
What is the grand vision of the FIG lab?
-The grand vision of the FIG lab is to create intelligent environments where devices, like smart speakers or watches, are aware of their surroundings and can interact with humans using nonverbal cues, such as gestures, gaze, and sounds, similar to how humans communicate with each other.
How does the FIG lab increase implicit input bandwidth in devices?
-FIG increases implicit input bandwidth by enhancing devices' ability to understand contextual information. For example, they use sound recognition to determine activities in a room, such as distinguishing between chopping vegetables or running a blender, so that devices can better assist users.
What is the uBaku Stiix project and how does it work?
-The uBaku Stiix project uses sound recognition to understand what is happening in an environment. By training computers to recognize distinctive sounds, like chopping vegetables or using a blender, the project explores how devices can use microphones to gather contextual information about their surroundings.
How does the lab use smartwatches in its research?
-The lab experiments with smartwatches by increasing their sensitivity. They overclock the accelerometer in a smartwatch to detect micro-vibrations, which allows the watch to sense subtle interactions, such as finger taps, transforming the watch into an input platform for controlling devices like lights or TVs using gestures.
What is Sensors, and how does it utilize camera feeds?
-Sensors is a startup that uses camera-based technology to turn existing cameras in public places into sensor feeds. This system can recognize actions like counting the number of people on a sofa or identifying objects like laptops or phones, helping automate monitoring in places like libraries, restaurants, or streets.
How is FIG’s camera-based technology applied in real-world scenarios?
-FIG’s camera-based technology is applied in real-world scenarios like real-time parking management, where cameras count available parking spaces and guide drivers to open spots, helping reduce congestion and pollution. This technology uses existing infrastructure to provide practical solutions.
What challenges do technologies developed by FIG face when moving from research to commercialization?
-Technologies developed by FIG face challenges such as practicality, feasibility, and ensuring security and privacy when transitioning from research to commercialization. These technologies must balance innovation with real-world impact and user concerns, especially in terms of data privacy and usability.
How does FIG address privacy and security concerns in its projects?
-FIG acknowledges that no technology is 100% secure or privacy-preserving. The lab focuses on making technologies that strike the right balance between innovation and privacy, ensuring that users understand and accept the potential trade-offs when adopting new devices with microphones, sensors, or cameras.

Outlines

00:00

📱 The Future of Human-Computer Interaction

This paragraph introduces how modern smartphones have advanced to detect different inputs, such as distinguishing between a knuckle or finger touch and recognizing when the phone is lifted to the ear. These innovations are part of ongoing projects at the Future Interfaces Group Lab at Carnegie Mellon University. Sponsored by tech giants like Google, Intel, and Qualcomm, the lab explores futuristic ways humans can interact with machines, beyond traditional interfaces like keyboards and touchscreens.

05:01

🔬 Creating the Future Interfaces Lab

The founder of the Future Interfaces Group, who joined Carnegie Mellon University (CMU) five years ago, reflects on setting up the lab. His research focused on using the human body as an interactive computing surface. With the support of students and researchers, the lab continues to push the boundaries of human-computer interaction. The grand vision involves creating intelligent environments where devices like smartphones and smart speakers understand contextual human interactions, similar to how humans use non-verbal cues in communication.

👂 Increasing Implicit Input for Devices

The lab's focus is on increasing devices' ability to understand their environment, often referred to as 'implicit input bandwidth.' One project, called Ubiquitous Acoustics (uBaku Stiix), enables devices to recognize environmental sounds to infer what’s happening, like detecting if someone is chopping vegetables or blending food. Using sensors like microphones already built into devices, they explore ways to collect and process contextual data affordably.

⌚ Transforming Devices with Enhanced Sensors

Smartwatches, which are highly capable computers, are the focus of another lab project. By overclocking the accelerometer in a smartwatch, the lab increases its ability to capture detailed micro-vibrations, enabling the watch to interpret subtle gestures and movements. This could allow users to control home devices through gestures, like snapping to turn on lights or clapping to control a TV. The lab develops hundreds of similar ideas each year, with a few transforming into startups.

📸 Turning Cameras into Sensors for Smart Environments

Some of the lab’s projects, such as 'sensors,' leverage existing technology, like cameras, to create smarter environments. They explore using video feeds to automatically analyze environments in real-time, for example, detecting how many people are present in a room or recognizing objects like laptops on a table. This approach could be used in public spaces or homes to offer smarter, real-time monitoring without human intervention.

🚗 Smart Cities and the Future of Parking

The lab collaborates with cities, such as using camera-based sensors to count cars and help solve real-world problems like parking. By utilizing existing infrastructure like traffic cameras, the system could potentially guide drivers to available parking spots, reducing congestion and air pollution. However, the success of such technologies depends on balancing practicality, feasibility, and cost with long-term value.

🔐 Ethical Considerations in Technology Adoption

The closing discussion addresses the ethical implications of advancing technologies, such as using surveillance cameras for sensing. While the research offers exciting possibilities, privacy and security concerns are inevitable. The lab believes that no technology can be 100% secure, but by making the right trade-offs and involving users in the design process, people may accept potential privacy risks if they see clear benefits.

Mindmap

Keywords

💡Future Interfaces Group

The Future Interfaces Group (FIG) is a research lab at Carnegie Mellon University, focused on developing new forms of human-computer interaction. Founded in 2014, the lab creates speculative and innovative ideas that explore how we communicate with machines, aiming to go beyond traditional input methods like keyboards and touchscreens. The lab is sponsored by major tech companies like Google, Intel, and Qualcomm.

💡Human-computer interaction

Human-computer interaction (HCI) refers to how humans communicate and interact with computers and other digital systems. The video emphasizes new ways to enhance this interaction, not just through conventional means like voice commands or touchscreens, but by developing systems that understand contextual and environmental cues, such as sound or gestures, to improve the user experience.

💡Implicit input bandwidth

Implicit input bandwidth refers to the ability of a device to gather contextual information from its surroundings to enhance interaction without direct input from the user. This concept is central to the video, where it is discussed in terms of expanding how devices like smartphones or smartwatches interpret environmental cues (e.g., sound or motion) to offer more natural and responsive interactions.

💡Contextual understanding

Contextual understanding in the video refers to a device's ability to recognize and interpret its environment, such as sound or motion, to anticipate and respond to user needs. This is key to advancing human-computer interaction beyond current methods. For example, a device that can identify sounds like chopping vegetables or blending in a kitchen could offer relevant assistance or features without being explicitly prompted.

💡Ubiquitous computing

Ubiquitous computing, also known as pervasive computing, is the concept of embedding computational capability into everyday objects and environments, making them 'smart.' The video explores how future devices will integrate seamlessly into daily life, collecting information from surroundings and reacting accordingly, such as smartwatches detecting micro-vibrations or sound to perform tasks.

💡Sensor fusion

Sensor fusion refers to combining data from multiple sensors to make more accurate assessments of a situation or environment. In the video, this is demonstrated with smartwatches that use various sensors to detect gestures, micro-vibrations, and other contextual cues, improving interaction. The technology is used to enhance how machines interpret complex human behavior and environmental signals.

💡Smart environments

Smart environments are physical spaces equipped with devices and sensors that gather and respond to data in real time to improve the user's experience. The Future Interfaces Group explores how such environments can assist users by gathering contextual information, such as using cameras in public spaces to monitor occupancy or integrating gesture-based controls for home devices like lighting and TVs.

💡Speculative design

Speculative design involves creating prototypes or concepts that push the boundaries of current technology to explore future possibilities. The Future Interfaces Group engages in speculative design by developing hundreds of ideas each year, some of which become startups, such as the technology behind touchscreen improvements or computer vision systems like Sensors, which aim to enhance smart environments.

💡Gesture recognition

Gesture recognition technology allows devices to interpret and respond to human gestures as a form of input. In the video, gesture recognition is exemplified by the use of smartwatches that can detect movements like snapping or clapping to control lights or other home devices. This is part of a broader effort to expand how users interact with machines in more intuitive ways.

💡Computer vision

Computer vision is a field of artificial intelligence that enables machines to interpret and process visual data from the world. The video discusses how the Sensors startup, developed by the Future Interfaces Group, uses cameras to detect and count objects or people in public spaces. This technology can be applied to real-time tasks like monitoring parking availability or determining the occupancy of a room.

Highlights

Over a hundred million phones can now detect if you're using your knuckle or finger to touch the screen, as well as whether you're lifting the device to your ear.

The Future Interfaces Group Lab at Carnegie Mellon University in Pittsburgh, Pennsylvania, has been pioneering advancements in human-computer interaction since 2014.

The lab is backed by major sponsors like Google, Intel, and Qualcomm and develops hundreds of speculative ideas each year to enhance communication between humans and machines.

The group’s focus is on creating new modes of interaction beyond keyboards, touchscreens, mice, or even voice commands.

Founder of the lab envisions intelligent environments where devices like Google Home or smartwatches can have full contextual awareness, similar to human assistants.

A key area of research is enhancing 'implicit input bandwidth,' enabling devices to gather contextual information about their environment, like sound-based understanding.

The Ubakustix project trains computers to use microphones to understand environmental sounds, such as identifying kitchen activities like chopping vegetables or running a blender.

Research in the lab shows how smartwatches can be transformed into high-precision devices by overclocking their accelerometers, detecting micro-vibrations and enabling gesture-based controls.

Innovative smart gestures, such as snapping fingers or twisting a wrist, allow users to control lights or navigate menus through gestures alone.

Several of the lab’s projects have led to real startups, such as Kiko (touchscreen technology) and Sensors (a computer vision startup).

Sensors uses cameras in public environments, like restaurants or streets, to turn video feeds into sensor data, identifying objects, people, or even parking spaces.

Real-time parking systems are being piloted using existing city cameras, aiming to direct drivers to available spaces, reducing congestion and pollution.

The lab’s approach emphasizes practical and scalable solutions that can transition from research into real-world applications.

Challenges remain with security and privacy as cameras and microphones gain contextual awareness; however, the lab prioritizes designing technologies that balance benefits and privacy concerns.

The lab actively engages users in testing and feedback, ensuring that the value proposition of new technologies is clear, increasing the likelihood of adoption.