WWDC19: Designing Audio-Haptic Experiences | Apple

Summary

This video presentation explores the design of audio-haptic experiences, focusing on how sound and haptics can enhance app interactions. Led by Camille Moussette and Hugo Verweij from Apple's design team, the session introduces the concept of audio-haptic experiences, the Core Haptics API, and key principles like causality, harmony, and utility. Practical techniques for designing immersive and intuitive audio-haptic feedback are discussed, with examples from Apple products like the iPhone and Apple Watch. The goal is to inspire developers to create seamless, engaging user experiences through thoughtful sound and haptic integration.

Takeaways

• 😀 An audio-haptic experience is a combination of sound and haptic feedback that enhances user interaction, making it more immersive.
• 😀 The Taptic Engine in iPhones is designed to play low-frequency haptics that can only be felt, not heard, adding a new layer to the sensory experience.
• 😀 Core Haptics API in iOS 13 allows developers to create custom haptic feedback, offering control over intensity, sharpness, and duration.
• 😀 Causality is a key design principle, ensuring that feedback is clearly linked to the user's action, like a soccer ball's impact sound and feel.
• 😀 Harmony ensures that sound, haptics, and visuals work together seamlessly, creating a unified and realistic user experience.
• 😀 Utility emphasizes the importance of using sound and haptics only when they add clear value to the app experience, avoiding unnecessary feedback.
• 😀 Experimenting with sound and haptics can lead to surprising and better results, such as reversing the order of haptic feedback to enhance the user experience.
• 😀 The Apple Watch crown provides an example of precise synchronization of sound and haptics, creating a realistic mechanical feel.
• 😀 When designing experiences, it's important to consider the dynamics of an object and its interaction, ensuring the sound and haptics are appropriate.
• 😀 Effective audio-haptic feedback requires careful consideration of timing, intensity, and the physical qualities of the object being represented, such as the right sound for a ball's bounce.

Q & A

What is an audio-haptic experience?

An audio-haptic experience combines sound and haptic feedback, where both the audio and tactile sensations are synchronized to create a more immersive interaction. This is often used in devices like the iPhone to enhance user experiences through the Taptic Engine.

How does the Taptic Engine contribute to audio-haptic experiences?

The Taptic Engine in devices like the iPhone is designed to play low frequencies that users can feel but not necessarily hear. It works in sync with the sound output to create a tactile experience that complements the audio, offering a deeper interaction.

What is Core Haptics, and what capabilities does it offer?

Core Haptics is an API introduced in iOS 13 that allows developers to create custom haptic experiences. It provides control over the intensity and sharpness of the haptic feedback, supporting both continuous and transient experiences, giving developers the tools to design precise and dynamic tactile feedback.

What are the three guiding principles in designing audio-haptic experiences?

The three guiding principles are causality, harmony, and utility. Causality ensures feedback is clearly linked to its cause, harmony ensures audio, haptics, and visuals work together, and utility ensures sound and haptic feedback add value to the user experience.

Can you explain the principle of causality with an example?

Causality refers to the need for feedback to make sense in the context of the interaction. For example, when kicking a soccer ball, the sound and haptic feedback should match the physical impact of the kick. The type of sound and sensation should match the characteristics of the objects involved, such as the shoe and ball.

How does harmony play a role in audio-haptic design?

Harmony ensures that the audio, haptic feedback, and visuals are synchronized and feel consistent. For instance, when designing a bouncing sphere animation, the sound and haptics should reflect the physics of the bounce, with the intensity of both matching the visual movement to create a cohesive experience.

What is the importance of utility in audio-haptic feedback design?

Utility emphasizes the need to add sound and haptic feedback only when they offer clear value to the user experience. It suggests using moderation to avoid overwhelming users with unnecessary feedback. For example, in an AR app, the feedback might only change when the user moves closer to or away from an object, enhancing the interaction without overcomplicating it.

What should developers consider when designing sound for a haptic experience?

Developers should think about the type of haptic feedback being used (sharp transient or continuous) and choose a sound that complements it. A sharp, precise sound works well with sharp haptic feedback, while a softer sound might pair better with a continuous, rumbling haptic experience.

Why is synchronization between audio, haptics, and visuals important?

Synchronization between the three is crucial to create a realistic and seamless experience. If there is latency between the feedback elements, the illusion of interaction is broken, as seen in the example where delayed audio or haptic feedback disrupts the bounce animation of a sphere.

How can developers use the 'ghost haptic' effect in their designs?

The 'ghost haptic' effect occurs when a user doesn't fully perceive the first haptic feedback in a sequence, which can be used deliberately as a priming effect. For example, when a user receives an important notification, the first weak haptic feedback prepares them to better perceive the main, more significant feedback.