Pillow App Science Test: Apple Watch Sleep Review

Summary

In this video, Rob, a postdoctoral scientist in Vienna, compares the sleep tracking capabilities of the Pillow app on the Apple Watch against a scientific EEG device. Over 10 nights, he records sleep stages and movements, analyzing the accuracy of sleep stage detection. Results show the Pillow app often confuses deep, light, and REM sleep stages, with only 13.9% of actual REM sleep correctly identified. Awake detection performs well, but the app sometimes registers sleep while the user is still awake or not in bed. Rob concludes that other Apple Watch sleep apps, like Sleep Cycle and AutoSleep, offer better tracking.

Takeaways

• 🧪 Rob, a postdoctoral scientist in Vienna, conducted a 10-night test comparing the Pillow app on Apple Watch to a scientific EEG device for sleep tracking.
• 📱 The Pillow app tracks sleep stages including deep sleep, light sleep, REM sleep, and wakefulness, and provides a sleep score and heart rate analysis.
• 🔬 Rob manually scored his sleep stages from EEG recordings and compared them to the Pillow app's data, noting discrepancies in sleep stage detection.
• 📈 The Pillow app showed poor correlation with the EEG device, particularly in detecting REM sleep, which was often misclassified as light or deep sleep.
• 🛌 Awake detection by the Pillow app was accurate, with only slight delays in detecting sleep onset and wake times.
• 🔄 The app seemed to follow hard-coded patterns, which may have contributed to the fragmented sleep stages observed.
• 📊 Statistical analysis revealed significant confusion between sleep stages by the Pillow app, with REM sleep particularly misidentified.
• 🆚 When compared to other Apple Watch sleep apps, the Pillow app was found to be less accurate and informative.
• 💤 Rob did not use the premium features of the Pillow app, which might offer additional insights but are behind a paywall.
• 📝 Limitations of the study include the small sample size, lack of a full polysomnography setup for comparison, and the subjective nature of manual sleep stage scoring.

Q & A

What is the main purpose of the Pillow app?

The Pillow app is used in combination with the Apple Watch to track sleep stages, including deep sleep, light sleep, REM sleep, and awake time. It also provides a sleep score, heart rate analysis, and helps users take naps.

What scientific device did Rob use to compare the accuracy of the Pillow app?

Rob used a portable scientific EEG device called the Hypnoline Z-Max, which is used by several of his colleagues in scientific studies, to compare the accuracy of the Pillow app.

How long did Rob wear both the Apple Watch with the Pillow app and the EEG device?

Rob wore both the Apple Watch with the Pillow app and the EEG device for 10 nights to collect data for comparison.

What did Rob manually record and analyze from the EEG device and the Pillow app?

Rob manually went through the recordings of the EEG device and scored each part of the night for different sleep stages. He also manually went through the Pillow app sleep stages and noted them down in an Excel table for analysis.

How did Rob evaluate the Pillow app's detection of when he fell asleep and woke up?

Rob used an infrared camera recording to check his movements and see if the Pillow app correctly predicted when he was awake. He also evaluated the app's automatic detection of the moment he fell asleep.

What were the main issues Rob found with the Pillow app's sleep stage tracking?

Rob found that the Pillow app had a poor match for deep sleep and REM sleep detection, often confusing these stages with light sleep. The app also had issues detecting sleep cycles accurately.

How did the Pillow app perform in detecting the times Rob was awake?

The Pillow app performed quite well in detecting when Rob woke up during the night, although there was a slight delay in detecting the start of sleep.

Did Rob find any patterns in the Pillow app's sleep stage tracking?

Yes, Rob noticed that the Pillow app's algorithm seemed to have hard-coded rules, such as always preceding deep sleep with light sleep and following any sleep stage with light or wake.

What were the statistical findings from Rob's comparison of the Pillow app and the EEG device?

The Pillow app predicted almost double the amount of deep sleep, about half the amount of light sleep, and more than double the awake time compared to the EEG device. It also confused most REM sleep with deep and light sleep.

How does the Pillow app compare to other Apple Watch sleep apps according to Rob's tests?

Rob found that other Apple Watch apps like AutoSleep and Sleep Cycle performed better in sleep tracking. He plans to make a dedicated video comparing different Apple Watch sleep apps soon.

What are some limitations Rob mentioned in his analysis of the Pillow app?

Rob mentioned that he only tested the app on himself for 10 nights, which is a limited sample size. He also noted that a full scientific polysomnography setup would be needed for a complete sleep comparison, and he is not a professional sleep stage scorer, which could introduce some subjectivity.