TrembleCV: Sensing Hand Tremors by Way of Computer Vision on Smart Eyeglasses
28th Annual Mersivity / WaterHCI Symposium 2026
Toronto and online, March 30 to May 10, 2026 · Published in the symposium proceedings (open access on Zenodo)
Figure 1 of the paper: the dual-stream forecaster (hand-landmark history plus the headband's IMU window), the per-activity error with and without the IMU, and the head-acceleration spectrum.
Hand tracking on smart glasses fails when the hands blur, leave the frame or get hidden. This paper asks whether the head-worn motion sensor you already have, here the IMU in a Muse S Athena headband, can help predict where the hand landmarks will be a second from now. A small two-branch network says yes: about 7 percent less error overall, more when the camera is struggling, and it runs in half a millisecond on a phone-class CPU. The data came from Meta Ray-Ban glasses during handwriting, cooking and drawing.
Abstract
Smart-glasses interfaces require hand-landmark predictors that are fast, small, and robust to egocentric visual failure. This paper studies whether a head-mounted inertial measurement unit (IMU) can improve one-second-ahead forecasting of MediaPipe hand landmarks when fused with recent visual keypoint history. We evaluate a compact dual-branch GRU that combines six frames of hand landmarks with a synchronized six-axis IMU window from a Muse S Athena headband. Across five temporal cross-validation folds from five participants and seven everyday activities, the spectral IMU model reduces mean per-joint position error from 104.8 to 96.4 pixels on a 1024 × 576 image plane, a 7.4% relative reduction. The gain is positive in every fold, disappears when the IMU is temporally shuffled, and grows from 6.2% to 11.9% as visual keypoint failure increases. Median inference time is 0.49 ms on a two-thread ARM-equivalent CPU proxy, supporting practical on-device use.
Authors
Mo Yang, Xiaoming Chen, Alexander Vicol, Steve Mann
Links
- Paper in the Mersivity / WaterHCI 2026 Symposium Proceedings (Zenodo, open access)
- Related project: Sousveillant Eyewear
- Related project: Neuromatic
- Related project: XR Cooking