← All research projects

TrembleCV: Sensing Hand Tremors by Way of Computer Vision on Smart Eyeglasses

28th Annual Mersivity / WaterHCI Symposium 2026
Toronto and online, March 30 to May 10, 2026 · Published in the symposium proceedings (open access on Zenodo)

Figure 1 of the paper: the dual-stream skeleton and head-IMU forecaster, per-activity error bars with and without the IMU, and the head-acceleration spectrum

Figure 1 of the paper: the dual-stream forecaster (hand-landmark history plus the headband's IMU window), the per-activity error with and without the IMU, and the head-acceleration spectrum.

Hand tracking on smart glasses fails when the hands blur, leave the frame or get hidden. This paper asks whether the head-worn motion sensor you already have, here the IMU in a Muse S Athena headband, can help predict where the hand landmarks will be a second from now. A small two-branch network says yes: about 7 percent less error overall, more when the camera is struggling, and it runs in half a millisecond on a phone-class CPU. The data came from Meta Ray-Ban glasses during handwriting, cooking and drawing.

Abstract

Smart-glasses interfaces require hand-landmark predictors that are fast, small, and robust to egocentric visual failure. This paper studies whether a head-mounted inertial measurement unit (IMU) can improve one-second-ahead forecasting of MediaPipe hand landmarks when fused with recent visual keypoint history. We evaluate a compact dual-branch GRU that combines six frames of hand landmarks with a synchronized six-axis IMU window from a Muse S Athena headband. Across five temporal cross-validation folds from five participants and seven everyday activities, the spectral IMU model reduces mean per-joint position error from 104.8 to 96.4 pixels on a 1024 × 576 image plane, a 7.4% relative reduction. The gain is positive in every fold, disappears when the IMU is temporally shuffled, and grows from 6.2% to 11.9% as visual keypoint failure increases. Median inference time is 0.49 ms on a two-thread ARM-equivalent CPU proxy, supporting practical on-device use.

Authors

Mo Yang, Xiaoming Chen, Alexander Vicol, Steve Mann

Links

← All research projects