XRaudinary: Spatially Anchored Live XR-Captions via Low-Cost Microphone Arrays and Vision Fusion
IEEE 39th International Symposium on Computer-Based Medical Systems (CBMS) 2026
Limassol, Cyprus, June 2026 · Published; full paper on IEEE Xplore
The wearable microphone array: synchronized I2S MEMS microphones on an ESP32, which streams to a server that estimates direction of arrival.
People with hearing loss carry a heavy cognitive load in group conversations, especially in noisy rooms. XRaudinary anchors live captions to the direction the sound came from, inside the wearer’s field of view.
A wearable, low-cost array of MEMS microphones on an ESP32 streams synchronized audio to a server that estimates time differences of arrival; constraining the source to the plane of the user’s vision resolves the ambiguities of a small array, and the caption is drawn at the right place in a VR/AR headset. An in-progress system with the architecture, localization approach and early results.
A caption anchored to the speaker in the headset view.
Abstract
People with hearing loss often face high cognitive load and reduced participation in group conversations, especially in noisy or reverberant environments. We present XRaudinary, an in-progress XR system that spatially anchors live captions to the direction of sound within the user’s vision, by combining a wearable, low-cost microphone array with the user’s real-time vision. An ESP32 Microcontroller Unit (MCU) ingests synchronized I2S MEMS microphone streams, forwarding them to a server that estimates time-differences-of-arrival for direction-of-arrival inference, and forwards post-processed directionality and captions to a VR/AR headset. Constraining the sound along the planar axis of the user’s vision resolves geometric ambiguities inherent to small arrays, enabling real-time captions that appear at the correct location in the user’s field of view. We describe the system architecture, sound source localization approach, and appropriateness for real-time conversational contexts.
Authors
Andy Gong, Matthew Tamura, Efe Tascioglu, Andrew Wu, Alexander Vicol, Steve Mann
Links
- Paper on IEEE Xplore (DOI 10.1109/CBMS69103.2026.00118)
- Related project: ColourSenseXR
- Related project: EEG Word Keyboard