ColourSenseXR: Helping the Hearing-Impaired Perceive Sound Through Colour
IEEE International Conference on Consumer Electronics (ICCE) 2026
Dubai, United Arab Emirates, February 2026 · Published; full paper on IEEE Xplore
A violin through ColourSenseXR: the passthrough view takes on the hue and brightness mapped from the sound.
ColourSenseXR turns sound into colour on a Meta Quest 2 in passthrough mode so that people with hearing impairments can see and interpret the sounds around them. Two translation strategies are built and compared: a rule-based note-to-colour mapping, where pitch sets hue and brightness, and an AI emotion-to-colour mapping, where a convolutional network estimates the emotion of environmental and musical sound and paints it as moving colour gradients, including valence-arousal models that work on short audio windows.
In a user study, note-to-colour was easiest for following pitch while emotion-to-colour was preferred for context and mood.
The same scene under two sounds.
Abstract
We present a real-time extended reality (XR) system, called ColourSenseXR, designed to enhance sensory perception through the integration of extended intelligence and extended reality. Using a Quest 2, we create a system to connect individuals with each other and the environment. Our goal is to help those with hearing impairments see and interpret sounds in daily life. ColourSenseXR creates a colour overlay from audio, allowing users to perceive sound visually, using passthrough mode on virtual/extended reality (VR/XR) headsets. The system implements and compares two audio-to-visual translation strategies. The first strategy uses rule-based mapping, and converts varying pitches directly into colour with varying hues and brightness levels, providing interpretable, frequency-driven feedback. The second strategy uses an Artificial Intelligence (AI)-driven emotion recognition model trained through a convolutional neural network (CNN) to estimate emotion from environmental and musical sounds, which are then visualized as dynamic colour gradients. In addition to a clip-level CNN trained on full songs, we further design and evaluate dynamic valence-arousal models that operate on short audio windows and temporal sequences, enabling finer-grained, time-varying colour feedback. A user study conducted on a Meta Quest 2 evaluates both methods in terms of intuitiveness, emotional expressiveness, and environmental awareness. These results suggest that note-to-colour is easiest for following pitch, while emotion-to-colour is preferred for context and mood communication.
Authors
Yawen Xiao, Wanhe Li, Richard Zeng, Alexander Vicol, Xiaoming Chen, Mo Yang, Steve Mann
Links
- Paper on IEEE Xplore (DOI 10.1109/ICCE67443.2026.11449648)
- Also in the Mersivity / WaterHCI December-2025 Symposium Proceedings (Zenodo)
- Related project: XRaudinary
- Related project: ELEVATE: XR for Elevators