Telekolympographic Swim Gesture Recognition for Teleoperation of Remote or Virtual Boat
IEEE International Conference on Mechatronics and Automation (ICMA) 2026
Changchun, China, August 2026 · Published; full paper on IEEE Xplore
The Ship of Kolympi, from the Peter Street Basin Master Plan: a vessel propelled and steered by swimming in a pool on top of it. The paper turns the same strokes into a control interface.
Keyboards and joysticks assume dry hands and a stable posture. This project makes swimming strokes the control input instead: a head-mounted camera watches the user’s arms, a computer-vision pipeline segments them and extracts contours, and the motion becomes steering and propulsion commands.
The same pipeline drives two targets: a Unity simulator built on a 3-D reconstruction of Peter Street Basin, and a remote-controlled racing boat over a low-latency wireless link. A natural user interface for water, in the lineage of the Ship of Kolympi.
The virtual domain: a Unity simulator built on a 3-D reconstruction of Peter Street Basin.
Abstract
This paper presents a Telekolympographic User Interface framework for remote or virtual vessel control through real-time swim gesture recognition on dry land. A head-mounted camera captures upper-body arm motion, which is processed by a computer vision pipeline that segments the user’s arms, extracts geometric contours, and converts motion into steering and propulsion commands. The resulting interaction paradigm treats swimming gestures not merely as body movements but as the primary control input of the system. To evaluate the approach, we applied the same gesture-recognition pipeline to both virtual and remote-physical domains. In the virtual domain, a Unity-based simulator incorporates a 3D reconstruction of Peter Street Basin, a safety pool that is part of Lake Ontario in the city of Toronto, allowing users to navigate a virtual vessel using upper-body swim gestures. In the physical domain, the same control logic is transmitted to a remote-controlled racing boat over a low-latency wireless link. Together, these implementations demonstrate that vision-based aquatic interaction can provide an intuitive Natural User Interface (NUI) alternative to conventional handheld controllers in environments where traditional interfaces are awkward or impractical.
Authors
Ibrahim Hassan, Varun Seshadri, Li-An Yang, Yuxuan Zhang, Alexander Vicol, Steve Mann
Links
- Paper on IEEE Xplore (DOI 10.1109/ICMA69663.2026.11647508)
- Also in the Mersivity / WaterHCI 2026 Symposium Proceedings (Zenodo)
- Related project: Swiminalization and (K)No(t)-Swimming
- Related project: Submersivity