← All research projects

XR Cooking: An Exploration of Physical AI and eXtended Reality in the Kitchen

28th Annual Mersivity / WaterHCI Symposium 2026
Toronto and online, March 30 to May 10, 2026 · Published in the symposium proceedings (open access on Zenodo)

The CookingXR interface: a camera view with a pan outlined and labelled, a pan temperature of 184.8 and a prompt to add batter

CookingXR sees the pan, reads its true surface temperature and tells you when to pour the batter.

Temperature is the hardest thing to judge in a kitchen and the one you cannot safely touch. This system pairs a wearable thermal camera with a colour camera, corrects the thermal reading for the emissivity of what it is looking at, uses a fine-tuned YOLO model to find the pan and the food, and runs a cook state machine that turns the numbers into instructions. On pancake trials it guided every cook end to end without a single manual temperature check.

Abstract

Temperature control is one of the most critical and difficult skills in cooking, yet existing solutions either require direct contact or manual operation. In this paper, we present an XR cooking assistance system that integrates wearable thermal imaging, computer vision, and a cook state machine to provide real-time, hands-free temperature guidance in the kitchen. The system combines a thermal camera with an RGB camera, applies emissivity correction using the Stefan-Boltzmann equation, and uses a fine-tuned YOLO11m model to map detected food items to their true surface temperatures. The state machine then translates this information into actionable cooking instructions for the user. We evaluate the system on pancake cooking trials, demonstrating successful end-to-end guidance across all tests without any manual temperature checks.

Authors

Mario Figallo, Jonas Martins, Alexander Vicol, Nishant Kumar, Steve Mann

Links

← All research projects