Project Overview
Taste as Form is a mixed reality system that turns vision-inferred food attributes and real-time chewing dynamics into generative visual structures. The system uses passthrough image analysis to infer a sensory profile of the food, including likely taste category, texture (mouthfeel), and color, synchronized with sEMG sensing of chewing dynamics.
Inferred taste cues inform geometric form and color palettes, while texture cues shape material physics and motion behavior. The user's chewing rhythm modulates the onset, intensity, and decay of the visual effects. This work proposes a framework for structuring inferred sensory qualities and mastication as coupled interaction signals for immersive dining and sensory-awareness experiences.
Background
Eating sensations are often understood through cross-sensory metaphors. We often perceive sweetness as rounded and bright, sourness as sharp and contrasting, while mouthfeel qualities like crunchiness suggest distinct material impressions. These crossmodal correspondences imply that food taste and texture (mouthfeel) can be represented through visible form, color, and dynamics.
While multisensory interaction has been widely explored in immersive media, eating sensations remain underrepresented as systematic input channels for real-time visual feedback. Existing immersive food experiences often augment dining contexts, but rarely integrate eating rhythms with food attributes to drive interpretable visualization. This limits their ability to support attentive eating, tasting education, and experiential food design, where both what a person eats and how they eat matter.
Research in HCI has extensively explored multisensory dining experiences through digital augmentation. Prior work has shown that immersive XR systems can manipulate visual food attributes, such as size, color, and appearance, to influence satiety and flavor perception. Complementary hardware-based interfaces, including olfactory displays and electrical stimulation, further extend gustatory experience by introducing artificial sensory cues. Together, these works highlight the potential of immersive and multisensory techniques in food-related interaction design.
Psychophysical and cognitive science research on crossmodal correspondences provides grounding for mapping eating sensations to vision \cite. Classic findings such as the Bouba/Kiki effect illustrate systematic associations between sensory impressions and geometric form, motivating shape-based metaphors for taste qualities (e.g., sweetness with roundness, sourness with angularity) . Complementary studies establish robust chromatic associations, linking specific tastes to distinct hues. Beyond taste, prior literature suggests that mouthfeel and texture can be expressed through dynamic visual parameters, including motion fluidity and material-like behavior, supporting the translation of embodied eating cues into visual dimensions
Approach
The system combines chewing dynamics with inferred food attributes to generate real-time visual feedback in mixed reality. Implemented in Unity 6 on Meta Quest 3, the prototype integrates two synchronized data streams. A Large Multimodal Model (GPT-4o) analyzes passthrough imagery to construct a design-oriented sensory profile of the food, extracting its likely dominant taste (sweet, sour, bitter, salty), top-3 texture features, and base color. The system does not directly sense gustation; rather, it uses visual food analysis as a proxy for selecting crossmodal mappings. In parallel, mastication signals are captured via surface electrodes using a MyoWare 2.0 EMG sensor placed on the masseter muscle. The EMG stream is smoothed and normalized to estimate chewing rate, intensity, and stability, which modulate the rhythm and continuity of the visual feedback in real time. A brief calibration at session start accounts for individual baseline variation.
The system translates inferred food taste and texture attributes into corresponding generative visual forms, guided by prior research on crossmodal perception.
Taste Visualization:
We translate inferred taste cues into visual form through geometric mapping and a dual-layer color strategy.
Geometric Structure:
Cognitive research informs the morphological mapping of taste-related cues. Sweetness maps to soft, circular forms; sourness to sharper, angular geometry; bitterness to more rigid, angular structures; and saltiness to compact, rectilinear forms with softened edges.
Dual-Layer Color Strategy:
The system employs two complementary palettes to balance sensation and identity. The \textit{Taste Palette} maps specific tastes to associated hues (e.g., vibrant pinks for sweetness) on smaller particle effects. The \textit{Identity Palette} samples the food’s natural base color, applying it to larger ambient forms to ground the visualization.
Texture Visualization:
Texture cues shape the material behavior and atmospheric qualities of the generated forms. Harder textures are rendered through rigid or fractured geometry, while softer and creamier textures appear as smooth, volumetric structures. Viscous textures are expressed through dense, cohesive motion with increased drag and delayed response. Temperature is integrated as part of texture expression, with cold sensations introducing frost-like overlays and heat producing subtle thermal shimmer. All texture effects are rendered using the food’s natural color palette.