Haptic augmented reality (AR) mixes a real environment with computer-generated virtual haptic stimuli, enabling the system to modulate the haptic attributes of a real object to desired values. This paper reports our second study on this functionality, with stiffness as a goal modulation property. Our first study explored the potential of haptic AR by presenting an effective stiffness modulation system for simple 1D interaction. This paper extends the system so that a user can interact with a real object in any 3D exploratory pattern while perceiving its augmented stiffness. We develop a complete set of algorithms for contact detection, deformation estimation, force rendering, and force control. The core part is the deformation estimation where the magnitude and direction of real object deformation are estimated using a contact dynamics model identified in a preprocessing step. All algorithms are designed in a way that maximizes the efficiency and usability of the system while maintaining convincing perceptual quality. In particular, the need for a large amount of preprocessing such as geometry modeling is avoided to improve the usability. The physical performance of each algorithm is thoroughly evaluated with real samples. Each algorithm is experimentally verified to satisfy the physical performance requirements that need to be satisfied to achieve convincing rendering quality. The final perceptual quality of stiffness rendering is assessed in a psychophysical experiment where the difference in the perceived stiffness between augmented and virtual objects is measured. The error is less than the human discriminability of stiffness, demonstrating that our system can provide accurate stiffness modulation with perceptually insignificant errors. The limitations of our AR system are also discussed along with a plan for future work.

Haptic augmented reality (AR) enables the user to feel a real environment augmented with synthetic haptic stimuli. This article addresses two important topics in haptic AR. First, a new taxonomy for haptic AR is established based on a composite visuo-haptic reality-virtuality continuum extended from the conventional continuum for visual AR. Previous studies related to haptic AR are reviewed and classified using the composite continuum, and associated research issues are discussed. Second, the feasibility of haptically modulating the feel of a real object with the aid of virtual force feedback is investigated, with the stiffness as a goal haptic property. All required algorithms for contact detection, stiffness modulation, and force control are developed, and their individual performances are thoroughly evaluated. The resulting haptic AR system is also assessed in a psychophysical experiment, demonstrating its competent perceptual performance for stiffness modulation. To our knowledge, this work is among the first efforts in haptic AR for systematic augmentation of real object attributes with virtual forces, and it serves as an initial building block toward a general haptic AR system. Finally, several research issues identified during the feasibility study are introduced, with the aim of eliciting more research interest in this exciting yet unexplored area.