This paper presents MiroSurge, a telepresence system for minimally invasive surgery developed at the German Aerospace Center (DLR), and introduces MiroSurge's new user interaction modalities: (1) haptic feedback with software-based preservation of the fulcrum point, (2) an ultrasound-based approach to the quasi-tactile detection of pulsating vessels, and (3) a contact-free interface between surgeon and telesurgery system, where stereo vision is augmented with force vectors at the tool tip. All interaction modalities aim to increase the user's perception beyond stereo imaging by either augmenting the images or by using haptic interfaces. MiroSurge currently provides surgeons with two different interfaces. The first option, bimanual haptic interaction with force and partial tactile feedback, allows for direct perception of the remote environment. Alternatively, users can choose to control the surgical instruments by optically tracked forceps held in their hands. Force feedback is then provided in augmented stereo images by constantly updated force vectors displayed at the centers of the teleoperated instruments, regardless of the instruments' position within the video image. To determine the centerpoints of the instruments, artificial markers are attached and optically tracked. A new approach to detecting pulsating vessels beneath covering tissue with an omnidirectional ultrasound Doppler sensor is presented. The measurement results are computed and can be provided acoustically (by displaying the typical Doppler sound), optically (by augmenting the endoscopic video stream), or kinesthetically (by a gentle twitching of the haptic input devices). The control structure preserves the fulcrum point in minimally invasive surgery and user commands are followed by the surgical instrument. Haptic feedback allows the user to distinguish between interaction with soft and hard environments. The paper includes technical evaluations of the features presented, as well as an overview of the system integration of MiroSurge.

The quality of a teleoperation system is decreased by time delays in the communication channel. Delays as low as a few hundred milliseconds between commanding an action and getting the visual feedback reduce the operator's performance. Predictive displays have proven their suitability to compensate for these delays, but at the expense of image quality when using computer-generated images. A photorealistic predictive display is presented that closes the feedback loop locally at the operator's side of a telepresence system. Photorealism is achieved using delayed camera images for texturing the predicted scene. Consumer graphics hardware is not only used for rendering but also for hardware-accelerated texture extraction. To allow concurrent access to model data, a multibuffer model structure is presented. A model of the teleoperator's environment is automatically acquired and updated by image processing techniques using a stereo camera as the only sensor.