While an update rate of 30 Hz is considered adequate for real-time graphics, a much higher update rate of about 1 kHz is necessary for haptics. Physics-based modeling of deformable objects, especially when large nonlinear deformations and complex nonlinear material properties are involved, at these very high rates is one of the most challenging tasks in the development of real-time simulation systems. While some specialized solutions exist, there is no general solution for arbitrary nonlinearities. In this work we present PhyNNeSS—a Physics-driven Neural Networks-based Simulation System—to address this long-standing technical challenge. The first step is an offline precomputation step in which a database is generated by applying carefully prescribed displacements to each node of the finite element models of the deformable objects. In the next step, the data is condensed into a set of coefficients describing neurons of a Radial Basis Function Network (RBFN). During real-time computation, these neural networks are used to reconstruct the deformation fields as well as the interaction forces. We present realistic simulation examples from interactive surgical simulation with real-time force feedback. As an example, we have developed a deformable human stomach model and a Penrose drain model used in the Fundamentals of Laparoscopic Surgery (FLS) training tool box. A unique computational modeling system has been developed that is capable of simulating the response of nonlinear deformable objects in real time. The method distinguishes itself from previous efforts in that a systematic physics-based precomputational step allows training of neural networks which may be used in real-time simulations. We show, through careful error analysis, that the scheme is scalable, with the accuracy being controlled by the number of neurons used in the simulation. PhyNNeSS has been integrated into SoFMIS (Software Framework for Multimodal Interactive Simulation) for general use.

In this paper, we present some recent advances in realistic surgery simulation including novel algorithms for simulating surgical cutting and techniques of improving visual realism of the simulated scenarios using images. Simulation of surgical cutting is one of the most challenging tasks in the development of a surgery simulator. Changes in topology during simulation render precomputed data unusable. Moreover, the process is nonlinear and the underlying physics is complex. Therefore, fully realistic simulation of surgical cutting at real-time rates on single processor machines is possibly out of reach at the present time. In this paper, we present a hybrid approach to the simulation of surgical cutting procedures by combining a node-snapping technique with a physically based meshfree computational scheme, the point-associated finite field (PAFF) approach, and empirical data obtained from controlled cutting experiments. To enhance the realism of the rendered scenarios, we propose an innovative way of using images obtained from videos acquired during actual surgical processes. Using a combination of techniques such as image mosaicing and view-dependent texture-mapping, we have been able to achieve excellent realistic effects with desired tissue glistening as the camera position is changed. Realistic examples are presented to showcase the results.

The generation of multimodal virtual environments for surgical training is complicated by the necessity to develop heterogeneous simulation scenarios such as surgical incision, cauterization, bleeding, and smoke generation involving the interaction of surgical tools with soft biological tissues in real time. While several techniques ranging from rapid but nonphysical geometry-based procedures to complex but computationally inefficient finite element analysis schemes have been proposed, none is uniquely suited to solve the digital surgery problem. In this paper we discuss the challenges facing the field of realistic surgery simulation and present a novel point-associated finite field (PAFF) approach, developed specifically to cope with these challenges. Based upon the equations of motion dictated by physics, this technique is independent of the state of matter, geometry and material properties and permits different levels of detail. We propose several specializations of this scheme for various operational complexities. The accuracy and efficiency of this technique is compared with solutions using traditional finite element methods and simulation results are reported on segmented models obtained from the Visible Human Project.