The interactive requirements of 3D games and physically driven virtual environments add strong constraints to the simulation of natural cloth collisions and self-collisions. In order to achieve interactive rates, we first define smoothness conditions over small patches of deformable surfaces and then resort to image-based collision-detection algorithms that we have developed. Our collision-detection system achieves interactive rates as it accurately tracks collisions and self-interactions of objects consisting of highly deformable surfaces. This method makes use of a novel technique for dynamically generating a hierarchy of cloth bounding boxes in order to perform object-level culling and image-based intersection tests using conventional graphics hardware support. Our results show that, for complex deformable surfaces with an excess of 50,000 triangular elements, we can track collisions at nearly interactive rates.