This paper presents a quantitative characterization of the instability that a human user often experiences while interacting with a virtual textured surface rendered with a force-reflecting haptic interface. First, we quantified the degree of stability/ instability during haptic texture rendering through psychophysical experiments. The stiffness of the virtual textured surface upon detection of instability was measured under a variety of experimental conditions using two texture rendering methods, two exploration modes, and various texture model parameters. We found that the range of stiffness values for stable texture rendering was quite limited. Second, we investigated the attributes of the proximal stimuli experienced by a human hand while exploring the virtual textured surface in an attempt to identify the sources of perceived instability. Position, force, and acceleration were measured and then analyzed in the frequency domain. The results were characterized by sensation levels in terms of spectral intensity in dB relative to the human detection threshold at the same frequency. We found that the spectral bands responsible for texture and instability perception were well separated in frequency such that they excited different mechanoreceptors and were, therefore, perceptually distinctive. Furthermore, we identified the high-frequency dynamics of the device to be a likely source of perceived instability. Our work has implications for displaying textured surfaces through a force feedback device in a virtual environment.