The projective transformation onto the retina loses the explicit 3D shape description of a moving object. Theoretical studies show that the reconstruction of 3D shape from 2D motion information (shape from motion, SFM) is feasible provided that the first- and second-order directional derivatives of the 2D velocity field are available. Experimental recordings have revealed that the receptive fields of the majority of the cells in macaque area middle temporal (MT) display an antagonistic (suppressive) surround and that a sizable portion of these surrounds are asymmetrical. This has led to the conjecture that these cells provide a local measure for the directional derivatives of the 2D velocity field. In this article, we adopt a nonparametric and biologically plausible approach to modeling the role played by the MT surrounds in the recovery of the orientation in depth (the slant and tilt) of a moving (translating) plane. A three-layered neural network is trained to represent the slant and tilt from the projected motion vectors. The hidden units of the network have speed-tuning characteristics and represent the MT model neurons with their surrounds. We conjecture that the MT surround results from lateral inhibitory connections with other MT cells and that populations of these cells, with different surround types, code linearly for slant and tilt of translating planes.