In this letter, a novel adaptive filter, the adaptive two-pass median (ATM) filter based on support vector machines (SVMs), is proposed to preserve more image details while effectively suppressing impulse noise for image restoration. The proposed filter is composed of a noise decision maker and two-pass median filters. Our new approach basically uses an SVM impulse detector to judge whether the input pixel is noise. If a pixel is detected as a corrupted pixel, the noise-free reduction median filter will be triggered to replace it. Otherwise, it remains unchanged. Then, to improve the quality of the restored image, a decision impulse filter is put to work in the second-pass filtering procedure. As for the noise suppressing both fixed-valued and random-valued impulses without degrading the quality of the fine details, the results of our extensive experiments demonstrate that the proposed filter outperforms earlier median-based filters in the literature. Our new filter also provides excellent robustness at various percentages of impulse noise.