The term “size contrast and assimilation” refers to a large class of geometrical illusions in which the apparent sizes of identical visual targets in various contexts are different. Here we have examined whether these intriguing discrepancies between physical and perceived size can be explained by a visual process in which percepts are determined by the probability distribution of the possible real-world sources of retinal stimuli. To test this idea, we acquired a range image database of natural scenes that specified the location of every image point in 3-D space. By sampling the possible physical sources of various size contrast or assimilation stimuli in the database, we determined the probability distributions of the size of the target in the images generated by these sources. For each of the various stimuli tested, these probability distributions of target size in different contexts accurately predicted the perceptual effects reported in psychophysical studies. We conclude that size contrast and assimilation effects are a further manifestation of a fundamentally probabilistic process of visual perception.