Contextual modulation reported in early- to intermediate-level visual areas could be an essential component to signal border ownership (BO) that specifies the direction of figure along a contour. The surrounding regions that evoke significant suppression or facilitation are highly localized and asymmetric with respect to the center of the classical receptive field (CRF). We propose a hypothesis that such surrounding modulation is a basis for BO-selectivity. Although this idea has been discussed for several years, it is uncertain how many of a vast variety of surrounding organizations could signal correctly the direction of ownership, and how many could signal consistently for various stimuli. We carried out computationally a population study of the surrounding effects to investigate how many cells exhibit effective and consistent BO signals. We tested hundreds of various organizations, and found that most of the asymmetric, iso-orientation suppressive regions, regardless of position or size, lead to surprisingly high consistency in the direction of ownership for various stimuli. The combinations of iso-orientation suppression and cross-orientation facilitation indicate both high robustness and consistency in the ownership determination. We constructed a model for BO-selective neurons based on the surrounding effects, and investigated whether the model reproduces major characteristics of the neuronal responses, including a variety in the BO selectivity among neurons, consistency with respect to various stimuli, invariance to stimulus size, and co-selectivity to BO and contrast. The model reproduced successfully the major characteristics of BO-selective neurons.