Despite the prevalence and potentially harmful consequences of first impression bias during social decision-making, its precise neural underpinnings remain unclear. Here, on the basis of the fMRI study using ultimatum games, the authors show that the responders' decisions to accept or reject offers were significantly affected by facial trustworthiness of proposers. Analysis using a model-based fMRI method revealed that activity in the right lateral OFC (lOFC) of responders increased as a function of negative decision bias, indicating a greater likelihood of rejecting otherwise fair offers, possibly because of the facial trustworthiness of proposers. In addition, lOFC showed changes in functional connectivity strength with amygdala and insula as a function of decision bias, and individual differences in the strengths of connectivities between lOFC and bilateral insula were also found to predict the likelihood of responders to reject offers from untrustworthy-looking proposers. The present findings emphasize that the lOFC plays a pivotal role in integrating signals related to facial impression and creating signal biasing decisions during social interactions.