Empirical studies of peer effects rely on the assumption that peer spillovers can be measured through observables. However, in the education context, many theories of peer spillovers center around unobservables, such as ability, effort, or motivation. I show that when peer effects arise from unobservables, the typical empirical specifications will not measure these effects accurately, which may help explain differences in the magnitude and even sign of peer effect estimates across studies. I also show that under reasonable assumptions, these estimates cannot be applied to determine the effects of regrouping students, a central motivation of the literature.