As a result of the increase in online course offerings at colleges and universities, the prevalence of online and computerized examinations has increased over the last two decades. Beyond academia, online examinations have been used to assess competence for certifications by both governments and private firms. Despite this importance, a randomized controlled trial that randomizes individual test takers into online and in-person examinations has not previously been conducted. In this paper, we rectify this by conducting this experiment in a large economics class at a large research university in the American southeast. We supplement our experiment results with a novel methodology of using screening surveys administered to potential volunteers which solicit preference information. We find that online testing reduces examination grades by over a tenth of a standard deviation, relative to in-person assessment. Additional exploratory exercises provide evidence that this effect is driven by students who prefer in-person examinations.