Financial Incentives and Educational Investment: The Impact of Performance-based Scholarships on Student Time Use Free

Educators have long worried about relatively low levels of educational performance among U.S. students.¹ Although on international tests of mathematics and science fourth- and eighth-grade elementary school students in the United States perform comparably to their peers in other developed countries (Provasnik et al. 2012), fifteen-year-old students in the United States perform poorly compared with their international peers in these same subjects (Kelly et al. 2013). Further, at the postsecondary level there is increasing concern that whereas the United States has one of the highest rates of college attendance in the world, rates of college completion lag those of other countries (OECD 2011). In response, educators have implemented many policies aimed at changing curricula, class size, teacher effectiveness, and other resources at all levels of schooling.

More recently, there has been interest in targeting another key component—the effort exerted by students themselves toward their studies. Fundamentally, if students are to succeed in college, they need to master the material taught in a number of courses in order to earn credits. This takes effort. Students put effort into their education in at least two dimensions—attending class and studying—and students can increase effort in these dimensions by attending class more frequently, increasing the time they spend studying, or improving the quality of their time spent studying or in class. Little is known about the causal impact of effort on educational outcomes, both because effort is not easily measured and because it is a choice variable. Observational studies suggest that class attendance has a positive impact on student grades (see, e.g., Romer 1993). However, the best evidence of a causal impact of effort on student grades comes from Stinebrickner and Stinebrickner (2008), who use an instrumental variables strategy and find that increased time spent studying has a positive and statistically significant impact on student grades. Specifically, they estimate that one additional hour spent studying per week increases first-semester grade point averages (GPAs) by 0.36 GPA points, the equivalent of a 1.1 standard deviation increase in ACT test scores. Therefore, increasing students' effort toward their studies leads to improved educational outcomes. Further, the large decline in time devoted to studies documented by Babcock and Marks (2011; from 40 hours per week for full-time students in 1961 to 27 hours per week for full-time students in 2004) suggests that there is scope for students to increase time spent investing in their education.

One approach to motivating students to work harder in school has been to offer them rewards for achieving prescribed benchmarks. Related to the conditional cash transfer strategies that have been growing in popularity in developing countries (see, e.g., Das, Do, and Ozler 2005 and Rawlings and Rubio 2005), U.S. educators have implemented programs in which students are paid for achieving benchmarks such as a minimum GPA or for reading a minimum number of books. The U.S. strategies are based on the belief that current payoffs to education are too far in the future (and potentially too “diffuse”) to motivate students to work hard in school. As such, by implementing more immediate payoffs, these incentive-based strategies are designed to provide students with a bigger incentive to work hard.

Unfortunately, the evidence to-date of such efforts has yielded somewhat mixed impacts on student achievement. For example, Jackson (2010a, b) finds some evidence that the Advanced Placement Incentive Program in Texas—which rewards high school students (and teachers) for AP courses and exam scores—increased scores on the SAT and ACT tests, increased rates of college matriculation and persistence (students were more likely to remain in school beyond their first year of college), and improved postsecondary school grades. These results are similar to those reported by Angrist and Lavy (2009) and Kremer, Miguel, and Thornton (2009) from high school incentive programs in Israel and Kenya, respectively, which reward students for exam scores. Somewhat in contrast, Fryer (2011) reports suggestive evidence that rewarding elementary and secondary school students for effort focused on education inputs (such as reading books) may increase test score achievement, but rewarding them for education outcomes (such as grades and test scores) does not. One might think, however, that the impacts of incentives for younger students may be different from the impacts of incentives for postsecondary students.

At the postsecondary level, estimated impacts of financial incentives have been modest. For example, Angrist, Lang, and Oreopoulos (2009) and Angrist, Oreopoulos, and Williams (2014) report small impacts on grades at a four-year college in Canada, although impacts may be larger for some subgroups.² Barrow et al. (2014) report positive impacts on enrollment and credits earned for a program aimed at low-income adults attending community college in the United States, and early results from MDRC's Performance-based Scholarship Demonstration (described below) suggest modest impacts on some academic outcomes (such as enrollment and credits earned) but little impact on the total number of semesters enrolled (see Patel et al. 2013). Results are somewhat mixed from studies of the larger state-run merit aid programs that target high school students meeting minimum qualification requirements and then require students to meet benchmarks for renewal. Dynarski (2008) finds a 3 percentage point increase in college completion rates from the Arkansas and Georgia merit aid programs; however, Sjoquist and Winters (2012) find no impact on the college completion rates for these same programs when using a larger sample.³ Furthermore, Cornwell, Lee, and Mustard (2005) find that the Georgia program increased time to degree. Scott-Clayton (2011) finds relatively sizable impacts of the West Virginia PROMISE program on the probability of earning at least thirty credits and maintaining the benchmark GPA in each of the first three years of college, as well as a 2 percentage point increase in the Bachelor of Arts (BA) completion rate relative to a base BA completion rate of 21.5 percent. Importantly, the PROMISE program had minimum credit requirements in addition to GPA requirements, and Scott-Clayton (2011) finds that students respond to the incentive more strongly during the first three years of PROMISE receipt, when meeting the benchmark is required for renewal.⁴

The fact that impacts of financial incentives on academic outcomes at the postsecondary level have been mostly modest raises the question of why. For example, can educational investments be effectively influenced through the use of incentives in the sense that they actually change student behavior? Alternatively, are these small, positive results a statistical anomaly, do they reflect the provision of additional income rather than the incentive structure, or do they represent changes along other dimensions, such as taking easier classes? And finally, can the variation in estimated impacts be explained by the size of the scholarships?

In this paper, we evaluate the effect of a performance-based scholarship (PBS) program for postsecondary students on a variety of outcomes, but especially on student effort as reflected in time use. High school seniors in California were randomly assigned to treatment and control groups where the treatments (the incentive payments) varied in length and magnitude and were tied to meeting performance, enrollment, and/or attendance benchmarks. To measure the impact of PBSs on student educational effort, we surveyed participants about time use over the prior week and implemented a time diary survey.

Our paper contributes to the literature on incentives in education along a number of dimensions. First, it provides the initial estimates of the impact of these types of scholarships on postsecondary school enrollment for high school seniors who have yet to enroll in college.⁵ Second, data on time use allow us to get inside the “black box” to understand how postsecondary students may have changed their time allocation in response to the incentives. Further, variation in the incentive structure of the scholarships allows us to test a variety of hypotheses about the impacts of incentive payments among college-aged students in the United States. Specifically, we test whether larger payments or longer payments differentially affect outcomes. Finally, we provide some evidence on whether students respond to the incentives or the increased income.

We find that high school students eligible for a PBS were more likely to enroll in college the following semester. Further, those eligible for a scholarship devoted more time to educational activities; increased the quality of effort toward, and engagement with, their studies; and allocated less time to other activities, such as work and leisure. Additional evidence indicates that the incentives did not affect student behavior—either positively or negatively—after incentive payments were removed, suggesting that students were motivated by the incentives provided by the scholarships rather than simply the additional money. We note that the surveys upon which these results are based only yielded response rates between 43 percent (second semester) and 45 percent (first semester), raising the concern that the results are driven by selection bias. We have implemented a variety of tests to assess this possibility and ultimately conclude the estimates are internally valid. We base this assessment on the fact that omnibus F-tests of the difference in means of background characteristics at baseline by treatment/control status for our analysis sample are not statistically significant at conventional levels. Further, our estimates are largely unchanged if we include baseline characteristics as controls or use an inverse-probability weighting estimator to correct for selection. Unfortunately, Lee bounds estimates (Lee 2009) are quite wide and largely uninformative. However, when we assess selection on unobservables using the strategy suggested by Altonji, Elder, and Taber (2005), we conclude that this is unlikely to overturn our primary findings. Keeping this in mind, overall our findings indicate that well-designed incentives can induce postsecondary students to increase investments in educational attainment. One remaining puzzle, however, is that larger incentive payments did not seem to induce students to increase effort more than smaller incentive payments.

We next discuss a theoretical framework for thinking about effort devoted to schooling and the role of incentive scholarships (section 2). We describe the intervention studied, the data, and sample characteristics of program participants in section 3. The estimation strategy and results are presented in section 4, and section 5 concludes.

We adopt the framework introduced by Becker (1967), in which students invest in their education until the marginal cost of doing so equals the marginal benefit. Investing in education is costly in terms of effort. Benefits in our setting include the present discounted value of the earnings increase associated with additional college credits (net of tuition and other costs) as well as incentive payments for eligible students meeting the minimum GPA benchmark. Students maximize utility by maximizing the net expected benefit of effort. If the marginal benefit is relatively low or the marginal costs are relatively high a student may not enroll or continue in college. For a student who enrolls, an increase in the payoff to meeting the benchmark will lead to an increase in effort toward her studies. Likewise, a fall in the payoff will lead to a decrease in effort.⁶

Although this is a simple static model of optimal choice of effort, there are potential dynamic effects that we will consider and test in addition to possible unintended consequences. For example, traditional need-based and merit-based scholarships provide an incentive to enroll in college by effectively lowering the costs of enrolling regardless of whether the student passes her classes once she gets there. In the theoretical framework outlined above, these types of scholarships have no impact on the marginal value or cost of effort conditional on enrolling in school. However, in a dynamic version of the model, future scholarship receipt may depend on meeting the benchmark in the current semester. Pell Grants, for example, stipulate that future grant receipt depends on meeting satisfactory academic progress, but within-semester grant receipt is unaffected by performance. Performance-based scholarships, on the other hand, increase the marginal benefit of effort toward schoolwork in the current semester by increasing the current payoff. For example, payments may be contingent on meeting benchmark performance goals such as a minimum GPA. Because a PBS increases the short-run financial rewards to effort, we would expect PBS-eligible students to allocate more time to educationally productive activities, such as studying, which should in turn translate into greater educational attainment, on average.

We may expect the effectiveness of PBS programs to depend on both the size of the scholarship as well as the impact of the program on a student's GPA production function. In the basic theoretical model, the direct effect of increasing the size of the incentive payment in the current semester is a contemporaneous increase in effort but no impact on effort in future semesters.⁷ However, there may also be indirect effects leading to impacts in future semesters. For example, an increase in effort in the current semester may reduce the marginal cost of effort in the future (suppose increased studying today is “habit-forming”), leading PBS eligibility to have a positive impact on student effort after eligibility has expired.⁸ Similarly, if increased effort today teaches students how to study more effectively, then PBS eligibility could also have a lasting positive impact on student outcomes.

At the same time, cognitive psychologists worry that although incentive payments may motivate students to do better in the short term, students may be motivated for the “wrong” reasons. They distinguish between internal (or intrinsic) motivation, in which a student is motivated to work hard because he or she finds hard work inherently enjoyable or interesting, and external (or extrinsic) motivation, in which a student is motivated to work because it leads to a separable outcome (such as a PB-incentive payment; see, e.g., Deci 1975 and Deci and Ryan 1985). A literature in psychology documents more positive educational outcomes the greater the level of “internalization” of the motivation (e.g., Pintrich and De Groot 1990). As such, one potential concern regarding performance-based rewards in education is that although such scholarships may increase external motivation, they may decrease internal motivation (e.g., Deci, Koestner, and Ryan 1999). A reduction in intrinsic motivation could be viewed as raising students’ cost of effort. Assuming this effect is permanent, we would expect to see a negative impact of PBS eligibility on effort in future semesters after PBS eligibility has expired (Huffman and Bognanno, forthcoming; Benabou and Tirole 2003). Some even hypothesize that the reduction in intrinsic motivation may more than offset the external motivation provided by the incentive. This would lead to a negative impact of PBS eligibility even during semesters in which incentive payments were provided (Gneezy and Rustichini 2000).

Finally, although the intention of a PBS is to increase student effort in educationally productive ways, it may unintentionally increase student attempts to raise their performance in ways that are not educationally productive. For example, Cornwell, Lee, and Mustard (2005) find that the Georgia HOPE scholarship—which had grade incentives but not credit incentives—reduced the likelihood that students registered for a full credit load, and increased the likelihood that students withdrew from courses presumably to increase the probability they would meet the minimum GPA benchmark. Other unintended consequences could include cheating on exams, asking professors to regrade tests and/or papers, or taking easier classes.

The data we analyze were collected as part of the Performance-Based Scholarship Demonstration conducted by MDRC at eight institutions and at a state-wide organization in California. The structure of the scholarship programs as well as the populations being studied vary across the sites; this study presents results from a supplementary “Time Use” survey module we implemented in California (CA).⁹ Our Time Use survey was coordinated with MDRC but was outside the scope of their project. In this demonstration, the scholarships supplemented any other financial aid for which the students qualified (such as federal Pell Grants and state aid).¹⁰

The California program is unique in the PBS demonstration in that random assignment took place in the spring of the participants’ senior year of high school, and students could use the scholarship at any accredited institution.¹¹ Individuals in the study were selected from participants in “Cash for College” workshops at which attendees were given assistance in completing the Free Application for Federal Student Aid (FAFSA). In order to be eligible for the study, they also had to complete the FAFSA by 2 March of the year in question. Study participants were selected from sites in the Los Angeles and Far North regions in 2009 and 2010, and from the Kern County and Capital regions in 2010. Randomization occurred within each workshop in each year. Because the students were high school seniors at the time of random assignment, this demonstration allows us to determine the impact of these scholarships not only on persistence among college students but also on initial college enrollment.

To be eligible for this study, participants had to have attended a Cash for College workshop in one of the participating regions; been a high school senior at the time of the workshop; submitted a FAFSA and Cal Grant GPA Verification Form by the Cal Grant deadline (early March); met the low-income eligibility standards based on the Cal Grant income thresholds; and signed an informed consent form or had a parent provide consent for participation.

The incentive varied in length (as short as one semester and as long as four semesters), size of scholarship (as little as $1,000 and as much as $4,000), and whether there was a performance requirement attached to it. For comparison, in 2009–10 the maximum Pell Grant award was $5,350 per year (Office of Postsecondary Education 2012). Also, the PBSs were paid directly to the students, whereas the non-PBS was paid to the institution. Table 1 shows the structure of the demonstration for the Fall 2009 cohort more specifically—the structure was similar for the Fall 2010 cohort. There are six treatment groups labeled 1 to 6 in table 1. Group 1 was randomly selected to receive a California Cash for College scholarship worth $1,000, which is a typical grant that has no performance component and is paid directly to the student's institution. Groups 2 through 6 had a performance-based component with payments made directly to the student. Group 2 was randomly assigned to receive $1,000 over one academic term (a semester or quarter); group 3 was randomly selected to receive $500 per semester for two semesters (or $333 per quarter over three quarters); group 4 was selected to receive $1,000 per semester for two semesters (or $667 per quarter over three quarters); group 5 was to receive $500 per semester for four semesters (or $333 per quarter over six quarters); and group 6 was to receive $1,000 per semester for four semesters (or $667 per quarter over six quarters). During fall semesters of eligibility, one-half of the PBS was paid conditional on enrolling for six or more credits at an accredited, degree-granting institution in the United States, and one-half was paid if the student met the end-of-semester benchmark (a final average grade of “C” or better in at least six credits). During spring semesters of eligibility, the entire scholarship payment was based on meeting the end-of-semester benchmark. Sixty percent of students in the program group received a performance payment for meeting the end-of-semester benchmark in their first semester after program enrollment, and 45.5 percent received an end-of-semester performance payment in their second semester (Richburg-Hayes et al. 2015).

In addition, aside from the institution being accredited, there were no restrictions on where the participants enrolled in college. That said, according to data from the Cash for College workshops, among the two-thirds of students who enroll in college the following fall, over 90 percent attend a public college or university within CA—about one-half in a two-year college and the other half in a four-year institution.

In table 2 we present information on the number of students in each cohort. In total, 5,160 individuals were recruited to be part of the PBS study; 1,720 were randomly assigned to the program-eligible group and 3,440 were assigned to the control group. We also surveyed an additional 1,500 individuals as part of our control group; they were randomly selected from the “excess” control group individuals who were not selected to be followed by MDRC for the MDRC study. Table A.1 shows means of background characteristics (at baseline) by treatment/control status. Although there are one or two characteristics that appear to differ between treatment and control groups, an omnibus F-test yielded a p-value of 0.50 suggesting that randomization successfully balanced the two groups, on average.

According to Ware and Patel (2012), in the first cohort 85 percent of the scholarship eligible participants received an enrollment payment and, of those, 60 percent earned the performance-based payment at the end of the fall 2009 semester.

To better understand the impact of PBSs on student educational effort, we implemented an independent (Web-based) survey of participants. We asked respondents general questions about educational attainment and work (roughly based on the Current Population Survey). The centerpiece included two types of questions designed to better understand how respondents allocated their time to different activities. To understand more “granular” time allocation we implemented a time diary for which we used the American Time Use Survey (ATUS) as a template. Accounting for an entire 24-hour time period, the ATUS asks the respondent to list his or her activities, describe where the activities took place, and with whom. In addition, we included questions about time use over the last 7 days to accommodate those activities that are particularly relevant to students and for which it would be valuable to measure over longer periods (such as time spent studying per day over the past week). Participants were offered an incentive to participate.

In addition to the questions regarding time use over the previous 24 hours or 7 days (that reflect the “quantity” of time allocated to activities), the survey also included questions to measure the quality of educational efforts. To capture these other dimensions of effort, we included questions on learning strategies, academic self-efficacy, and motivation. To measure learning strategies that should help students perform better in class, we included questions from the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al. 1991). The seven-point scale consists of five questions, such as: “When I become confused about something I'm reading, I go back and try to figure it out” (responses range from not at all true [1] to very true [7]). In addition, researchers have documented a link between perceived self-efficacy (e.g., an individual's expectations regarding success or assessment of his or her ability to master material) and academic performance (see, e.g., Pintrich and De Groot 1990). Therefore, we included five questions that form a scale to capture perceived academic efficacy (the Patterns of Adaptive Learning Scales [PALS] by Midgley et al. 2000). These questions are of the form, “I'm certain I can master the skills taught in this class this year” with responses on a similar seven-point scale.

Finally, we attempted to assess whether the incentives also induced unintended consequences, such as cheating, taking easier classes, or attending classes simply to receive the reward and not because of an inherent interest in the academics (which some psychologists argue can ultimately adversely affect academic achievement, as discussed earlier). As such, on the survey we asked participants about life satisfaction, whether they had taken “challenging classes,” if they had ever asked for a regrade, and if they had ever felt it necessary to cheat. To capture external and internal motivation, both current students and those not currently enrolled were asked questions along the lines of, “If I do my class assignments, it's because I would feel guilty if I did not” (also on a seven-point scale).¹³

We focus this analysis on time use and effort in the first semester after random assignment because the majority of both program and control students were enrolled in a postsecondary institution such that an analysis of time use is most compelling as one of the factors that determine educational success. That said, we also surveyed each cohort in the second semester after random assignment to gauge the extent to which time use changed and whether differences between the program and control group members persisted.

Table 3 presents selected mean baseline characteristics for study participants at the time of random assignment and compares them to nationally representative samples of students from the National Postsecondary Student Aid Study (NPSAS) of 2008 designed to be comparable to the participants in the study.¹⁴ There were slightly more women in the study sample than in the NPSAS sample. Further, the proportion of Hispanic and black participants was much higher in this study than nationally. For example, 63 percent of participants were Hispanic compared with 15 percent nationally, and only 4 percent were black. Similarly, a language other than English is more likely to be spoken in the study sample.

To facilitate interpretation and to improve statistical power, we group impacts on individual time use into two “domains” of most interest for this study: academic activities and nonacademic activities.¹⁵ Further, we also summarize impacts of measures that reflect the quality of educational effort and those that capture potential “unintended consequences.”

We can then summarize the individual estimates using a seemingly unrelated regression (SUR) approach (Kling and Liebman 2004). This approach is similar to simply averaging the estimated effect of being randomly assigned to be eligible for a PBS, if there are no missing values and no covariates.

From our data, we focus on respondents to the survey administered late in the first semester after random assignment, although we also report some results based on second semester surveys. In the first semester, the response rate for the treatment group was 58 percent and for the control group was 40 percent; in the second semester, the response rates were 53 percent and 39 percent, respectively—differences that are statistically significant at traditional levels.¹⁷ An analysis of who responded suggests that women were more likely to respond (see table A.2). Further, respondents were somewhat more likely to be Asian and less likely to be black or Native American. They also had parents with lower education levels (more high school drop-outs and fewer high school diploma/GED holders), were less likely to speak English at home, and had somewhat higher high school GPAs. Omnibus F-tests of whether baseline characteristics jointly predict survey response status yielded a p-value of 0.000.

To create the analysis sample, we drop individuals who did not complete the time diary (or who had more than four “noncategorized” hours in the 24-hour time period) and those for whom we did not have data in the first part of the survey (due to an error by the survey contractor). We have data from 2,874 complete surveys for the first semester and 2,743 complete surveys for the second semester. These complete surveys represent roughly 96 percent of the total number of survey respondents in each semester. The response rates after dropping these additional observations are 56 percent and 51 percent for the treatment group in the first and second semesters, respectively. For the control group, the response rates are 39 percent and 38 percent, respectively, for the first and second semesters. Based on observable characteristics the analysis sample does not appear representative of the full experimental sample. That said, we believe these estimates are internally valid. Table A.3 shows means of background characteristics (at baseline) by treatment/control status for our analysis sample. A few characteristics appear to differ between treatment and control groups; however, omnibus F-tests are not statistically significant at conventional levels, and our estimates are largely unchanged if we include baseline characteristics as controls (see tables 4 and 5 for estimates including baseline characteristics). That said, we assess the potential impact of survey selection bias on the estimates in a separate section (see the Assessing the Possible Impact of Survey Selection Participation Bias section).

In table 4 we present estimates of the effect of program eligibility on individual measures of time-use based on our survey—in this table we do not distinguish between the types of PBSs offered. In column 1 we provide outcome means for the control group participants. Program effect estimates with standard errors in parentheses are presented in columns 2 and 3. Note that the estimates in column 2 reflect the impact of being eligible for a PBS, and the estimates in column 3 reflect the impact of being eligible for a non-PBS. The p-value corresponding to the test that the PBS program impact equals the non-PBS program impact is presented in column 4. Program effects are estimated including controls for “randomization pool” fixed effects but no other baseline characteristics.¹⁸ Corresponding estimates that also control for baseline characteristics—age; an indicator for sex (female); race/ethnicity indicators (Hispanic/Latino, black/African American, white, American Indian or Alaska native, or other); indicators for parents' highest level of education (less than high school, high school diploma, associate's degree); an indicator for being the first in family to attend college; indicators for speaking Spanish or English at home; and standardized responses to exit questions based on motivation—are presented in columns 5 through 7. The number of observations in each estimation sample is listed in column 8.

Focusing on the coefficients reported in column 2 of table 4, we find that PBS-eligible students were 5.2 percentage points more likely than the control group to report ever enrolling at a postsecondary institution, a difference that is statistically significant at the 1 percent level. Further, the PBS-eligible students reported studying about eight minutes more per day than those in the control group, were 7.3 percentage points more likely to have been prepared for class in the last seven days, and were 6.7 percentage points more likely to report attending all or most of their classes in the last seven days. Estimates controlling for baseline characteristics (reported in column 5) are quite similar.

An important dimension to this demonstration was the inclusion of a treatment group that was eligible for a “regular” scholarship that did not require meeting performance benchmarks. In particular, as discussed earlier, this non-PBS does not affect the marginal value of effort because payment is not tied to meeting benchmarks and is only valid for one semester. We generally find the impacts are larger for those eligible for a PBS than for those offered a non-PBS, however, in most cases we are unable to detect a statistically significant difference. Tests of the difference in impact between the PBS and the non-PBS also potentially provide insight into whether students are responding to the incentives in the PBS or the additional income. We discuss this implication below.¹⁹

Before turning to how participants allocated their time to other activities, we consider two measures that may indicate ways of increasing academic effort without necessarily spending more time studying: (1) learning strategies and (2) academic self-efficacy. As discussed above, PBS eligibility may induce participants to concentrate more on their studies by encouraging them to employ more effective study strategies, making the time devoted to educational activities more productive. Similarly, by raising their academic self-efficacy the scholarships may also induce students to be more engaged with their studies. Results using scales based on the MSLQ Learning Strategies index and the PALS academic self-efficacy index are presented in the last two rows of table 4. We have standardized the variables using their respective control group means and standard deviations—the coefficients therefore reflect impacts in standard deviation units. We estimate that eligibility for a PBS had positive and statistically significant impacts on these dimensions that range from 12 to 22 percent of a standard deviation. Note as well that the impacts on learning strategies and academic self-efficacy for those selected for a non-PBS were substantially smaller than those selected for a PBS, consistent with increased academic effort on the part of PBS-eligible individuals.

Results presented thus far generally suggest that participants selected for a PBS devoted more time and effort to educational activities. Given that there are only 24 hours in the day, a key question is what did PBS-eligible participants spend less time doing? Table 5 presents results from three other broad time-categories based on the 24-hour time diary: (1) work, (2) household production, and (3) leisure and other activities.²⁰ For work, we find only suggestive evidence that PBS-eligible participants may have spent fewer hours working. We find no evidence that the typical PBS-eligible participant spent fewer hours working during the last 24 hours and a negative but statistically insignificant reduction in hours worked in the past week. We also find no evidence that PBS eligibility reduced time spent on household production activities. Instead, we find that participants accommodated increased time spent on educational activities by spending (statistically) significantly less time on leisure activities, including reducing the number of nights out for fun during the past week.

Finally, concerns about using incentives for academic achievement include the possibility of unintended consequences of the programs, such as cheating or taking easier classes to get good grades, or reducing students’ internal motivation to pursue more education. In the bottom rows of table 5 we present impacts on several potential unintended consequences. The results regarding unintended consequences are somewhat mixed. For example, on the one hand those who were eligible for a PBS were more satisfied with life and more likely to take challenging classes compared with the control group (a difference that is statistically significant at the 1 percent level). On the other hand, PBS-eligible participants reported an increase in behavior that is consistent with external motivation compared to both control group participants and those randomly selected for a non-PBS.

Overall, the results in tables 4 and 5 suggest that eligibility for a scholarship that requires achieving benchmarks results in an increase in time and effort devoted to educational activities, with a decrease in time devoted to leisure. Further, there is at best mixed evidence that the same incentives result in adverse outcomes, such as cheating, “grade grubbing,” or taking easier classes. However, for many of the outcomes the impacts are not statistically different from zero.

To improve precision, in table 6 we combine the individual outcomes into four “domains” using the SUR approach described above. Specifically, we focus on academic activities, quality of educational input, nonacademic activities, and unintended consequences.²¹ The impacts reported in table 6 and subsequent tables have been standardized such that they represent average impacts as a percentage of the control group standard deviation. Note that now we estimate a positive impact on academic activities of about 10 percent of a standard deviation and that the impact is statistically significant at the 5 percent level. We also continue to estimate a positive and statistically significant impact on the quality of educational effort. In addition, we estimate a reduction in nonacademic activities, although the coefficient estimate is only significant at the 10 percent level. Further, we estimate that, overall, there is not an increase in “unintended consequences” as a result of the academic financial incentive. In sum, these results suggest that scholarship incentives change time allocation in the sense that students spend more time and effort on academic activities and less time on other activities.

To explore the possibility that results are driven by an incentive effect only on the extensive margin, we have reestimated the impacts limiting the samples to those students who enrolled in school, as shown in columns 5 through 8 of table 6. We find that estimated impacts on academic activities (including quality of academic input) are somewhat smaller but still statistically different from zero, suggesting that PBS eligibility affects both the extensive and intensive margins. Of course, this relies on the assumption that those who are induced by the scholarship to enroll in school would not have otherwise put forth more effort toward their studies. Further, we note that the results are based on self-reported data and therefore the possibility that respondents’ self-reports are correlated with their treatment status may also bias the estimated impacts.

Three key questions are (1) whether the size of the potential scholarship affects the impact on student behavior, (2) whether scholarship eligibility impacts student behavior even after incentives are removed, and (3) whether it is the incentive structure or the additional income that generates changes in student behavior. Prior studies of PBSs have tended to focus on one type of scholarship of a particular duration with variation in other types of resources available to students (such as student support services or a counselor). In this study, students eligible for a PBS were also randomly assigned to scholarships of differing durations and/or sizes as well as a non-PBS. As noted in table 1, students selected for scholarship eligibility were assigned to one non-incentive scholarship worth $1,000 for one term or to one of five types of incentive scholarships that ranged from $1,000 for one term to $1,000 for each of four terms or $500 for each of two terms to $500 for each of four terms. We exploit this aspect of the design of the demonstration to study the impact of scholarship characteristics on student behavior.²² We present results using SUR in tables 7 and 8.²³

Theoretically one would expect that the larger-sized scholarships would induce increased effort compared with smaller-sized scholarships during the semesters for which the students were eligible. As such, in the first semester after random assignment, we might expect to see a difference between scholarships worth $1,000 and those worth $500 per term. In table 7, we begin by examining whether larger scholarships generated larger impacts than smaller scholarships. Using results from the first academic term after random assignment (fall), impact estimates of the $500/term scholarships are presented in column 1 and the $1,000/term scholarships in column 2. The p-values for the test of equality of the coefficient estimates in columns 1 and 2 are presented in column 4. Interestingly, we do not find large differences in the effect of PBS eligibility related to the size of the scholarship. Students who were eligible for a $500 per semester scholarship responded similarly on most outcomes to students who were eligible for a $1,000 per semester scholarship, suggesting that larger incentive payment amounts did not lead to larger impacts on student effort. Although this result is familiar in the context of survey implementation, where experimental evidence suggests that larger incentives do not increase response rates (see, e.g., James and Bolstein 1992), other laboratory and field experiments have found that paying a larger incentive improves performance relative to a smaller incentive (e.g., Gneezy and Rustichini 2000 and Lau 2017). This finding remains a puzzle, although we offer some potential explanations for further consideration in section 5.

If incentives have no effect on the GPA production function, then one would expect that only the longer-duration scholarships would affect effort during the additional semesters of eligibility. As we note above, however, some of the literature on incentives and motivation predicts that we might observe reductions in effort after incentives are removed if the PBS has a negative effect on intrinsic motivation (Benabou and Tirole 2003; Huffman and Bognanno, forthcoming). Alternatively, one might expect increased effort in the first semester to be habit-forming or make students more efficient at transforming study time into a higher GPA in the second semester. In the latter case, PBSs may continue to have positive impacts on student outcomes in semesters after eligibility has expired. In table 8 we look at results for outcomes measured in the second semester after random assignment and consider impacts for the one-term scholarships that have expired (PBS and non-PBS) and the four PBSs for which eligibility continued two or more terms.

To begin, the results reported in column 1 examine the impacts of PBS eligibility on second-semester student outcomes for participants who are no longer eligible for PBS payments. We find that the impacts of PBSs are largely contemporaneous. We find no difference in enrollment probabilities or the index of all academic activities between one-term PBS-eligible participants and the control group during the second program semester. However, there is suggestive evidence of a lasting positive impact on the quality of educational inputs. Namely, one-term PBS-eligible students have higher quality of effort than the control group in the semester after eligibility has expired, but the result is only statistically significant at the 10 percent level.

The results in column 2 represent the impacts of PBS eligibility in the second semester after random assignment for those who continue to be eligible for PBS payments. Here we continue to find positive impacts of PBS eligibility on academic effort and quality of effort relative to control group participants, and we find negative impacts of PBS eligibility on nonacademic activities and unintended consequences. As such, we find that a PBS primarily affects student behavior in the semester in which the student is eligible for the scholarship. In contrast to predictions from some dynamic models, we do not detect a negative impact of incentive eligibility on educationally productive behavior once the incentive is removed nor do we find strong evidence of a lasting change in student behavior as a result of prior eligibility for an incentive.

Finally, an important question regarding any results with PBSs is whether the impacts are driven by the additional income or by the incentive structure of the scholarship. In other words, does it matter that the PBS comes with an incentive structure or would a simple monetary award with no incentives generate the same impacts? In our study, a comparison of the (fall term) impacts of the non-PBS (worth $1,000 in one term) and the PBS of $1,000 per term, potentially provides a test of the impact of the incentive structure in the PBS compared with just awarding additional money. This test can be made in table 7 by comparing the impacts of the $1,000 PBSs (column 2) to those of the $1,000 per term non-PBS in the first term (column 3) (the p-values of the tests of equality are in column 5). With the exception of the coefficient on “unintended consequences,” we find that the magnitudes of the PBS coefficient estimates are larger in absolute value than those for the non-PBS, although the differences cannot be detected at conventional levels of statistical significance.²⁴ These results are consistent with the incentive structure in the scholarships, rather than primarily the additional income inducing the changes in behavior.

However, there was a critical difference in how the two types of scholarships were awarded that might also explain the larger impacts for the incentive scholarships: The incentive-based scholarships were paid directly to the students whereas the non-PBS was paid to the institutions. As such, the non-PBS may not have been as salient to the student. There is also the possibility that institutions at least partly offset the non-PBS with reductions in other forms of financial aid. One might interpret the fact that the PBS (column 2 of table 7) had a larger impact on enrollment than the non-PBS (column 3 of table 7) as providing support for these alternative interpretations. If the PBSs and non-PBSs were awarded in the same way, one would expect the non-PBS to have a larger impact on ever enrolling in an institution than the PBS since the non-PBS was a guaranteed payment. The fact that the non-PBS does not appear to have affected enrollment suggests the students may have been unaware of the award or that institutions offset the award with reductions in other financial aid.

Although we cannot completely rule out these alternative explanations, we suspect they do not explain the results for two reasons. First, the point estimate in column 3 suggests that the non-PBS had no impact on enrollment. Given the literature on the effect of college subsidies on enrollment, this would only occur if the institutions completely offset the non-PBS with reductions in other financial aid, which is unlikely since institutions most often treat outside scholarship aid favorably (as detailed in footnote 8). Second, Deming and Dynarski (2010) conclude that the best estimates of the impact of educational subsidies on enrollment suggest that eligibility for a $1,000 subsidy increases enrollment by about 4 percentage points, and the coefficient estimate for the impact of the non-PBS is not statistically different from this impact. In contrast, the estimated impact of the PBSs on enrollment, when scaled by how much students actually received, is larger.

Taken together, we believe the evidence suggests that incentives played a key role in changing student behavior. We note this finding is also consistent with Scott-Clayton (2011), who studies the West Virginia PROMISE scholarship—a merit scholarship with continuing eligibility that depends on meeting minimum credit and GPA benchmarks and that had relatively large impacts on student educational attainment. As credit completion tended to be concentrated around the renewal thresholds, she concludes that the scholarship incentive was a key component for the success of the program.

As we noted earlier, the overall survey response rate was only about 58 percent, and response rates for PBS-eligible participants were statistically greater than response rates for students in the control group. We explore the extent to which selection may be affecting our PBS impact estimates in table 9. We reproduce the table 6 estimates in column 1 for ease of comparison.

In column 2 of table 9 we simply include controls for baseline characteristics—student age, sex, race/ethnicity, parental education, first in family to attend college, English or Spanish spoken at home, and standardized responses to exit questions on motivation.²⁵ In column 3 we use the baseline controls to predict treatment status and then use inverse probability weighting to adjust the impact estimates for selection. Adjusting for selection either by controlling for baseline characteristics directly or by inverse probability weighting has very little effect on our estimates of the impact of PBS eligibility on any of the index outcomes. Although both strategies can only address the potential for selection on observable characteristics, they suggest that selection on unobservable characteristics would have to be quite large to overturn our estimates.

In the final columns of table 9, we make use of the strategy of Altonji, Elder, and Taber (2005) to estimate the bias in the PBS estimate under the assumption of equal (standardized) selection on observables and unobservables (column 4) and to calculate the implied ratio of selection on unobservables to selection on observables under the assumption of no PBS effect (column 5).²⁶ For the index of all academic activities, the index of nonacademic activities, and the index of unintended consequences, the estimates assuming equal selection on observables and unobservables suggest that the bias is pushing the treatment effect estimates toward zero. In other words, in the absence of selection bias we would estimate larger treatment effects in absolute value. For the index of quality of educational input, the estimate of the bias assuming equal selection on observables and unobservables suggests that there may be no positive impact of PBS eligibility on the quality of educational input.

So how much selection on unobservables is necessary to overturn our findings? When we translate the bias estimates into ratios of selection on unobservables to selection on observables consistent with no treatment impact, we find that selection on unobservables would have to be 2.6 times as large as selection on observables and in the opposite direction in order to conclude that there is no impact of PBS eligibility on the index of all academic activities. In other words, if survey nonresponse generates positive selection into treatment status based on observable characteristics, the selection must be negative (and much larger) based on unobservable characteristics. This seems unlikely and therefore we conclude that PBS eligibility likely has a positive effect on academic activities. Similarly, we believe that selection on unobservables is unlikely to explain the estimates of the impact of PBS eligibility on nonacademic activities and unintended consequences. In both cases the degree of selection on unobservables does not have to be very large relative to the selection on observables but it does have to be in the opposite direction. In contrast, for the quality of educational inputs, selection on unobservables is more likely to be able to explain our estimated treatment effect. Selection on unobservables that is roughly 65 percent as large as selection on observables would explain all the estimated effect of PBS eligibility on quality of educational inputs.

Although education policy makers have become increasingly interested in using incentives to improve educational outcomes at the postsecondary level, the evidence continues to generate mixed impacts, leading to the question of whether such incentives can actually change student effort toward educational attainment, as suggested by Becker's model of individual decision making. As a whole, we find evidence consistent with this model: Students eligible for PBSs increased effort in terms of the amount and quality of time spent on educational activities and decreased time spent on other activities. Further, it appears that such changes in behavior do not persist beyond eligibility for the PBS, suggesting such incentives do not permanently change students’ cost of effort or their ability to transform effort into educational outcomes.

An important question arising from this study is why the larger incentive payments did not generate larger increases in effort. We offer a few potential explanations worthy of further consideration. First, the result may suggest that students need just a small prompt to encourage them to put more effort into their studies but that larger incentives are unnecessary. Further, it is possible that as the value of the incentive payment (external motivation) increases, students’ internal motivation declines at a faster rate such that negative impacts on intrinsic motivation increasingly moderate any positive impacts of the incentive on educational effort. Finally, these results could also be consistent with students not fully understanding their own “education production function,” that is, how their own effort and ability will be transformed into academic outcomes like grades. Although the students seem to understand that increases in effort are necessary to improve outcomes, they may overestimate their likelihood of meeting the benchmark and underestimate the marginal impact of effort on the probability of meeting the benchmark leading to suboptimal levels of effort.²⁷ Our data do not allow us to thoroughly understand why larger incentive payments did not generate larger changes in behavior, yet understanding why they did not would be important for the optimal design of incentive schemes to improve educational attainment.

Finally, while the evidence from this study of PBSs suggests modest impacts, such grants may nonetheless be a useful tool in postsecondary education policy as they appear to induce positive behavioral changes, and evidence from other similar studies (such as Barrow et al. 2014), suggests that even with small impacts on educational attainment, such relatively low-cost interventions may nonetheless be cost-effective.

We thank Eric Auerbach, Elijah de la Campa, Ross Cole, Laurien Gilbert, Ming Gu, Steve Mello, and Lauren Sartain for expert research assistance; Leslyn Hall and Lisa Markman Pithers with help developing the survey; and Reshma Patel for extensive help in understanding the MDRC data. We are also grateful to Todd Elder for sharing programs for exploring selection bias. Orley Ashenfelter, Todd Elder, Alan Krueger, Jonas Fisher, Luojia Hu, Derek Neal, Reshma Patel, Lashawn Richburg-Hayes, and Shyam Gouri Suresh, as well as seminar participants at Cornell University, the Federal Reserve Bank of Chicago, Federal Reserve Bank of New York, Harvard University, Michigan State, Princeton University, University of Chicago, University of Pennsylvania, and University of Virginia, provided helpful conversations and comments. Some of the data used in this paper are derived from data files made available by MDRC. We thank the Bill & Melinda Gates Foundation and the Princeton University Industrial Relation Section for generous funding. The authors remain solely responsible for how the data have been used or interpreted. The views expressed in this paper do not necessarily reflect those of the Federal Reserve Bank of Chicago or the Federal Reserve System. Any errors are ours.