On the Utility of Positive and Negative Feedback in a Paired-associate Learning Task

Instructive feedback is an integral part of learning. It is provided in the school environment, at home, as well as in the work place. Feedback can serve different functions. It may inform the learner of the correct or expected response when there is no other way of determining what that response should be. It may resolve uncertainty and facilitate learning when knowledge of the correct response is still not fully stored. It also informs the learner about the extent to which learning is successful. Broadly speaking, feedback in a learning task changes its role from being informative (guiding learning) to being increasingly evaluative (assessing learning). The error-related negativity (ERN) is an ERP associated with error commission. It is elicited when participants make an erroneous response in speeded RT tasks (e.g., Gehring, Goss, Coles, Meyer, & Donchin, 1993; Falkenstein, Hohnsbein, Hoormann, & Blanke, 1990) and when the fact that a response was erroneous is communicated by a feedback event (e.g., Miltner, Braun, & Coles, 1997). The ERN elicited by feedback (namely feedback-related negativity[FRN] or fERN) is elicited in time estimation tasks (e.g., Ferdinand, Mecklinger, Kray, & Gehring, 2012; Gruendler, Ullsperger, & Huster, 2011; Oliveira, McDonald, & Goodman, 2007; Miltner et al., 1997), in gambling tasks (e.g., Goyer, Woldorff, & Huettel, 2008; Hajcak, Moser, Holroyd, & Simons, 2007; Gehring & Willoughby, 2002), as well as in learning tasks (e.g., Arbel, Goforth, & Donchin, 2013; Sailer, Fischmeister, & Bauer, 2010; van der Helden, Boksem, & Blom, 2010; Eppinger, Mock, & Kray, 2009; Krigolson, Pierce, Holroyd, & Tanaka, 2009; Pietschmann, Simon, Endrass, & Kathmann, 2008; Holroyd & Coles, 2002). The reinforcement learning theory of the ERN (Holroyd & Coles, 2002) suggests that this component is generated in ACC as a consequence of a phasic decrease in the activity of the mesencephalic dopamine system occurring when the monitoring system evaluates events as worse than expected. It is also hypothesized that this FRN signal is used by ACC for the adaptive modification of behavior. It is still to be elucidated the extent to which the FRN signal represents the process of evaluating performance (signals worse or better than expected outcomes), the process of extracting information from the feedback to facilitate behavior adaptation (i.e., signals the utility of the feedback), or both. Learning tasks appear optimal for the examination of this question. In gambling tasks, in which responses are not learnable, feedback serves as a deliverer or denier of rewards. Its role can be considered stable across the task. In learning tasks, however, the role of the feedback is fluid as feedback serves different functions at different stages of learning. Therefore, learning tasks that capture the changing role of feedback can shed light of the functional significance of the FRN and elucidate the extent to which the process(es) giving rise to feedback related ERPs is(are) related to the utilization of feedback for task performance (i.e., the extent to which feedback is informative). Such learning task was designed by Tricomi and Fiez (2008), who divided a paired-associate learning task into three rounds, each containing 60 pairs presented once within each round. This design allowed the separation of informative feedback (feedback provided during the first round of trials) from evaluative feedback (feedback provided on the following rounds). Their fMRI investigation displayed differences between the processing of informative and evaluative feedback. More specifically, they found an increased caudate activation and decreased left dorsolateral prefrontal cortex (DLPFC) activation as feedback changed its role from being informative to becoming more evaluative (i.e., an “earned” feedback), with minimal caudate and DLPFC activation differences between positive and negative when feedback is purely informative (feedback provided on Round 1). Their data indicated that it was the processing of positive feedback that was associated with the changes in DLPFC activation (Tricomi & Fiez, 2008) and with the relationship between caudate activation and learning outcomes (Tricomi & Fiez, 2012). In a previous report, we demonstrated a relationship between the processing of positive feedback as measured by the FRN and learning outcomes (Arbel et al., 2013). We found that the activity elicited by positive rather than negative feedback was association with learning outcomes. However, in this previous report, a four-choice paired-associate learning task was employed in which positive feedback was more informative than negative feedback. In a four-choice task, a positive feedback confirms the correct selection, whereas a negative feedback implies that there are three other possible correct responses. In light of the differences in the information carried by the positive and negative feedback, the relationship found between positive feedback and learning outcomes could be attributed to the valence of the feedback, the information carried by the feedback, or both. One of the goals of the study reported here is to resolve this uncertainty by employing a two-choice paired-associate learning task in which positive and negative feedback are equally informative. In such a task, whereas positive feedback informs the learner that the current choice is accurate, negative feedback informs the learner that the alternative choice is correct. Additionally, this design permits the evaluation of the extent to which the processing of the very first informative feedback provided in association with an item is indicative of whether or not the item will be learned.

Twenty undergraduate students (5 men, 15 women) aged 19–36 years (mean age = 22.85 years) from the Department of Psychology at the University of South Florida participated in this experiment. Participants reported to be right-handed with no history of developmental disorders or any other neurological deficits. They received course credit for their participation in the study.

A two-choice paired-associate learning task was employed in which participants were instructed to try to learn the names (nonword) of 60 novel objects. Participants were presented with four blocks, which will be referred to as “rounds” in this paper. ERPs were recorded during the first three rounds. During the first round, participants' choices were followed by positive and negative feedback of equal probability (.5 positive, .5 negative). Therefore, associations between objects and names were determined during this round (and were different for each participant) and were kept throughout the remainder of the experiment. With 60 trials in each of the three rounds, the number of positive and negative feedback presentations was identical during the first round (30 trials during which positive feedback was presented, and 30 trials with negative feedback). The number of trials associated with positive and negative feedback during the following two rounds varied based on individual learning speed. The fourth round served to evaluate learning and to allow participants to complete a confidence rating scale for each of the 60 pairs. During each trial, participants were presented with a novel object accompanied by two possible names. Participants were asked to choose one of the two names by pressing one of two buttons on a response box. Each response was followed by a performance feedback indicating whether the participant made the correct choice. Positive feedback was presented as “√√√,” and negative feedback was presented as “xxx.” Each of the 60 objects was presented once within each round.

To evaluate the relationship between the processing of the initial informative feedback and learning outcomes, we segmented the data of Round 1 based on the valence of the feedback (positive and negative) and based on learning outcomes. Learning was determined based on the performance on Round 4. Items that were correctly paired with their name on Round 4 and for which participant chose the confidence rating of 3 or 4 (3 = maybe correct, 4 = sure correct) were categorized as “learned”; items that were not correctly paired on Round 4 and for which the confidence rating was 1 or 2 (1 = sure incorrect, 2 = maybe incorrect) were categorized as “not learned.” Confidence rating data were used to ascertain that correct guesses and incorrect learning are excluded from the analysis. This segmentation yielded four categories:

1. Positive feedback learned: Positive feedback provided during Round 1 to items that were subsequently learned based on performance on Round 4 (received positive feedback on Round 4 with high confidence rating).

2. Positive feedback not learned: Positive feedback provided during Round 1 to items that were not subsequently learned.

3. Negative feedback learned: Negative feedback provided during Round 1 to items that were subsequently learned.

4. Negative feedback not learned: Negative feedback provided during Round 1 to items that were not subsequently learned.

To evaluate the potential differences in the processing of positive and negative feedback across the three rounds, ERPs were segmented based on the valence of the feedback (positive and negative) and based on Rounds (Round 1, Round 2, and Round 3). This segmentation yielded six categories: Round 1-Positive, Round 1-Negative, Round 2-Positive, Round 2-Negative, Round 3-Positive, and Round 3-Negative.

The novel objects were borrowed from Kroll and Potter (1984). Nonwords were produced from the ARC Nonword Database (Rastle, Harrington, & Coltheart, 2002). The nonwords were in three-letter consonant–vowel–consonant (CVC) format (e.g., joz) and were phonologically legal in English. The number of orthographic neighbors (i.e., real words in English that are written similarly) was between 0 and 10 (M = 5.9); the number of phonological neighbors (i.e., real words in English whose pronunciation is similar) was between 5 and 20 (M = 14).

The EGI System 200 was used to acquire and analyze dense array EEG data. The EEG was recorded using 129-channel HydroCel Geodesic Sensor Nets from EGI. The EEG was continuously recorded at a 250-Hz sampling rate with a band pass of 0.1–100 Hz. The electrode impedances were kept below 50 kΩ. The continuous EEG data were filtered using an offline 40-Hz low-pass filter. The filtered data were then segmented into 800-msec long epochs, each starting 200 msec before the presentation of the feedback stimulus and ending 600 msec after the feedback presentation. Baseline correction was performed on the 100 msec preceding the onset of the feedback. An algorithm developed by Gratton, Coles, and Donchin (1983) for offline removal of ocular artifacts was used to correct for eye movements and blinks. On average 0.018 of the 160 ERP epochs was excluded from the analysis due to excessive artifacts. Averages of the artifact free baseline corrected epochs were calculated for each type of feedback (Positive and Negative) and were separated by rounds (Round 1, Round 2, and Round 3) and by learning outcomes based on performance on Round 4 (learned, not learned). No ERP data were collected during Round 4. The averaged EEG epochs were re-referenced to linked mastoid.

Analysis was done on 600-msec-long epochs, starting at the onset of the feedback stimuli and ending 600 msec following the feedback. To reduce the dimensionality of the large data set and disentangle overlapping ERP components, a spatiotemporal PCA as described by Spencer, Dien, and Donchin (2001) was utilized. The spatial PCA was performed by computing the covariance among electrode sites across the time points of each of the feedback stimuli and participants, yielding a set of spatial factors. In the next step of the analysis, the factor scores for each of the participants, feedback stimuli, and electrodes were computed for all time points. The plot of these factor scores across the time axis created “virtual ERPs” (Spencer et al., 2001), which were submitted to a temporal PCA, analyzing the covariance among time points for each of the spatial factors, feedback stimuli, and participants. The resulting temporal factor scores for each spatial factor were used to measure the activity in the ERP with the morphology and scalp distributions of interest. For both spatial and temporal PCAs, the factors that were required to account for 95% of the variance in the input data set were retained for Varimax rotation. The factor scores of the temporal and spatial factors of interest were used for statistical analysis.

On average, participants committed errors on 47.5% of the trials on Round 2, 45.3% on Round 3, and 40.8% on Round 4. This relatively high error rate was expected as each novel object was presented only four times (once during each of the four rounds) throughout the experiment. Although error rate was relatively high, it is important to note that on average 50% (SD = 12, range = 40–80%) of the correct responses on Round 1 were also correct on Round 2, 60% of correct response on Round 2 were also correct on Round 3 (SD = 11, range = 45–77%), and 70% (SD = 13, range = 56–95%) of the correct responses on Round 3 were also correct on Round 4. These findings suggest that, although participants did not learn all of the associations by Round 4, they became more consistent in their correct responses, indicating that their responses after Round 1 were not merely guesses. Repeated-measure analysis of error rate revealed a round effect, F(3, 57) = 5.72, p = .01. Post hoc paired comparison indicated that the reduction in error rate occurred between Round 1 and Round 2, F(1, 19) = 4.6, p = .047, and between Round 3 and Round 4, F(1, 19) = 9.93, p = .006, whereas no significant differences were found between Rounds 2 and 3, F(1, 19) = .63, p = .43. We first defined learning as a correct response on Round 4 accompanied by a high-confidence rating (a rating of 3 or 4). An examination of learning as a function of feedback valence on Round 1 indicated that more learned items were associated with positive feedback on Round 1 than with negative feedback, F(1, 19) = 6.4, p = .02. An examination of the confidence rating on Round 4 as a function of feedback valence on Round 1 indicated no significant differences, F(1, 19) = 2, p = 1.7, suggesting that confidence rating on Round 4 was not affected by valence on Round 1. When learning was defined as correct responses on Round 4 regardless of confidence rating, the effect of feedback valence remained, F(1, 19) = 8.6, p = .009. These results suggest that, although positive and negative feedback on Round 1 were equally informative and equally probable, positive feedback resulted in better learning outcomes than negative feedback.

To examine whether the processing of the initial informative feedback (feedback on Round 1) is different from the processing of the proceeding feedback stimuli which become increasingly evaluative (feedback on Rounds 2 and 3), separate averages were computed for each of the rounds. Figure 1 presents the grand-averaged ERP data from electrode FCz in which the FRN is typically examined. As can be seen in the figure, the ERPs elicited during Round 1 appear different from those elicited during Rounds 2 and 3. The difference can be described as a smaller difference between the activities associated with positive and negative feedback on Round 1 in comparison with the other two.

A more detailed analysis is provided by the STPCA. The spatial factor that captures the FRN activity is SF1 (fronto-central; see Figure 2). The virtual ERPs presented in Figure 2 seem to mirror the activity shown in the grand average of FCz, with a small difference between positive and negative feedback in Round 1. Temporal factor 4 was selected for further analysis as it represents the epoch during which the FRN was elicited (negativity with a latency of about 250 msec).

A 2 × 3 repeated-measure analysis with two levels of Feedback Valence (positive and negative feedback) and three levels of Rounds (Round 1, Round 2, and Round 3) was conducted on the scores of SF1-TF4. A main effect of Feedback Valence was found, F(1, 19) = 35.46, p < .0001. A main effect of Rounds was not found, F(2, 38) = 1.82, p = .17. However, an interaction between Valence and Rounds was found, F(2, 38) = 3.4, p = .05. A post hoc paired comparison revealed that this effect was driven by the difference between the activity elicited by positive feedback on Round 1 and the activity elicited by positive feedback on the next two rounds, such that the activity associated with positive feedback was the smallest during Round 1 (positive feedback: Round 1 vs. Round 2, t(1, 19) = −3.24, p = .005; Round 1 vs. Round 3, t(1, 19) = −2.56, p = .02). No differences were found between the negative feedback on Round 1 and the other two rounds (negative feedback: Round 1 vs. Round 2, t(1, 19) = −0.48, p = .96; Round 1 vs. Round 3, t(1, 19) = −0.69, p = .5). These results support the visual evaluation of a smaller difference between positive and negative feedback during the first round.

A separate analysis has been conducted (including PCA) to evaluate the potential relationship between the processing of the initial informative feedback (feedback processing on Round 1) and learning outcomes. To examine this relationship, we examined the ERPs obtained on Round 1 as they were related to performance on Round 4 which served as the learning outcome measure. A 2 × 2 (2 levels of feedback valence and 2 levels of learning) repeated-measure analysis was conducted on the factor scores of the FRN component (SF1-TF3, fronto-central with a latency of about 250 msec; see virtual ERPs in Figure 3). A Valence effect was found, F(1, 19) = 11.69, p = .003. No Learning effect, F(1, 19) = .44, p = .51, was found. However, an interaction between Feedback Valence and Learning was found, F(1, 19) = 4.2, p = .05. Post hoc paired comparison suggests that differences related to learning were only associated with positive feedback, such that positive feedback provided to items that were subsequently learned was different from positive feedback provided to items that were not learned, t(1, 19) = −3.6, p = .006. No such differences were found for the negative feedback, t(1, 19) = 0.78, p = .45. These results suggest that the processing of the very first positive feedback is associated with learning outcomes and therefore may be indicative of successful and unsuccessful learning.

These findings of the association between the processing of the initial feedback and learning outcomes called for an additional analysis of the possible relationship between the feedback provided on Rounds 2 and 3 and learning outcomes. No associations were found between positive and negative feedback provided on Rounds 2 and 3 and learning outcomes (i.e., no Learning effect, F(1, 19) = 3.05 p = .58, or an interaction between Valence and Learning, F(1, 19) = 1.07, p = .31).

The study examined two questions related to the processing of feedback in a learning task. The first question was concerned with the extent to which the neurophysiological markers of feedback processing are sensitive to the role of the feedback in the learning process. The second question was whether the processing of the initial informative feedback is related to learning outcomes. Our data point to processing differences between the initial informative feedback and the proceeding performance feedback stimuli. More specifically, we found that the ERP activity elicited in association with positive feedback changed as the feedback's role developed from being purely informative to being both informative and evaluative, with a reduction in FRN amplitude from the first round to the other two. Our findings that changes in the FRN amplitude were associated with positive rather than negative feedback are in line with previous reports of greater modulation of the FRN to positive feedback (e.g., Arbel et al., 2013; Kreussel et al., 2012; Baker & Holroyd, 2011; Foti, Weinberg, Dien, & Hajcak, 2011; San Martin, Manes, Hurtado, Isla, & Ibanez, 2010; Eppinger, Mock, & Kray, 2009; Eppinger, Kray, Mock, & Mecklinger, 2008; Cohen, Elger, & Ranganath, 2007). Interestingly, on Round 1, when positive and negative feedback were equally informative and probable, an FRN was elicited by both positive and negative feedback, with greater magnitude of FRN to negative feedback. The examination of the FRN on Round 1 also revealed that the activation associated with positive feedback was related to the learning outcomes. Positive feedback provided to items that were subsequently learned elicited a larger FRN when compared with positive feedback provided to items that were not learned. These results are in line with our previous findings of a relationship between FRN elicited by positive feedback in a learning task and learning outcomes (Arbel et al., 2013). The results of the current experiment suggest that, even when the positive and negative feedback are equally informative and probable, it is the positive feedback which is associated with learning outcomes.

In the next section, we will examine our findings within the framework of existing theories and within our proposed utility account.

Growing evidence of greater modulation of the FRN to positive feedback has led to some revisions to the reinforcement learning theory of the ERN. In the original report (Holroyd & Coles, 2002), the FRN was presented as a negative signal of worse than expected outcomes. It was later suggested that the FRN is an N200 elicited by events that fail to indicate that a task goal has been achieved (Holroyd, Pakzad-Vaezi, & Krigolson, 2008) or by any unexpected outcomes regardless of valence (Holroyd, Krigolson, & Lee, 2011) and that the amplitude of this component is reduced by reward, which elicits the feedback-related correct response positivity (fCRP) in the same time window. It is important to note that a reward prediction component, namely the P2a, was previously described by Potts, Martin, Burton, and Montague (2006). Although the N200 fCRP account is appealing, it is not yet clear whether the N200 is elicited by all unexpected outcomes and is being reduced by the fCRP or whether it is elicited only by negative outcomes. It is also not clear whether the fCRP is sensitive to reward expectancy or to the assessment of task achievement. In a learning task, the initial positive feedback is assumed to be the most unpredicted positive outcome. Therefore, if the fCRP or P2a reflects the processing of unexpected reward, in a learning task it should be the largest during the first phase of the learning process and to be reduced as positive feedback becomes more predictable. On the other hand, if the fCRP represents the assessment of task achievement, one would expect the amplitude of this positivity to become larger as learning progresses. It appears that these two possible interpretations create opposite predictions in a learning task. Our data do not support the reward expectancy account, as the fCRP became more positive during Rounds 2 and 3 when positive feedback was less surprising. If we adopt the alternative account that the fCRP represents the assessment of task achievement, it can be argued that as the learning process progressed, the increase in fCRP amplitude reflected a clearer assessment of the positive feedback as an indication of task achievement. Our finding that positive feedback that was associated with successful learning elicited a larger negativity does not sit well with either the expectancy account of the fCRP or with the task achievement one. If the fCRP reflects better than expected outcomes or an achievement of a task goal, one would expect that larger positivity will be associated with successful learning.

Several FRN theories discuss the FRN as a product of the assessment of outcomes in relation to expectancy. However, there are conflicting views about the relationship between valence and the processing of expectancy. Whereas the reinforcement learning account of the FRN distinguishes between the processing of worse than expected from that of better than expected (or unexpected reward), others suggest that the FRN is sensitive to violation of expectancy regardless of feedback valence. Ferdinand et al. (2012) reported that in an experiment whose design allowed the separation of valence and expectancy, FRN was elicited by both positive and negative feedback when expectancy was violated. Our findings that FRN was elicited by both positive and negative feedback during the first round can be explained within the framework of Ferdinand et al.'s expectancy account. One may suggest that during the first round of our experiment participants did not form specific expectancies and therefore any feedback was in a sense unexpected. This explanation is difficult to test, and the validity of the notion that participants were equally surprised by positive and negative feedback during this phase is in our opinion questionable. A more likely scenario is that learners had some expectations that varied based on previous learning experiences. Generally speaking, it would have been expected for learners to be more surprised by positive feedback than by negative feedback at this initial stage of learning. In that case, the expectancy account will not fit well with our findings as the larger FRN amplitude was associated with negative feedback.

We propose that the activity associated with the processing of feedback as depicted by the FRN is indicative of the utility of the feedback. More specifically, we suggest that at the stage at which feedback is “purely” informative, the utility rather than the valence of the feedback is given a processing “priority.” Because both negative and positive feedback stimuli are equally informative at this stage, they both elicit the FRN. The larger FRN magnitude associated with negative feedback during this round may suggest that, although the utility of the feedback played a primary role, feedback valence also affected the processing of this informative feedback. The reduction in the amplitude of the FRN associated with positive feedback is consistent with this view. In our data set, the amplitude of the FRN elicited in association with positive feedback decreased from the first round to the other two rounds. Within our proposed utility account, this change may reflect the process of decreased relevance of the positive feedback to the learning process after the correct associations were learned (on Rounds 2 and 3). Whereas positive feedback loses its relevance after the correct association has been learned, negative feedback remains relevant for task performance across the three rounds as the learners continue to use the feedback to extract information. Within this framework, negative feedback was associated with a stable activation of this process as it served to inform the learner of the correct association throughout the task, whereas positive feedback became redundant or less informative after the correct associations have been learned. Given that in this task negative feedback was more likely to follow prelearning errors rather than postlearning slips, we hypothesize that a reduction in the FRN amplitude would have been observed after negative feedback had lost its relevance for task performance if more trials were presented. Although this hypothesis cannot be tested with our data, it is in line with previous reports of reduced FRN amplitude in probabilistic learning tasks after the correct associations have been learned (e.g., Holroyd & Coles, 2002).

Our data suggest that the activity associated with positive feedback during the initial round was related to learning outcomes. One explanation for the results could be the initial positive feedback is more important for the learning process than the initial negative feedback. This suggestion is supported by our behavioral data that indicate that items that received positive feedback on Round 1 were more likely to be subsequently learned when compared with items that received negative feedback on Round 1. An alternative explanation is that, although the assumption was that in a two-choice paired-associate learning task positive and negative feedback are equally informative, it is possible that extracting information from the positive feedback is less demanding than extracting information from the negative feedback. Whereas positive feedback reinforces the accuracy of the current choice, negative feedback requires the learner to reject the current hypothesis and to adopt the alternative, resulting in the need to switch the item to be held in working memory. The utility account can be applied to explain the amplitude differences between positive feedback that resulted in successful learning and positive feedback that resulted in unsuccessful learning. Within the framework of this account, larger negativity was associated with greater utilization of the feedback.

Within our proposed utility account, we suggest that the utility is defined by the role of the feedback in a particular task. For example, FRN known to be elicited in association with losses and its amplitude is reported to be sensitive to violations of reward expectancies (e.g., Bellebaum, Polezzi, & Daum, 2010; Holroyd & Krigolson, 2007; Holroyd, Nieuwenhuis, Yeung, & Cohen, 2003). In these cases, the feedback is not informative for task performance in the same manner feedback in a learning task is. However, in a task in which the participant's goal is to gain as much money as possible, feedback can be viewed as providing relevant information that may be used by the participant to adjust future choices or make predictions. We suggest that the process giving rise to the FRN is concerned with extracting relevant information from the feedback. It is not merely a “good” versus “bad” process or one that weighs outcomes in comparison with expected outcomes, but rather a more complex process that may extract different types of information from the feedback depending on the needs of the processor of information. The lack of FRN in association with gains in gambling tasks may point to a separate activation that is concerned with the process of reward delivery. It is possible that the feedback-related ERPs reflect a combination of processes associated with both the utility of the feedback and its valence (or reward value).

It is important to emphasize that, in the current report, differences in the processing of positive feedback for items that were subsequently learned and those that were not were detected following the very first feedback provided in association with a particular new object. The results imply that signs of successful and unsuccessful learning may be detected as early as the initial positive feedback provided in a learning task. Future studies should examine the extent to which predictions can be made about learning outcomes based on the initial processing of positive feedback.

Reprint requests should be sent to Yael Arbel, MGH-Institute of Health Professions, 36 1st Ave., Boston, MA 02129, or via e-mail: [email protected], [email protected].