Abstract

When people talk, their speech is often accompanied by gestures. Although it is known that co-speech gestures can influence face-to-face communication, it is currently unclear to what extent they are actively used and under which premises they are prioritized to facilitate communication. We investigated these open questions in two experiments that varied how pointing gestures disambiguate the utterances of an interlocutor. Participants, whose event-related brain responses were measured, watched a video, where an actress was interviewed about, for instance, classical literature (e.g., Goethe and Shakespeare). While responding, the actress pointed systematically to the left side to refer to, for example, Goethe, or to the right to refer to Shakespeare. Her final statement was ambiguous and combined with a pointing gesture. The P600 pattern found in Experiment 1 revealed that, when pointing was unreliable, gestures were only monitored for their cue validity and not used for reference tracking related to the ambiguity. However, when pointing was a valid cue (Experiment 2), it was used for reference tracking, as indicated by a reduced N400 for pointing. In summary, these findings suggest that a general prioritization mechanism is in use that constantly monitors and evaluates the use of communicative cues against communicative priors on the basis of accumulated error information.

INTRODUCTION

In face-to-face communication, people tend to enrich their language with multiple sources of covert information expressed in prosody, eye gaze, facial expression, and gestures. It is currently unclear to what extend this covert information plays an active role in communication and, if so, whether and how the cognitive system prioritizes the use of this information. We investigated these open issues in the context of co-speech gesture information and explored the communicative value and prioritization of pointing gestures (i.e., deixis) during the perception of face-to-face discourses.

In a communicative situation, it is of utmost importance that all interlocutors can unambiguously identify the characters talked about in their narrative. This identification is more effortful when an interlocutor refers to a character indirectly by, for instance, using a pronoun (i.e., she) instead of a character's name. Here, the character information needs to be retrieved on the basis of the forgoing narrative, and the perceiver has to track what the pronoun refers to. Interestingly, in face-to-face communication, this reference tracking can be assisted by co-speech gestures (Clark & Bangerter, 2004), in particular, a special form of pointing gestures (Kita, 2003). Consider a conversation about classical literature where your interlocutor says, “As a student, I've read several books of Shakespeare and Goethe.” While uttering “Shakespeare,” she points to the left, and while uttering “Goethe,” she points to the right. After the conversation lingers on, she replies to one of your questions: “Well, I like his books better because they have more suspense,” accompanying the ambiguous pronoun “his” with a pointing gesture to the left, thereby referring to Shakespeare (see also Goodrich Smith & Hudson Kam, 2012).

Thus, although pointing1 is typically thought of identifying a physically present object (Peeters & Özyürek, 2016; Peeters, Chu, Holler, Hagoort, & Özyürek, 2015), this gesture type can also identify objects that are absent, through “abstract” gesturing in a shared empty space that is visible to all interlocutors (i.e., gesture space; see Stec & Huiskes, 2014; see also McNeill, Cassell, & Levy, 1993). During abstract pointing, different parts of gesture space are hypothesized to temporarily attain a referential value for the purpose of discourse.2 Abstract pointing can therefore be seen as metaphoric in that the gesture indicates a space, but space has a nonspatial meaning in that it stands for the referent that otherwise could not have a spatial locus. McNeill (2003) suggests that perceivers may use this pointing information for tracking referents in discourse. A discourse or, more generally, storytelling can be described on three hierarchical levels with the narrative level followed by the para-narrative level and then the meta-narrative level, which is on top of the hierarchy (Hühn, Pier, Schmid, & Schönert, 2009; see also McNeill et al., 1993). Abstract pointing is typically done on the narrative level where it can have a referential function (McNeill, Levey, & Duncan, 2015; McNeill et al., 1993). The narrative level is the most detailed level of a story structure, which refers to the emplotted events. Abstract pointing thus seems to be on the basis of a discourse. Typically, an abstract pointing gesture first marks the existence of a referent after which it helps via its repetition to maintain reference to it. By means of this repetition, it differentiates between the initial referent and other possible referents in the context, thereby helping to reduce verbal working memory load. From a more general perspective, one could suggest that both concrete and abstract pointing functions to direct a listener's attention to some topic of conversation and that it additionally functions to smooth the turn-taking process in discourse (cf. Healy, 2012). In summary, adult speakers often use speech-accompanying gestures as a device to build up and maintain a coherent discourse where they repeat particular gestures that maintain continuity with respect to their location in space, the hand with which they are produced, and/or their form (for a detailed discussion, see McNeill & Levy, 1993).

In contrast to the idea of McNeill (2003) that perceivers may use abstract pointing for tracking referents in discourse, So, Kita, and Goldin-Meadow (2009) found that abstract pointing during speech production mostly accompanies referents that have already been introduced and may thus be a conversationally redundant means to reduce cognitive demand for the speaker (Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014; Marstaller & Burianová, 2013). However, during speech comprehension, perceivers are sensitive to mismatches of speech and abstract pointing when watching filmed interviews on dualistic themes (Gunter, Weinbrenner, & Holle, 2015). As in the example above, the speaker in Gunter et al. (2015) pointed systematically to the left and right to refer to Shakespeare or Goethe. In the last sentence, the name of one of the classical writers occurred, accompanied by a pointing gesture that was either consistent (i.e., left for Shakespeare) or inconsistent (i.e., right for Shakespeare) with its originally established position in gesture space. The mismatches between speech and abstract pointing led to an increased N400–P600 response in the event-related brain potential, showing that the participants tried to integrate gesture and speech but failed. Note that such mismatch data do not show whether participants would actually have used the outcome of correct speech–gesture integration during communication. On the basis of gesture–speech mismatch experiments, it is a mere assumption that gestures are always used in a complementary fashion in communication.

Gunter et al. (2015) therefore did not decisively show that abstract pointing actually aids referent identification during face-to-face communication. To investigate whether abstract pointing is indeed a primary means of referent identification during speech comprehension, we conducted two EEG studies that did not rely on gesture–speech mismatches but rather on disambiguation, which can only take place when abstract pointing and speech have been integrated correctly. This will provide novel insight on the communicative influence of abstract pointing and will direct our considerations on how communicative cues in general impact cognition.

EXPERIMENT 1

In the first experiment, participants watched a video of an interview where two persons discussed topics of a dualistic nature such as the classical literature of Goethe and Shakespeare (see Appendix). During the first part of a dialogue on a particular topic, the interviewee conducted abstract pointing, thereby establishing the representation of two referents in gesture space (i.e., Goethe to the left and Shakespeare to the right). Then, the interviewee uttered a sentence that was initially ambiguous with regard to the referent but was disambiguated later in the utterance when the target word occurred (e.g., “Then this author would win, because I've rarely read something as beautiful as Shakespeare's Romeo and Juliet.”). The ambiguous element (i.e., “this author”) was linked to three experimental conditions. In the first condition (here called “same-referent” condition), the pointing gesture of the actress and the subsequent target word referred to the same referent (i.e., pointing to the right for Shakespeare). In the second condition (here called “different-referent” condition), the pointing and the subsequent target word referred to different referents (i.e., pointing to the left for Shakespeare). In the third condition (here called “baseline” condition), no pointing gesture was provided. This baseline condition made it possible to examine directly whether abstract pointing is beneficial for communication.

On the basis of our earlier ERP study (Gunter et al., 2015), we inferred that abstract pointing allows to activate/access referent information. Thus, pointing in the same-referent condition will activate the target word, whereas pointing in the different-referent condition should activate another word. Therefore, we hypothesized that the target word is easier to process in the same-referent condition compared with the different-referent condition. In the baseline condition, there is no information revealing a particular referent. This will lead to a similar activation of both referents (i.e., target words). Accordingly, we hypothesized that the same-referent condition is easier to process than the baseline, whereas the different-referent condition should be more difficult. These differences in processing difficulty can be measured by ERP responses to the target words (Shakespeare/Goethe). We hypothesized that, when co-speech abstract pointing aids referent identification in speech, the same-referent condition relative to the baseline condition should facilitate lexical–semantic retrieval of the target word, as reflected in the ERP by an amplitude decrease of the N400. The N400 is a negative deflection that peaks roughly around 400 msec after stimulation, the amplitude of which increases with lexical and semantic retrieval demands from long-term memory in the light of a preceding context (Kutas & Federmeier, 2011). Similarly, the different-referent condition, relative to the baseline condition, should also show a larger N400, because the wrong referent had been preactivated by the gestures. In addition, we hypothesized the different-referent condition, relative to the baseline condition, to induce a revision3 of the current mental representation of what is being communicated (MRC; Brouwer, Fitz, & Hoeks, 2012), as indicated by an amplitude increase of the P600 ERP component. The P600 is a positive deflection that peaks around 600 msec after stimulation, the amplitude of which increases with reanalysis demands (van de Meerendonk, Kolk, Chwilla, & Vissers, 2009; Kuperberg, 2007) in cases of syntactic (Osterhout & Holcomb, 1992), semantic (Kim & Osterhout, 2005), and general communicative (Brouwer et al., 2012) conflicts in language comprehension. The concept of the MRC is very useful in the current context because referent identification is done on a narrative level, which is projected in the MRC. Thus, in the present experiment, the P600 was used as a marker of updating processes related to the representation of the communicative situation and the N400 as a marker of lexical access.

Methods

Participants

Forty-nine German native speakers participated after giving written informed consent, following the guidelines of the ethics committee of the University of Leipzig. They were paid €7 per hour of participation. To ensure a reasonable signal-to-noise ratio in the ERPs, 13 participants were excluded from further analysis because they did not satisfy our a priori criterion of providing at least 20 valid experimental trials of 30 total trials across conditions after artifact rejection (see below). Therefore, we analyzed the data of 36 right-handed (mean laterality quotient = 89.4, SD = 11.0; Oldfield, 1971) participants (18 women; mean age = 23.9 years, SD = 3.1 years) with normal or corrected-to-normal vision and no self-reported hearing or neurological impairment.

Stimuli

The participants watched a conversation between an interviewer and an interviewee about 90 dualistic topics (e.g., Goethe vs. Shakespeare). Each topic consisted of an establishing phase and a critical phase (Figure 1). In the establishing phase, the interviewee situated the two referents in gesture space in front of her by pointing to the left while saying Goethe and to the right when Shakespeare was uttered. In the subsequent critical phase, the interviewee responded to a question. The interviewee's response was at first ambiguous (e.g., “Then this classic would win….”) and ended downstream with an explicit mentioning of the intended referent (e.g., “…because I've rarely read something as beautiful as Goethe's Faust.”). In one third of the cases, the ambiguous part was accompanied by a pointing gesture that matched the referent's site (same-referent condition). In another third, the pointing mismatched the referent's side (different-referent condition), and in the last third, there were no gestures at all (baseline condition). During the critical question, only the interviewer was visible on the screen; during the critical utterance, only the interviewee was shown. For the same-referent and different-referent conditions, the cut between both shots was placed before the gesture was conducted. For the baseline condition, the cut was placed after the gesture, making sure that the gesture could not be observed (Figure 2; Table 1).

Figure 1. 

The top (A) shows the three different shots used for videotaping. The left picture shows the medium shot of the interviewer; the middle picture, the long shot; and the right picture, the medium shot of the interviewee conducting an abstract pointing gesture. The bottom (B) shows a graphical representation of the basic structure of an interview topic.

Figure 1. 

The top (A) shows the three different shots used for videotaping. The left picture shows the medium shot of the interviewer; the middle picture, the long shot; and the right picture, the medium shot of the interviewee conducting an abstract pointing gesture. The bottom (B) shows a graphical representation of the basic structure of an interview topic.

Figure 2. 

Sketch of how the three gesture conditions were created. In the baseline condition, the cut was postponed leading to no gesture-related referent information. In the same-referent or different-referent condition, opposing gestures were presented. In this example, “Goethe” was established on the left; and “Shakespeare,” on the right side of gesture space (as seen from the viewpoint of the participant).

Figure 2. 

Sketch of how the three gesture conditions were created. In the baseline condition, the cut was postponed leading to no gesture-related referent information. In the same-referent or different-referent condition, opposing gestures were presented. In this example, “Goethe” was established on the left; and “Shakespeare,” on the right side of gesture space (as seen from the viewpoint of the participant).

Table 1. 

Example for the Three Versions of a Topic's Critical Response in Experiment 1

Referent of Gesture and SpeechUtterance of the Actress
(1) Baseline Da würde der Klassiker[0] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[0] would win, because I've rarely read something as beautiful as Goethe's Faust.b 
(2) Same Da würde der Klassiker[Goethe] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[Goethe] would win, because I've rarely read something as beautiful as Goethe's Faust.b 
(3) Different Da würde der Klassiker[Shakespeare] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[Shakespeare] would win, because I've rarely read something as beautiful as Goethe's Faust.b 
Referent of Gesture and SpeechUtterance of the Actress
(1) Baseline Da würde der Klassiker[0] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[0] would win, because I've rarely read something as beautiful as Goethe's Faust.b 
(2) Same Da würde der Klassiker[Goethe] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[Goethe] would win, because I've rarely read something as beautiful as Goethe's Faust.b 
(3) Different Da würde der Klassiker[Shakespeare] gewinnen, denn so etwas schönes wie bei Goethes Faust habe ich selten gelesen.a 
Then this classic[Shakespeare] would win, because I've rarely read something as beautiful as Goethe's Faust.b 

The referent information indicated by abstract pointing either did not specify any referent in the baseline condition (1) or was similar (2) or different (3) from speech. To balance the two referents of the stimulus material, the target word “Goethe” was changed into “Shakespeare,” and “Faust” was changed into “Romeo and Juliet.” Gesture side was counterbalanced across participants.

ERPs were measured at the onset of the italicized words. Subscripts contain the referent indicated by abstract pointing (aOriginal. bTranslation).

The variables gesture referent (same, different, or baseline) and gesture side (left or right) were counterbalanced. Each participant watched only one version of each topic. To prevent participants from predicting the occurrence of the critical phase, the interviewee performed different numbers of establishing gestures. In one third of the topics, she conducted two per side; in another third, three; and in the last third, four gestures per side. Within a topic, she always used the same number of establishing gestures per side. Within a single participant, the levels of all mentioned variables were balanced and pseudorandomized. The experiment was carried out in two sessions, containing 42 topics each; session order was counterbalanced. Details concerning interview preparation, equipment, shooting, and postproduction of the stimulus material can be found in the appendix.

Procedure

The participants sat in a dimly lit room and were informed about the EEG's susceptibility to artifacts from body and eye movements. To keep participants attentive throughout the experiment, a memory task was included. After each meta-topic, participants had to answer three dual-choice questions about the videos by pressing a key, after which immediate feedback was given. Participants were instructed to put the emphasis on accuracy rather than speed. The memory task was never about the content of a topic's critical phrase or the gestures. For instance, for the topic “computer versus laptop,” participants were asked: “Which pointing device is used by Sabine?” In this case, the right button press meant “touchpad”; the left button press meant “TrackPoint.” After each video, there was a pause of self-determined length. The videos subtended approximately 9° visual angle horizontally and 7° vertically. Each of the two experimental sessions lasted approximately 2.5 hr, which included the preparation, the actual experiment, and an additional debriefing in the last session.

The EEG was recorded using 51 Ag–AgCl electrodes, positioned according to the International 10–10 system, and amplified using a PORTI-32/MREFA amplifier from DC to 135 Hz at a sample rate of 500 Hz. Sternum served as ground. Impedances were kept below 5 kΩ. During data acquisition, the EEG was referenced against the left mastoid; a linked mastoid reference was calculated offline. Horizontal and vertical EOG was measured. Blink artifacts were corrected based on a logistic regression method (Hoffmann & Falkenstein, 2008), affecting 7.6% of trials on average. Muscle artifacts were detected automatically (sliding window, 200 msec) as well as epochs exceeding an amplitude of 40 μV (EOG channels) or 30 μV (EEG channels). A final manual check was conducted. Mean rejection rate was 16.2%. On average, 25.3 (SD = 2.6) same-referent, 25.2 (SD = 2.48) different-referent, and 24.8 (SD = 3.1) baseline trials entered the analysis.

ERPs were time-locked to the onset of the target word. Epochs lasted from −200 to 1000 msec after word onset. The 200 msec before word onset served as baseline for later correction. Single-participant averages were calculated for the three conditions. Ten ROIs were a priori defined: anterior outer left (AF7, F5, FC5), anterior inner left (AF3, F3, FC3), anterior central (AFZ, FZ, FCZ), anterior inner right (AF4, F4, FC4), anterior outer right (AF8, F6, FC6), posterior outer left (CP5, P5, PO7), posterior inner left (CP3, P3, PO3), posterior central (CPZ, PZ, POZ), posterior inner right (CP4, P4, PO4), and posterior outer right (CP6, P6, PO8). On the basis of visual inspection, the P600 was analyzed in a time window between 550 and 750 msec.

A repeated-measures ANOVA using Referent (same, different, and baseline), L/R (outer left, inner left, center, inner right, outer right), and Ant/Pos (anterior, posterior) as within-participant variables was calculated for the time window. Only effects involving the variable of interest, Referent, will be reported. Where appropriate, corrected p values were calculated (Greenhouse & Geisser, 1959). As the difference between baseline and both gesture conditions was the main focus of Experiment 1, we carried out two additional Bonferroni-corrected t tests exploring the difference between the baseline and the same-referent as well as the difference between the baseline and different-referent conditions.

Results

Memory Task

On average, the participants answered 94.3% (SD = 4.9%) of the memory questions correctly showing that the stimulus material was processed attentively.

ERPs

As can be seen in Figure 3, the different-referent condition showed a positive deviation compared with the baseline and same-referent conditions starting around 550 msec with a scalp distribution roughly focused between CPz and Pz.

Figure 3. 

Detailed view of relevant electrodes and topographical maps comparing the different-referent condition with the same-referent and baseline conditions.

Figure 3. 

Detailed view of relevant electrodes and topographical maps comparing the different-referent condition with the same-referent and baseline conditions.

The ANOVA of the 550- to 750-msec time window showed a significant main effect for Referent (F(2, 70) = 3.27, p = .04, ε = .98). The t tests revealed a significant difference between the baseline and different-referent conditions (t(35) = −2.49, p = .02) and a nonsignificant difference between the baseline and same-referent conditions (t(35) = −0.27, p = .79). Thus, when the referent turns out to be different, as was inferred by the gesture, a P600 is elicited.

Discussion

For Experiment 1, we had hypothesized that abstract pointing is beneficial for referent identification in face-to-face communication. Although the ERP results during disambiguation indicate that referential mismatches are identified (i.e., an increased P600 for the different-referent condition as compared with the baseline condition), we found no evidence for an additional communicative benefit from the gestures (i.e., no N400 difference between the baseline and same-referent conditions) and thus no indication that abstract pointing helps referent identification. If abstract pointing had helped referent identification, we should instead have observed a reduced N400 for the same-referent condition as compared with the baseline condition. This also entails that the P600 effect during the different-referent condition cannot be related to a conflict between expected and given referents, as it is highly probable that, on the basis of the gestures, there is no expectation for a particular referent (no N400 effect). Hence, the current P600 effect most likely indicates some kind of cue validity check, which would be in accordance with the general idea that the P600 relates to monitoring processes (van de Meerendonk et al., 2009; van Herten, Chwilla, & Kolk, 2006; see General Discussion). Thus, although it seems that abstract pointing was not very helpful for predicting a particular referent in this experiment, it certainly is not the case that gestures were discarded completely and incongruous gesturing was noted very clearly.

In contrast to our hypothesis that abstract pointing has a beneficial effect for communication, the baseline and same-referent conditions did not show a differential N400 suggesting that adding valid abstract pointing is not facilitating the integration of the target word. Such a result is in contrast to most of the gesture literature where numerous observations and experiments showed beneficial and general effects of gestures (cf. iconic gestures: Obermeier, Dolk, & Gunter, 2012; Holle & Gunter, 2007; McNeill, 1992, 2003; emblems: Gunter & Bach, 2004; Nakamura et al., 2004). Possibly, abstract pointing dissociates functionally from iconic gestures and emblems because abstract pointing gestures do not contain predefined semantic information, and their communicative function lies rather at a discourse level. Still, a substantial fraction of participants in prior studies on speech production used the spatial location of their own gestures for referential purposes (Alamillo, Colletta, & Kunene, 2010; So et al., 2009); likewise, speakers and addressees use gestures and their spatial location to mutually identify referents, across longer stretches of natural dyadic discourse (Stec & Huiskes, 2014).

Thus, before rejecting our hypothesis that abstract pointing helps reference identification, we have to carefully explore the experimental design of the current study and look for an alternative explanation why the participants were erroneously guided to disregard the available gestures for referent identification. In hindsight, it seems that the complete balancing of the three conditions is the reason for these unexpected effects. From the perspective of the participant, it became clear after a few trials that, in contrast to what can pragmatically be expected from a communicative situation (cf. Gricean maxims; Grice, 1989), the gestures were not a very useful cue for reference tracking because only half of them gave the correct information. In Experiment 2, we therefore changed the experimental design in a pragmatically more appropriate way and presented the participants solely with the baseline and same-referent conditions.

EXPERIMENT 2: WHEN ABSTRACT POINTING IS RELIABLE

Methods

Participants

Thirty-eight German native speakers participated in Experiment 2. None of them had participated in Experiment 1. The data of six participants had to be discarded, as they had too many trials contaminated by EEG artifacts (i.e., less than 20 valid experimental trials after artifact rejection). Thus, 32 right-handed participants (16 men; mean age = 25.1 years, SD = 2.3 years; laterality quotient = 94.4, SD = 8.5; Oldfield, 1971) entered the analyses. Their vision was normal or corrected to normal, and none had hearing or neurological impairment. They received 7 EUR per hour of participation.

Stimuli and Experimental Design

Stimulus materials were identical to Experiment 1. The different-referent condition, however, was dropped. Because of the modified design with only four conditions (baseline, left/right; same-referent, left/right), six randomly chosen topics were removed from the interview, resulting in 84 presented topics.

EEG Recordings and Analysis

The same EEG setup as in Experiment 1 was used. An automatic artifact rejection was carried out using a sliding window of 200 msec. Epochs were rejected in the case of a 40-μV deviation on the EOG channels or a 50-μV deviation on the EEG channels. Then, a manual rejection was carried out based on visual inspection. On average, 26.3% of the trials were rejected, and a mean of 30.8 (SD = 4.7) trials of the same-referent condition and 31.1 (SD = 5.4) trials of the baseline condition entered statistical analysis. Epoch and ROI definition were analogous to Experiment 1. On the basis of visual inspection, the N400 was analyzed between 200 and 350 msec, and the P600 was analyzed between 450 and 700 msec. Statistical analysis was analogous to Experiment 1. Note, however, that the variable Referent used in the ANOVA has now two levels (same, baseline).

Results

Memory Task

The participants answered 93.8% of all memory questions correctly (SD = 3.7) indicating that the participants focused on the stimuli.

ERPs

Figure 4 shows two differences between the same-referent and baseline conditions. In the difference wave, a relatively early negative deviation for the baseline condition can be observed around 200–350 msec with a more accentuated right parietal scalp distribution. Between 450 and 700 msec, a positive deviation for the baseline condition can be seen at more left central electrodes.

Figure 4. 

The top part provides a detail view of the ERP waves and the difference waves at selected electrodes. The bottom part depicts the topographical maps of the difference between the baseline and the abstract pointing (same-referent) condition for both time windows.

Figure 4. 

The top part provides a detail view of the ERP waves and the difference waves at selected electrodes. The bottom part depicts the topographical maps of the difference between the baseline and the abstract pointing (same-referent) condition for both time windows.

In the N400 time window, we found an interaction of Referent with Ant/Pos (F(1, 31) = 13.34, p = .001) and an interaction of Referent with L/R (F(4, 124) = 5.14, p = .01, ε = .48). A step-down analysis of the first interaction holding the variable Ant/Pos constant revealed that the difference between the baseline condition and the same-referent condition was significant at posterior sites (t(1, 31) = −2.74, p = .01), but not at anterior sites (t(1, 31) = −0.79, p = .44). A step-down analysis of the second interaction holding the variable L/R constant showed that the difference between the baseline and same-referent conditions had a right lateralization, as it was significant at inner and outer right sites (inner right: t(1, 31) = −2.09, p = .04; outer right: t(1, 31) = −2.23, p = .03).

For the P600 time window, we found a significant main effect for Referent (F(1, 31) = 5.98, p = .02) and a significant two-way interaction between Referent and L/R (F(4, 124) = 3.05, p = .05, ε = .55). A step-down analysis of the interaction holding the variable L/R constant showed that the difference between the baseline and same-referent conditions was significant at all five levels, but to varying degrees (outer left: t(31) = 2.52, p = .02; inner left: t(31) = 2.55, p = .02; center: t(31) = 2.52, p = .02; inner right: t(31) = 2.30, p = .03; outer right: t(31) = 2.02, p = .05).

Discussion

In contrast to Experiment 1 where no beneficial effects of abstract pointing could be found, Experiment 2 clearly shows such beneficial effects when a gesture was used in a valid way.

The negativity between 200 and 350 msec was identified as an N400-like effect4 indicating a more effortful retrieval of information from long-term memory (Kutas & Federmeier, 2011; Gouvea, Phillips, Kazanina, & Poeppel, 2010) in the baseline condition. Here, no information was available that could be used for active reference tracking making the lexical retrieval of the target word more effortful. In the gesture condition, however, the abstract pointing at the ambiguous element indicated consistently which of the two potential referents was meant, thereby preactivating (or priming) the target word that facilitated lexical retrieval.

In the baseline condition, the underspecified (i.e., unknown) referent is revealed by the target word. This information needs to be integrated into a higher-order representation of the discourse. This requires more updating of the MRC than the abstract pointing condition leading to a larger P600 (cf. Burkhardt, 2006).

Thus, compared with the baseline condition, the gesture condition showed a reduced N400–P600 pattern indicating easier lexical access of the intended referent (i.e., target word) and a minor modification of the MRC because a clear unambiguous referent was already primed/selected5 earlier on the basis of the consistent abstract pointing in the preceding dyad.

GENERAL DISCUSSION

To our knowledge, this is the first demonstration that consistent abstract pointing facilitates referent identification in the observation of naturalistic face-to-face communication. The data show that consistent abstract pointing leads to a reduced N400–P600 pattern during referent identification at the target word and indicate facilitated lexical access and only minimal pragmatic processing (i.e., updating of the mental representation of the communicative situation).

Intriguingly, whereas lexical access to the target word is not influenced by abstract pointing when gestures are unreliable, pragmatic processing load, as indexed by the P600, is still increased when abstract pointing is related to the unintended referent (cf. Experiment 1). We hypothesize that the cognitive system gathers this error information to monitor the cue validity of the gestures. On the basis of normal life interaction (i.e., Gricean principles of good communication; Grice, 1989) the cognitive system assumes (i.e., has a prior) that gestures can be helpful during communication. If not, this is relevant information that needs to be taken into account in the representation of the communicative situation (i.e., the MRC is updated, and a P600 is elicited). When, during the course of communication, enough evidence is accumulated suggesting that a gesture is not helpful, the cognitive system stops priming/selecting the referent information on the basis of abstract pointing and will only gather information related to the cue validity of the gestures6 until there is enough evidence that gesture is helpful again.7 When this normal state of affairs is present, gesture will prime/select the intended referent once more (see Figure 5). Thus, as soon as gestures are diagnosed as helpful, abstract pointing preactivates (or primes) the intended referent.

Figure 5. 

A graphical representation of how inconsistent abstract pointing impacts gesture usage. As long as valid abstract pointing (Valid AP) is conducted, the cue validity index (i.e., the inverse of the accumulation of error information) will be high, and gesture will be used for reference identification. Depending on the amount of invalid abstract pointing (Invalid AP), the cue validity index will be lowered. When the amount of invalid information drives the index below a particular threshold, gesture priming/referent selection will stop. (A) When gesture becomes valid again, the cue validity index will be heightened, and after some time, gesture priming/selection starts again. (B) When gesturing stops, for example, because the dyadic interaction stopped, the evidence to block priming/selection will decay with the rate given by the time constant. When the cue validity index is above threshold, gesture priming/selection can potentially start again. Note that it is very well possible that the monitoring of the validity/helpfulness of gestures is something specific for abstract pointing. This type of gesture has no inherent semantic content and is typically involved in more structural language processing.

Figure 5. 

A graphical representation of how inconsistent abstract pointing impacts gesture usage. As long as valid abstract pointing (Valid AP) is conducted, the cue validity index (i.e., the inverse of the accumulation of error information) will be high, and gesture will be used for reference identification. Depending on the amount of invalid abstract pointing (Invalid AP), the cue validity index will be lowered. When the amount of invalid information drives the index below a particular threshold, gesture priming/referent selection will stop. (A) When gesture becomes valid again, the cue validity index will be heightened, and after some time, gesture priming/selection starts again. (B) When gesturing stops, for example, because the dyadic interaction stopped, the evidence to block priming/selection will decay with the rate given by the time constant. When the cue validity index is above threshold, gesture priming/selection can potentially start again. Note that it is very well possible that the monitoring of the validity/helpfulness of gestures is something specific for abstract pointing. This type of gesture has no inherent semantic content and is typically involved in more structural language processing.

We suggest the following scenario how speech and abstract pointing interact (see Figure 5). As a default strategy, the cognitive system (or parser) assumes that gestures are communicative, and unless proven different, abstract pointing will be used as a tool for reference tracking in longer discourses where gesture space will get a representational value for a specific referent. During this multimodal communication, gesture input is constantly monitored and evaluated on a short-term local level. In situations where gesture is less clear or plain wrong, they will get less priority (see Holle & Gunter, 2007) or will be discarded completely (see present results). This may also indicate that the cognitive system constantly adapts the priority of gesture use, depending on, for instance, the stability of the personal gesture style of a particular interlocutor (see also Obermeier & Gunter, 2015) possibly by a different thresholding of the gesture validity cue. Please note that we have to make a distinction between gesture use for referencing, which is a process across a longer period impacting the discourse and the monitoring process as such which is a shallow local process that only compares gesture position and uttered speech content without impacting discourse processing at all.

In the present experiment, we found either an isolated P600 effect or a biphasic N400–P600 pattern. This seems unexpected because virtually all ERP studies on gesture processing have observed N400 effects, both in violation (Kelly, Healey, Özyürek, & Holler, 2012; Habets, Kita, Shao, Özyürek, & Hagoort, 2011; Ibanez et al., 2010, 2011; Wu & Coulson, 2005, 2007, 2010, 2011; Kelly, Creigh, & Bartolotti, 2010; Cornejo et al., 2009; Kelly, Ward, Creigh, & Bartolotti, 2007; Özyürek, Willems, Kita, & Hagoort, 2007; Sheehan, Namy, & Mills, 2007; Kelly, Kravitz, & Hopkins, 2004) as well as in disambiguation (Obermeier et al., 2012; Obermeier, Holle, & Gunter, 2011; Holle & Gunter, 2007) paradigms. In very rare cases, these studies also show a late positive component (Cornejo et al., 2009; Wu & Coulson, 2005), which was linked to either memory or task-driven processes. All of these N400 studies used so-called iconic gestures, which have a “close formal relationship to the semantic content of speech” (McNeill, 1992, p. 12), and although they contain some rudimentary amount of semantic information in themselves, they can only be clearly interpreted within a particular context (Hadar & Pinchas-Zamir, 2004; Krauss, Morrel-Samuels, & Colasante, 1991). Thus, iconic gesture and context are highly intertwined and influence each other mutually8 at a semantic level, which increases the lexical and semantic retrieval demands from long-term memory leading in the end to an N400 for the mismatching/ambiguous condition (Kutas & Federmeier, 2011). Because the vast majority of ERP studies on gestures reported an isolated N400, without a subsequent P600, it seems that there was no need for any update or revision processes in those studies and that gesture-related information only needed to be retrieved from memory. Thus, the basis of processing and integrating iconic gestures is directly related to the semantic content of the context. In contrast, referencing via pointing gestures (i.e., concrete deixis) has a different, more interactive, and social basis. Peeters and Özyürek (2016) suggest that traditionally referring to objects, people, and so forth had been described as an addressee-blind act purely driven by the speaker's belief about what their addressee knows. More recently, however, referencing is seen as a collaborative enterprise involving both the speaker and the addressee working together to an equal extend on how to conceptualize and name a particular entity (cf. Clark & Bangerter, 2004). In line with this, Peeters et al. (2015) found in a pointing study that explored proximal and distal demonstratives (i.e., “this” or “that”) evidence for a sociocentric approach to deixis, suggesting that interlocutors construe a shared space during their conversation, thereby emphasizing the social basis of deixis. The role of such social and interactive processes has even more prominence in the case of abstract pointing. Here, people employ gesture space to refer to particular discourse information, although nothing is actually present. Stec and Huiskes (2014) showed in natural dyadic narrative that indeed abstract pointing is used to identify particular referents and that this information is used by both interlocutors across longer stretches of discourse, thereby “…demonstrating a tight, on-line, synergetic coupling of both speaker and addressee—both in terms of the situation or mental models they have of the talk at hand, and of the multimodal manifestation of those models, made in physical space” (p. 56). Thus, the joined and structured use of gesture space clearly shows that it is of utmost importance that abstract pointing is used in a valid way. It seems that our data point toward a possible monitoring mechanism, as reflected in the P600, which deals with this issue. Our data suggest that, in multimodal communication, the visual channel is constantly monitored and evaluated for how helpful or effective it is, thereby revealing a mechanism, which prioritizes the relevance of cues in communication. Whether this is a universal mechanism that can be generalized to other visual communicative cues needs to be investigated in further experiments.

In conclusion, the current studies showed that abstract pointing facilitates referent identification during the observation of face-to-face communication and is therefore an important cue in human communication. The studies also revealed that the use of this communicative cue depends on its validity, which is monitored and evaluated in an online fashion throughout the communicative dyad. The suggested general prioritizing mechanism should pave the way toward a better understanding of how communicative cues impact interactive comprehension.

APPENDIX

The following list contains all topics that were presented during Experiment 1. For Experiment 2, six topics were removed.

  • 1. 

    homeopathy – conventional medicine

  • 2. 

    private health insurance – compulsory health insurance

  • 3. 

    tattoo – piercing

  • 4. 

    nicotine – cannabis

  • 5. 

    pipe – cigar

  • 6. 

    synthetic fill sleeping bag – down fill sleeping bag

  • 7. 

    gas stove – fuel stove

  • 8. 

    digital camera – film camera

  • 9. 

    photograph – video

  • 10. 

    TV set – book

  • 11. 

    studies – apprenticeship

  • 12. 

    Diplom (German university degree) – bachelor

  • 13. 

    university – university of applied sciences

  • 14. 

    studio apartment – flat share

  • 15. 

    Brockhaus (German encyclopedia) – Wikipedia

  • 16. 

    bicycle – car

  • 17. 

    train – plane

  • 18. 

    Transrapid (German maglev) – ICE (German conventional train)

  • 19. 

    Goethe – Shakespeare

  • 20. 

    Hesse – Kafka

  • 21. 

    Mickey – Donald

  • 22. 

    Asterix – Lucky Luke

  • 23. 

    Stern (German newsmagazine) – Spiegel (German newsmagazine)

  • 24. 

    FAZ (German newspaper) – Bild (German newspaper)

  • 25. 

    public broadcasting – commercial broadcasting

  • 26. 

    countryside – city

  • 27. 

    owner-occupied house – rented apartment

  • 28. 

    new building – old building

  • 29. 

    attic floor – first floor

  • 30. 

    jogging – swimming

  • 31. 

    open air pool – lake

  • 32. 

    ski – snowboard

  • 33. 

    Maske – Beckenbauer

  • 34. 

    barbecue – raclette

  • 35. 

    McDonald's – Subway

  • 36. 

    pizza – doner kebab

  • 37. 

    vegetarian – vegan

  • 38. 

    Rewe (German supermarket) – Aldi (German supermarket)

  • 39. 

    standard vegetables – organic vegetables

  • 40. 

    deposit bottle – Tetra Pak

  • 41. 

    Eastern Germany – Western Germany

  • 42. 

    Dresden – Leipzig

  • 43. 

    Thomaskirche (church in Leipzig) – Nikolaikirche (church in Leipzig)

  • 44. 

    Protestantism – Catholicism

  • 45. 

    Hinduism – Islam

  • 46. 

    family child care – day nursery

  • 47. 

    all-day school – half-day school

  • 48. 

    French – Latin

  • 49. 

    military service – civilian service

  • 50. 

    carpeted floor – parquet floor

  • 51. 

    spruce – oak

  • 52. 

    gas kitchen stove – electric kitchen stove

  • 53. 

    cell phone – landline phone

  • 54. 

    dog – cat

  • 55. 

    terrarium – aquarium

  • 56. 

    notebook – desktop

  • 57. 

    CRT – LCD

  • 58. 

    PC – Mac

  • 59. 

    Windows – Linux

  • 60. 

    Word – LaTeX

  • 61. 

    razor – wax

  • 62. 

    wet shaving – electric shaving

  • 63. 

    condom – birth-control pill

  • 64. 

    wine – beer

  • 65. 

    whisky – rum

  • 66. 

    vodka – tequila

  • 67. 

    coffee – tea

  • 68. 

    Italy – France

  • 69. 

    Norway – Spain

  • 70. 

    the mountains – the sea

  • 71. 

    package holiday – individually organized holiday

  • 72. 

    Suitcase – backpack

  • 73. 

    Schwarzenegger – Stallone

  • 74. 

    Hitchcock – Spielberg

  • 75. 

    theater – cinema

  • 76. 

    vinyl record – CD

  • 77. 

    Star Trek – Star Wars

  • 78. 

    Freud – Einstein

  • 79. 

    fountain pen – ball pen

  • 80. 

    digital watch – analogue watch

  • 81. 

    contact lenses – glasses

  • 82. 

    umbrella – rain jacket

  • 83. 

    wool sweater – fleece sweater

  • 84. 

    skirt – trousers

  • 85. 

    new car – used car

  • 86. 

    diesel fuel – gasoline

  • 87. 

    natural gas car – hybrid car

  • 88. 

    Mini – Volkswagen Beetle

  • 89. 

    station wagon – SUV

  • 90. 

    car – motorcycle

Interview Preparation

In a first step, 102 topics of dualistic nature were selected. They were sent to the actress to make her familiar with them. Subsequently, we determined the questions the interviewer was supposed to ask during the establishing phase. We also looked for arguments the interviewee could potentially use for her responses. Note that we did not prepare fully scripted answers for the establishing phase. Instead, we wanted to be able to give the actress information at hand in cases where she did not have the knowledge to answer a question.

In addition, the question–answer pair for the critical phase was prepared. There were two versions of each of the responses. In contrast to the establishing phase, we prepared fully scripted responses for the critical utterance, because it was our goal to have the same wording until the target word was uttered. In the end, only 90 topics were used during the experiment.

Equipment, Shooting, and Postproduction of the Stimulus Material

Three cameras were used for the shooting of the interview: one Sony HDR-HC5E and two Sony HDR-HC7E. All cameras used the recording format HDV 1080i50. The resulting videos had a resolution of 1440 × 1080, 25 frames per second, and an aspect ratio of 16:9 and were interlaced. The audio signal was recorded separately with a Roland CD-2 CF/CD Recorder and was saved as .wav files at 44.1 kHz/16 bit. A clapperboard was used for the synchronization of the video and audio materials during postproduction. Three different shots were used for videotaping (see Figure 2). Final Cut Pro 5.1.4 was used for video postproduction. For the different-referent versions, we simply switched the audio tracks of the same-referent versions; during this procedure, the gesture stroke was used as the reference point. Because of this procedure, the voice did not match the lip movements in the different-referent versions. To conceal this from the participants, a blur mask was put on the interviewee’s face during the complete interview. During the experiment, the mask was accepted as a means to achieve anonymity for the interviewee.

To get a coherent interview, topics with a similar subject were grouped into “meta-topics.” For instance, the topics “PC versus Mac” and “Linux versus Windows” were combined into the meta-topic “computers.” For the final interview, we used 90 topics that were distributed over 18 meta-topics, each containing two to seven single topics. All videos were compressed in the Audio Video Interleave (AVI) container format with Xvid as video codec and PCM (44.1 kHz/16 bit) as audio codec. The whole interview lasted 120 min, and the length of the meta-topics/single video files ranged from 4 to 11 min. Approximately 55 hr of raw video footage were used to create the stimulus material.

Acknowledgments

We thank Ina Koch and Cornelia Henschel for help during the data acquisition; Sven Gutekunst for technical assistance; Lars Meyer, Sonja Kotz, Spencer Kelly, Hyeon-Ae Jeon, Chiao-Yi Wu, Christian Obermeier, Trevor Penney, and two anonymous reviewers for helpful discussions/editing; and Angela D. Friederici for supporting our research.

Reprint requests should be sent to Thomas C. Gunter, Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, P.O. Box 500355, 04303 Leipzig, Germany, or via e-mail: gunter@cbs.mpg.de.

Notes

1 

Think of very helpful pointing gestures when buying a sandwich in a foreign country or in a very noisy environment.

2 

Note that reference-related usage of space is similarly done in sign language, suggesting an ecological usefulness of abstract pointing gestures (Emmorey & Falgier, 2004; Neidle, Kegl, MacLaughlin, Bahan, & Lee, 2000; for a recent critical discussion on pronouns and pointing in sign language, see Cormier, Schembri, & Woll, 2013).

3 

That is, a change to the other referent at a discourse level.

4 

Note that there are some studies (cf. van Berkum, Brown, & Hagoort, 1999) that have identified a specific component related to referent processing. This so-called Nref effect is broadly distributed with a distinct frontal focus starting around 200 msec after word onset. Clearly, the negativity elicited in the present experiment has a different scalp distribution and does not show any frontal focus.

5 

Note that we cannot decide whether abstract pointing selected, activated, or primed the associated referent. This depends on how one hypothesizes that an ambiguous referential situation is processed. One could, for example, assume that, as is the case with homonyms, both referents will be activated at the ambiguity (cf. Swinney, 1979) and that abstract pointing functions like a context, thereby inhibiting the inappropriate referent. Our experimental design is not able to shed light on this issue.

6 

In ERP terms, this means that no N400 effects will be seen for any of the three conditions in Experiment 1 because gesture is not actively used for referencing in the discourse. Although not used for active reference tracking, we suggest that the gestures are still being monitored on a local level because of their communicative importance. Therefore, the different-referent condition will show a P600 because, here, gesture and speech are invalid on a local (here the sentence) level.

7 

From this perspective, one should predict that, in the first few trials of Experiment 1, gestures will prime the referent pointed at, and the MRC is updated every time an incongruent gesture occurs. Unfortunately, we cannot explore this issue for reasons related to the signal-to-noise ratio.

8 

The iconic gesture gets its meaning via the context, and an ambiguous target word gets its meaning via the gesture.

REFERENCES

REFERENCES
Alamillo
,
A. R.
,
Colletta
,
J. M.
, &
Kunene
,
R. N.
(
2010
).
Reference-tracking in gesture and speech: A developmental study on French narratives
.
Rivista di Psicolinguista Applicata
,
10
,
75
96
.
Brouwer
,
H.
,
Fitz
,
H.
, &
Hoeks
,
J.
(
2012
).
Getting real about semantic illusions: Rethinking the functional role of the P600 in language comprehension
.
Brain Research
,
1446
,
127
143
.
Burkhardt
,
P.
(
2006
).
Inferential bridging relations reveal distinct neural mechanisms: Evidence from event-related potentials
.
Brain and Language
,
98
,
159
168
.
Clark
,
H. H.
, &
Bangerter
,
A.
(
2004
).
Changing ideas about reference
. In
I. A.
Noveck
, &
D.
Sperber
(Eds.),
Experimental pragmatics
(pp.
25
50
).
New York
:
Palgrave Macmillan
.
Cormier
,
K.
,
Schembri
,
A.
, &
Woll
,
B.
(
2013
).
Pronouns and pointing in sign languages
.
Lingua
,
137
,
230
247
.
Cornejo
,
C.
,
Simonetti
,
F.
,
Ibanez
,
A.
,
Aldunate
,
N.
,
Ceric
,
F.
,
Lopez
,
V.
, et al
(
2009
).
Gesture and metaphor comprehension: Electrophysiological evidence of cross-modal coordination by audiovisual stimulation
.
Brain and Cognition
,
70
,
42
52
.
Emmorey
,
K.
, &
Falgier
,
B.
(
2004
).
Conceptual locations and pronominal reference in American sign language
.
Journal of Psycholinguistic Research
,
33
,
321
331
.
Goodrich Smith
,
W.
, &
Hudson Kam
,
C. K.
(
2012
).
Knowing ‘who she is’ based on ‘where she is’: The effect of co-speech gesture on pronoun comprehension
.
Language and Cognition
,
4-2
,
75
98
.
Gouvea
,
A. C.
,
Phillips
,
C.
,
Kazanina
,
N.
, &
Poeppel
,
D.
(
2010
).
The linguistic processes underlying the P600
.
Language and Cognitive Processes
,
25
,
149
188
.
Greenhouse
,
S. W.
, &
Geisser
,
S.
(
1959
).
On methods in the analysis of profile data
.
Psychometrika
,
24
,
95
112
.
Grice
,
H. P.
(
1989
).
Studies in the way of words
.
London
:
Harvard University Press
.
Gunter
,
T. C.
, &
Bach
,
P.
(
2004
).
Communicating hands: ERPs elicited by meaningful symbolic hand postures
.
Neuroscience Letters
,
372
,
52
56
.
Gunter
,
T. C.
,
Weinbrenner
,
J. E. D.
, &
Holle
,
H.
(
2015
).
Inconsistent use of gesture space during abstract pointing impairs language comprehension
.
Frontiers in Psychology
,
6
,
80
.
Habets
,
B.
,
Kita
,
S.
,
Shao
,
Z.
,
Özyürek
,
A.
, &
Hagoort
,
P.
(
2011
).
The role of synchrony and ambiguity in speech–gesture integration during comprehension
.
Journal of Cognitive Neuroscience
,
23
,
1845
1854
.
Hadar
,
U.
, &
Pinchas-Zamir
,
L.
(
2004
).
The semantic specificity of gesture: Implications for gesture classification and function
.
Journal of Language and Social Psychology
,
23
,
204
214
.
Healy
,
C.
(
2012
).
Pointing to show agreement
.
Semiotica
,
192
,
175
196
.
Hoffmann
,
S.
, &
Falkenstein
,
M.
(
2008
).
The correction of eye blink artefacts in the EEG: A comparison of two prominent methods
.
PLoS One
,
3
,
e3004
.
Holle
,
H.
, &
Gunter
,
T. C.
(
2007
).
The role of iconic gestures in speech disambiguation: ERP evidence
.
Journal of Cognitive Neuroscience
,
19
,
1175
1192
.
Hühn
,
P.
,
Pier
,
J.
,
Schmid
,
W.
, &
Schönert
,
J.
(
2009
).
Handbook of narratology
.
Berlin
:
Walter de Gruyter
.
Ibanez
,
A.
,
Manes
,
F.
,
Escobar
,
J.
,
Trujillo
,
N.
,
Andreucci
,
P.
, &
Hurtado
,
E.
(
2010
).
Gesture influences the processing of figurative language in non-native speakers: ERP evidence
.
Neuroscience Letters
,
471
,
48
52
.
Ibanez
,
A.
,
Toro
,
P.
,
Cornejo
,
C.
,
Hurquina
,
H.
,
Manes
,
F.
,
Weisbrod
,
M.
et al
, (
2011
).
High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design
.
Psychiatry Research
,
191
,
68
75
.
Kelly
,
S. D.
,
Creigh
,
P.
, &
Bartolotti
,
J.
(
2010
).
Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing
.
Journal of Cognitive Neuroscience
,
22
,
683
694
.
Kelly
,
S. D.
,
Healey
,
M.
,
Özyürek
,
A.
, &
Holler
,
J.
(
2012
):
The communicative influence of gesture and action during speech comprehension: Gestures have the upper hand
.
Journal of the Acoustical Society of America
,
131
,
3311
.
Kelly
,
S. D.
,
Kravitz
,
C.
, &
Hopkins
,
M.
(
2004
).
Neural correlates of bimodal speech and gesture comprehension
.
Brain and Language
,
89
,
253
260
.
Kelly
,
S. D.
,
Ward
,
S.
,
Creigh
,
P.
, &
Bartolotti
,
J.
(
2007
).
An intentional stance modulates the integration of gesture and speech during comprehension
.
Brain and Language
,
101
,
222
233
.
Kim
,
A.
, &
Osterhout
,
L.
(
2005
).
The independence of combinatory semantic processing: Evidence from event-related potentials
.
Journal of Memory and Language
,
52
,
205
225
.
Kita
,
S.
(
2003
).
Pointing: Where language, culture, and cognition meet
.
Mahwah, NJ
:
Lawrence Erlbaum
.
Krauss
,
R. M.
,
Morrel-Samuels
,
P.
, &
Colasante
,
C.
(
1991
).
Do conversational hand gestures communicate?
Journal of Personality and Social Psychology
,
61
,
743
754
.
Kuperberg
,
G. R.
(
2007
).
Neural mechanisms of language comprehension: Challenges to syntax
.
Brain Research
,
1146
,
23
49
.
Kutas
,
M.
, &
Federmeier
,
K. D.
(
2011
).
Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP)
.
Annual Review of Psychology
,
62
,
621
647
.
Marstaller
,
L.
, &
Burianová
,
H.
(
2013
).
Individual differences in the gesture effect on working memory
.
Psychonomic Bulletin & Review
,
20
,
496
500
.
McNeill
,
D.
(
1992
).
Hand and mind: What gestures reveal about thought
.
Chicago
:
The University of Chicago Press
.
McNeill
,
D.
(
2003
).
Pointing and morality in Chicago
. In
S. Kita (Ed.)
,
Pointing: Where language, culture, and cognition meet
(pp.
293
306
).
Mahwah, NJ
:
Lawrence Erlbaum Associates, Publishers
.
McNeill
,
D.
,
Cassell
,
J.
, &
Levy
,
E. T.
(
1993
).
Abstract deixis
.
Semiotica
,
95
,
5
19
.
McNeill
,
D.
, &
Levy
,
E. T.
(
1993
).
Cohesion and gesture
.
Discourse Processes
,
16
,
363
386
.
McNeill
,
D.
,
Levy
,
E. T.
, &
Duncan
,
S. D.
(
2015
).
Gesture in discourse
. In
D.
Tannen
,
H. E.
Hamilton
, &
D.
Schiffrin
(Eds.),
The handbook of discourse analysis
(2nd ed., pp.
262
289
).
West Sussex, England
:
John Wiley & Sons, Inc
.
Nakamura
,
A.
,
Maess
,
B.
,
Knösche
,
T. R.
,
Gunter
,
T. C.
,
Bach
,
P.
, &
Friederici
,
A. D.
(
2004
).
Cooperation of different neuronal systems during hand sign recognition
.
Neuroimage
,
23
,
25
34
.
Neidle
,
C.
,
Kegl
,
J.
,
MacLaughlin
,
D.
,
Bahan
,
B.
, &
Lee
,
R.
(
2000
).
The syntax of American sign language
.
Cambridge, MA
:
MIT Press
.
Obermeier
,
C.
,
Dolk
,
T.
, &
Gunter
,
T. C.
(
2012
).
The benefit of gestures during communication: Evidence from hearing and hearing impaired persons
.
Cortex
,
48
,
857
870
.
Obermeier
,
C.
, &
Gunter
,
T. C.
(
2015
).
Multisensory integration: The case of a time window of gesture–speech integration
.
Journal of Cognitive Neuroscience
,
27
,
292
307
.
Obermeier
,
C.
,
Holle
,
H.
, &
Gunter
,
T. C.
(
2011
).
What iconic gesture fragments reveal about gesture–speech integration: When synchrony is lost, memory can help
.
Journal of Cognitive Neuroscience
,
23
,
1648
1663
.
Oldfield
,
R. C.
(
1971
).
The assessment and analysis of handedness: The Edinburgh inventory
.
Neuropsychologia
,
9
,
97
113
.
Osterhout
,
L.
, &
Holcomb
,
P. J.
(
1992
).
Event-related brain potentials elicited by syntactic anomaly
.
Journal of Memory and Language
,
31
,
785
806
.
Özyürek
,
A.
,
Willems
,
R. M.
,
Kita
,
S.
, &
Hagoort
,
P.
(
2007
).
On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials
.
Journal of Cognitive Neuroscience
,
19
,
605
616
.
Peeters
,
D.
,
Chu
,
M.
,
Holler
,
J.
,
Hagoort
,
P.
, &
Özyürek
,
A.
(
2015
).
Electrophysiological and kinematic correlates of communicative intent in the planning and production of pointing gestures and speech
.
Journal of Cognitive Neuroscience
,
27
,
2352
2368
.
Peeters
,
D.
, &
Özyürek
,
A.
(
2016
).
This and that revisited: A social and multimodal approach to spatial demonstratives
.
Frontiers in Psychology
,
7
,
222
.
Pouw
,
W. T. J. L.
,
de Nooijer
,
J. A.
,
van Gog
,
T.
,
Zwaan
,
R. A.
, &
Paas
,
F.
(
2014
).
Toward a more embedded/extended perspective on the cognitive function of gestures
.
Frontiers in Psychology
,
5
,
359
.
Sheehan
,
E. A.
,
Namy
,
L. L.
, &
Mills
,
D. L.
(
2007
).
Developmental changes in neural activity to familiar words and gestures
.
Brain and Language
,
101
,
246
259
.
So
,
W. C.
,
Kita
,
S.
, &
Goldin-Meadow
,
S.
(
2009
).
Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand
.
Cognitive Science
,
33
,
115
125
.
Stec
,
K.
, &
Huiskes
,
M.
(
2014
).
Co-constructing referential space in multimodal narratives
.
Cognitive Semiotics
,
7
,
35
59
.
Swinney
,
D. A.
(
1979
).
Lexical access during sentence comprehension: (Re)consideration of context effects
.
Journal of Verbal Learning and Verbal Behavior
,
18
,
645
659
.
van Berkum
,
J. J. A.
,
Brown
,
C. M.
, &
Hagoort
,
P.
(
1999
).
Early referential context effects in sentence processing: Evidence from event-related brain potentials
.
Journal of Memory and Language
,
41
,
147
182
.
van de Meerendonk
,
N.
,
Kolk
,
H. H. J.
,
Chwilla
,
D. J.
, &
Vissers
,
C. T.
(
2009
).
Monitoring in language perception
.
Language and Linguistics Compass
,
3
,
1211
1224
.
van Herten
,
M.
,
Chwilla
,
D. J.
, &
Kolk
,
H. H. J.
(
2006
).
When heuristics clash with parsing routines: ERP evidence for conflict monitoring in sentence perception
.
Journal of Cognitive Neuroscience
,
18
,
1181
1197
.
Wu
,
Y. C.
, &
Coulson
,
S.
(
2005
).
Meaningful gestures: Electrophysiological indices of iconic gesture comprehension
.
Psychophysiology
,
42
,
654
667
.
Wu
,
Y. C.
, &
Coulson
,
S.
(
2007
).
How iconic gestures enhance communication: An ERP study
.
Brain and Language
,
101
,
234
245
.
Wu
,
Y. C.
, &
Coulson
,
S.
(
2010
).
Gestures modulate speech processing early in utterances
.
NeuroReport
,
21
,
522
526
.
Wu
,
Y. C.
, &
Coulson
,
S.
(
2011
).
Are depictive gestures like pictures? Commonalities and differences in semantic processing
.
Brain and Language
,
119
,
184
195
.