Psychophysiological Markers of Performance and Learning during Simulated Marksmanship in Immersive Virtual Reality

Humans are adept at interacting skillfully in dynamically changing environments. We are able to maintain internal and external representations of our bodies, extrapolate motion of objects, and engage physically with these objects to achieve complex and precise results. These skills are largely because of the remarkable visual–motor integration enabled by our brains, eyes, and bodies.

Studies addressing visual–motor integration have led to a wealth of knowledge about the mechanisms that translate goals and states into motor commands (Elliott, Hayes, & Bennett, 2012; Wolpert, Ghahramani, & Jordan, 1995), motor learning (Wolpert & Flanagan, 2016; Schmidt & Lee, 2011), and the neural characteristics of expertise in highly practiced individuals (Yarrow, Brown, & Krakauer, 2009). Visually guided movements have been studied in a range of contexts to characterize the influence of visual information on movement control, demonstrating a balance between preprogrammed ballistic movements and visual-feedback-mediated refinements (Urbin, Stodden, Fischman, & Weimar, 2011; Elliott et al., 2010; Desmurget & Grafton, 2000; Meyer, Keith-Smith, Kornblum, Abrams, & Wright, 1990), as classically characterized by Fitts' Law (Fitts, 1954).

Scalp-recorded EEG has been particularly instrumental in understanding the cortical activity involved in visual–motor integration. This high-temporal resolution approach has allowed for the identification of processes essential for visual scene (Appelbaum, Ales, & Norcia, 2012; Anokhin et al., 2006) and motion perception (Jensen, Kaiser, & Lachaux, 2007; Kuba, Kubová, Kremlácek, & Langrová, 2007), such as is characterized by contralateralized visual-evoked potentials (VEPs) within the first ∼200 msec after stimulus presentation. Moreover, EEG has been widely used to understand attention (Appelbaum & Norcia, 2009; Jensen et al., 2007) and memory (Robitaille et al., 2010; Klimesch, 1999) processes necessary for goal-oriented behaviors, as well as error-related responses that have been shown to reflect learning through reweighting of sensory information (Cohen, Elger, & Ranganath, 2007; Hajcak, Moser, Holroyd, & Simons, 2006). These studies have led to a broad understanding of the chronometry of processes within the visual–motor cascade, including evidence that higher attentional states, indexed by less default-mode processing and lower alpha power, are associated with better processing of visual stimuli (e.g., Macdonald, Mathan, & Yeung, 2011). These findings and the ease-of-use of EEG have led to a growing number of studies evaluating expertise in activities such as marksmanship (Berka, Behneman, Kintz, Johnson, & Raphael, 2010; Janelle & Hatfield, 2008; Hatfield, Haufler, Hung, & Spalding, 2004; Hillman, Apparies, Janelle, & Hatfield, 2000), golf putting (Arns, Kleinnijenhuis, Fallahpour, & Breteler, 2008; Babiloni et al., 2008), table tennis (Hülsdünker, Ostermann, & Mierau, 2019), badminton (Hülsdünker, Strüder, & Mierau, 2017), and archery (Seo et al., 2012; Landers, Han, Salazar, & Petruzzello, 1994). Moreover, because the EEG power spectra and ERPs associated with sensory processing, decision making, and error recognition in these tasks correlate with trial-by-trial response speed (Hülsdünker et al., 2019) and different levels of accomplishment (Hatfield et al., 2004), they may provide potentially useful biomarkers in closed-loop neurofeedback and neurostimulation approaches (Gruzelier, 2014; Paulus et al., 2009).

The prospect of investigating brain dynamics of actively behaving participants in complex 3-D environments has been challenging. In most natural environments, conditions cannot be controlled and are difficult to replicate. In addition, studies using noninvasive modalities for recording brain activity, such as EEG, usually aim to minimize artifact-producing muscle activity by limiting mobility. Despite this, real-world tasks involve dynamic movements that must be achieved to fully engage perception, planning, and motor control for executing complex objectives. Studies that limit mobility may not capture all the cognitive and neural processes involved in performing natural, full-body movements, thereby creating a need for more advanced platforms for naturalistic brain and body monitoring.

Recent improvements in simulation technology, motion tracking, and mobile EEG have contributed to the development of mobile brain/body imaging (MoBI), an approach that investigates the links between distributed brain dynamics and natural behavior (Gramann et al., 2011; Makeig, Gramann, Jung, Sejnowski, & Poizner, 2009). Integration of simulation technologies, such as immersive and interactive virtual reality (VR) with brain and behavioral measurements, allow for complex tasks to be performed in controlled, indoor 3-D environments with high ecological validity. In particular, previous studies have used the MoBI approach to investigate physical interactions with dynamically moving objects (Jungnickel & Gramann, 2016), cognitive control during locomotion (Wagner, Makeig, Gola, Neuper, & Müller-Putz, 2016), and physical exertion during high-intensity cycling (Enders et al., 2016). Moreover, skill training in VR translates successfully to many real-world tasks that require procedural (Ragan, 2010), motor (dos Santos Mendes et al., 2012), and athletic (Gray, 2017; Miles, Pop, Watt, Lawrence, & John, 2012; Fink, Foo, & Warren, 2009; Aggarwal et al., 2006) learning, and can be used to simulate critical elements of stress and pressure providing an avenue for widespread application (Sanz, Multon, & Lécuyer, 2015; Stinson & Bowman, 2014).

In the current study, we combined these elements to investigate the neural and behavioral mechanisms underlying visual–motor skill learning during a three-visit simulated marksmanship training protocol based on the rules, physics, and timing of Olympic Trap Shooting. In this coincidence-anticipation task, individuals shot a simulated flying clay pigeon with a firearm game controller while kinematic movement tracking and 13-channel EEG were recorded within a six-sided immersive and interactive VR environment. Building upon our previous findings demonstrating kinematic changes in the ballistic and refinement stages of movement on this task (Rao et al., 2018), the goals of the current study were to characterize the cascade of kinematic and electrophysiological activity that contribute to the successful shooting of targets. In particular, the primary aim of this study was to test if EEG biomarkers early in the neural processing cascade correlated with ensuing behavioral outcomes, including RTs and eventual shot precision. As such, this study sought to establish biomarkers of performance that unfold over time by leveraging the high temporal resolution of EEG and the characteristic sequence of actions involved in the trap shooting event. Therefore, based on the temporal dependence of this trap shooting task, we sought to establish brain states early in a trial (either preparatory prior to the launch of the pigeon or early in the evoked cascade of sensory responses) that are able to reliably capture variance in measured kinematics and EEG occurring later in the trial. Secondary goals were to characterize changes in the latency and amplitude of kinematic and electrophysiological responses that occur as participants practice this task and to determine if error-related responses at the end of a trial contribute to performance in the next trial, as would be expected from reinforcement learning theories (Walsh & Anderson, 2012).

Based on the timing and movement constructs of this trap shooting simulation and the literature describing neuro-kinematic processes involved in visual–motor orienting, several hypotheses were formulated and tested. A central expectation was that VEPs would be present contralateral to the launch direction of the target and that larger amplitudes for this potential would correlate with better shooting performance. A second expectation in this study was that EEG spectral power over para-Rolandic, motor control areas during the preparatory interval prior to the launch of each target would correlate with movement outcomes of the trial. Finally, although highly exploratory, it was hypothesized that the occurrence of misses, versus hits, on each trial would elicit characteristic feedback-related negativity (FRN; Holroyd & Coles, 2002). Such findings would add to understanding of motor system physiology and aid in the development of a naturalistic neuroscience approaches that can be used for evaluation and training in applied contexts.

Twenty-four participants (mean age = 25 ± 4.28 years, 13 women) were recruited to take part in this study. Four participants were excluded from subsequent analyses for the following reasons: Two participants were left-handed, creating discrepancies in the lateralized EEG analyses; one participant had extensive marksmanship experience; and one participant did not complete the entire protocol. The remaining 20 participants constituted a homogenous set of right-handed novice marksmen, with normal or corrected-to-normal vision, no history of neurological deficits or family history of photosensitive epileptic seizures, and no previous experience with the protocol presented in this study. Participation was voluntary, and participants were compensated $10/hr for their involvement. The experimental protocol was approved by the Duke University institutional review board.

The study was conducted in the Duke immersive Virtual Environment (DiVE; Figure 1A), a six-sided cave automatic virtual environment (CAVE) VR system (Cruz-Neira, Sandin, & DeFanti, 1993), where participants stood in the center of a room-size 3 × 3 × 3 m cube with projectors directed at each of the cube's six walls. Projectors were run at 120 Hz with a total resolution of 1920 × 1920 pixels per wall. Participants wore 3-D shutter glasses operating at 60 Hz to view stereoscopic graphics. A head tracking device was mounted on the glasses to control the system viewpoint according to the participant's head movements. An Intersense IS-900 tracking system was used to record the 3-D position and orientation of the controller and head throughout the experiment. Data from both the controller and head tracking sensors were sampled at 60 Hz.

An Xbox Top Shot Elite firearm game controller was used for target shooting. The controller, furnished with a six degrees-of-freedom tracking sensor, was held in the participant's right hand and stabilized with the opposite hand placed along the barrel of the controller, as shown in Figure 1B. Target acquisition was performed using a ray casting technique, but rather than a visible ray, only a white dot was shown at the target depth to mimic a laser sight.

EEG signals were recorded using a BrainVision actiCAP system with 13 active gel-based electrode channels, which were placed on the scalp according to the 10–20 system at locations F3, Fz, F4, C3, Cz, C4, T5, T6, P3, Pz, P4, O1, and O2, with mastoid references. Two additional electrodes were placed on the right and left outer canthi of the eyes to record horizontal eye movements via HEOG. All electrode impedances were kept below 10 kΩ to record high-quality signals that were sampled at 1000 Hz. The dominant environmental artifact at 60 Hz (power line frequency) was attenuated using a 0.1- to 30-Hz bandpass filter.

The task presented inside the DiVE was modeled after the Olympic Trap Shooting event (International Shooting Sport Federation, 2013), where the goal for the participant was to shoot a digital clay pigeon that was launched from behind a trap house. The design of the simulation abided by the laws of physics for projectile motion (including gravitational pull, air resistance, and lift force) to mimic realistic projectile flight.

To begin the task, participants aimed the controller toward a digital trap house, which was displayed as a rectangle above the ground, 54 ft in front of them in simulated space. After an initial 500-msec waiting period, the trap house changed color from red to green and a second waiting period began that was variable between 0.5 and 1.0 sec. During the initial waiting period, if the participant aimed the controller away from the trap house before the color changed from red to green, the timer was reset and did not begin until the participant aimed the controller back toward the trap house. At the end of the second waiting period, a digital target (displayed as an orange sphere of radius 1 ft) was launched in one of six directions with the initial speed of 95.34 ft/sec.

Participants were given one chance to shoot the target. The longer a participant took to pull the trigger, the further the target moved away from them, making it more difficult to hit because of the change in depth and corresponding reduction in visual angle. Once the trigger was pulled, the screen immediately froze to display feedback that included the participant's shot location relative to the location of the target, their cumulative shot success (percent hits), and the number of trials remaining, as shown in Figure 1C. If the controller's ray was in contact with the target at the time of the shot, the target changed color from orange to green, indicating a hit. If not, the color changed from orange to red to indicate a miss. The feedback remained on the screen until the controller was returned to a position over the trap house, which started the next trial sequence.

The six possible target trajectories, illustrated in Figure 1D, consisted of three horizontal directions relative to the center of the trap house (left = −45°, center = 0°, right = 45°) and two elevations relative to the ground plane (upper = 25.17°, lower = 12.95°). To increase ecological validity (e.g., fluctuations in outdoor environmental conditions such as wind currents), random horizontal noise ranging from −3° to 3° was added to the launch angle. The maximum flight times for the target were 1.772 sec and 3.085 sec for the lower and upper trajectory elevations, respectively.

Participants practiced the simulated marksmanship task on three separate visits within 1 week. During each visit, which lasted approximately 2 hr, the practice session was split into eight blocks of 60 trials. Within each block, all six target trajectories were presented 10 times in a random order. During the first visit, participants were given two practice rounds to become acquainted with the task (approximately 7 min total). The first practice round consisted of trials where the target was launched only in the upper right direction, with the target at a larger size than in the real simulation. The second practice round consisted of six trials where a normal-sized target was launched in each possible direction. The second practice round was administered twice if the participants' hit rate was below 50%. After completing the practice rounds, participants began the eight blocks of training and had the option to take a break between Blocks 2, 4, and 6 to prevent fatigue.

The shooting task presented in this study is particularly useful for investigating psychophysiological skill learning because it produces discrete measures of performance (participants either hit or miss the target) while still requiring complex visual–motor coordination. In order to determine the types of skill learning that occurred through practice, dependent variables were extracted from the kinematic motion tracking and EEG data to analyze changes that occurred over time and to study the relationships between motor proficiency and neural activity. Figure 1E illustrates the sequence of event actions within a single trial from which the EEG and kinematic variables were extracted. Gray shaded regions indicate approximate intervals and biomarkers during which kinematic (blue) and EEG (orange) responses were calculated. These psychophysiological measures are discussed individually below.

Shot success and shot error were used as indicators to determine performance improvements through practice. Shot success was calculated as the percentage of target hits out of the total number of shots taken. Shot error was measured as the Euclidean distance (in millimeters) between the location of the shot and the center of the target at the time of the trigger pull and therefore reflects the precision of the shot attempt. High shot success rates and lower shot error (defined by shorter distances to the target) signify superior performance that is reproducible and consistent.

The timings of key behavioral events, such as the onset of movement, were extracted from the motion tracking and HEOG data. RT—the elapsed time from the target launch to the onset of movement—was calculated for the movement of the eyes, controller, and head. Controller RT and head RT were calculated using 10% of the peak acceleration trace, as measured with the Intersense motion trackers (van Donkelaar & Franks, 1991). Acceleration was computed as the derivative of the speed trace, after the speed trace was smoothed with a seventh order finite impulse response filter. An example acceleration trace collected for a single trial is shown in Figure 2A. Oculomotor RT was calculated using a rectified sum of the corneoretinal potential measured from the two HEOG channels for the left and right horizontal target trajectories, as shown in Figure 2B. The corneoretinal potential occurred when the positive part of the eye (the cornea) rotated closer to either the left or right HEOG electrode, resulting in an upward voltage deflection for the closer electrode and a downward deflection for the opposite electrode. The final oculomotor RT was recorded when a threshold of 3 SDs above a baseline mean (100 msec postlaunch) was exceeded. Trigger response time was calculated as the elapsed time between the target launch and when the trigger was pulled to shoot. Trials were excluded from the analyses if the participant did not pull the trigger to shoot (1.67%, 482 trials) or the movement was initiated too quickly for a given trial, defined as a controller RT of less than 16.667 msec, or one rendered video frame (0.39%, 23 trials).

EEG data were divided into three separate segments of the task for analysis: (1) the prelaunch preparatory EEG phase, which is thought to represent the participant's mental state prior to target launch; (2) the postlaunch VEP that reflects visual-cortical processing of the target prior to kinematic movement; and (3) the postshot FRN that reflects the participants' response to hits versus misses on each trial.

Preparatory EEG signals were extracted from a 500-msec window in the EEG data after the trap house turned green. Baseline drifts were corrected by subtracting the mean value of the signal in the preceding 50-msec window. The preparatory phase was characterized by spectral power. The baseline-corrected EEG signals were transformed to compute the power spectral density estimates using Welch's method (window size = 250 msec, overlap = 50%) and divided into the standard EEG frequency bands for analysis: delta (0.1–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), and beta (13–30 Hz). Trials containing excess muscle artifact were rejected by removing voltages that exceeded a threshold of 200 μV (0.97%, 279 out of 28,800 trials).

VEPs were extracted from the 200-msec window immediately following a target launch in the left or right directions to assess the contralateral/ipsilateral visual response in the brain. Baseline correction was performed by subtracting the average voltage in the 50-msec prelaunch window. Electrodes P3 and P4 were selected for analysis because of their posterior locations over the left and right hemispheres of the visual cortex, respectively, and because of their higher signal quality as compared to Channels O1 and O2, which tended to include more noise because of cap fitting issues and neck muscle artifacts produced during movement. Single-trial VEP amplitudes were calculated using the average over a 20-msec window centered on the participant's peak VEP latency for that visit. Trials were excluded from the averages if they exceeded a threshold of ± 35 μV (3.02%, 580 out of 19,200 trials) or contained data outside of 5 SDs of the joint probability distribution observed at each time point (0.35%, 67 out of 19,200 trials). There were no systematic differences in rejection rates across the different trajectory directions. The use of EEG artifact removal techniques (as opposed to rejection) would have been challenging because of the lack of clear biological templates (e.g., ocular artifact correction) to base exclusion on. Moreover, based on the low prevalence of rejected trials (3.37% total), it was evident that the signal under consideration offered an unimpeded view of the neural activity that was meant to be scrutinized in the planned hypothesis tests.

The FRN was calculated by extracting data epochs spanning from 200 msec before the trigger pull to 800 msec after the trigger pull at channel Cz. These “shot-locked potentials” were baseline corrected by subtracting the average of a 100-msec preshot window. Individual trial data were separately averaged for all successful shots (hits) and for all unsuccessful shots (misses). Preliminary analyses determined nonsignificant differences over the shot elevations (high vs. low) and shot directions (left vs. center vs. right). Therefore, the FRN was calculated using the average over all trajectories, separately on each visit of the protocol for all 20 participants. Single-trial FRN amplitudes were calculated using the average of the 40 msec surrounding the peak latency for that visit, whereas single trial onset latencies were calculated as the latency at 1/3 peak amplitude.

The purpose of this study was twofold: (1) to test if EEG biomarkers early in the cascade of processes induced during the coincidence anticipation task correlated with ensuing behavioral outcomes, including shooting movement RTs and eventual shot precision, and (2) to analyze skill learning as a function of practice by exploring changes in both EEG and behavior over time to determine if error-related responses at the end of a trial contributed to performance in the next trial.

All the preprocessing of EEG recordings was performed in MATLAB (The MathWorks) using the EEGLab toolbox (Delorme & Makeig, 2004). To investigate relationships between EEG variables and motor behavior variables in a data set showing substantial clustering structure (i.e., intraclass coefficient estimates are in [0.36, 0.92]), linear mixed-effects models (LMEMs) were fit and tested using the lme4 and lmerTest package in R Version 3.5.3 (R Core Team, 2019). LMEMs were fit to EEG and behavioral data that included trial-level observations (task block and visit [as a statistical nuisance factors to offset variable discontinuities], EEG measures [preparatory delta/theta/alpha/beta band power and VEP amplitude], launch direction), nested within individual participants. In different models, controller RT or shot error served as outcome variables while allowing for the intercept parameter to vary among participants. The LMEMs involving controller RT or shot error also included an exponential function term of trial number (i.e., e^{-Trial Number}), whose slope parameter was treated as a random effect and allowed to vary among participants. All the LMEMs reached normal convergence using restricted maximum likelihood estimator. In the LMEMs, the statistical significance of the parameter estimates was tested using t distribution with degree of freedom adjusted using Satterthwaite's method (see Kuznetsova, Brockhoff, & Christensen, 2017).

Repeated-measures analysis of variance (rANOVA) was performed in SPSS v25 to test skill learning across time on both EEG and behavioral variables. A given rANOVA involved either the main effect of Visit (i.e., Visit 1–3) or the main effects of Block (i.e., Block 1–8) and interactions from Visit and Block, depending on the specific question being evaluated. Mauchly's method was used to examine the assumption of sphericity in the data. In case(s) of assumption violation, model outcomes based on the Greenhouse–Geisser correction were adopted.

A final analysis was performed on the FRN to test its role in reinforcement learning, where epochs were resegmented to create the sequential categories: hits following hits, hits following misses, misses following hits, and misses following misses. Single-trial mean amplitudes within the 40 msec surrounding the peak FRN for each of the four segment classes were submitted to rANOVA to evaluate main effects and interactions between the Shot Success (i.e., hit or miss) of successive trial pairs. Effects were considered statistically significant if multiple-comparison adjusted p values were less than .05.

The data set in this study consists of metrics characterizing task performance, movement kinematics, and electrophysiology that change over the course of learning and illustrate relationships between the brain and body. In the following sections, results from these three classes of metrics are presented to infer the detailed chronometry and influence of psychophysiological activity associated with performance on the marksmanship task.

Shot success rate—the percentage of hits out of total shots taken—was calculated for each block and visit for all 20 participants. The average results are shown in Figure 3A. As expected, participants significantly improved their success rate through practice, with rANOVA demonstrating main effects of Visit, F(2, 38) = 68.50, p < .001, η² = .78, and Block on the first visit, F(7, 133) = 22.78, p < .001, η² = .55, and the second visit, F(7, 133) = 10.04, p < .001, η² = .346. However, there was not a main effect of Block on the third visit, F(7, 133) = 1.54, p = .16, η² = .08, suggesting that performance may have begun to plateau as participants became more familiar with the task.

Shot error—the Euclidean distance between the shot location and the center of the target—significantly decreased across visits, F(2, 38) = 50.65, p < .001, η² = .73, as shown in Figure 3B. A significant decrease across block was also observed on the first visit, F(7, 133) = 26.91, p < .001, η² = .59; the second visit, F(7, 133) = 11.27, p < .001, η² = .37; and the third visit, F(7, 133) = 3.27, p < .001, η² = .15. Here, lower values represent shots that were closer to the target. Therefore, decreases in this distance show that participants continuously enhanced their performance as they practiced.

In addition to the shot success and shot error outcome measurements described above, kinematic tracking of the participants allowed for calculating onset movement times of eye, hand, and head. As illustrated in Table 1, these movement times proceeded first with oculomotor RTs indexing the start of horizontal eye movements recorded through the HEOG channels at approximately 194 msec after target launch. This was followed within the first 300 msec by controller RTs and then head RTs, which marked the onset of hand and head movement, respectively. Finally, trigger response time occurred at approximately 1.5 sec, marking the average elapsed time between the target launch and the execution of the shot. Whereas oculomotor RTs were unchanged throughout the study and did not show a main effect of block or visit, other measures sped up throughout task practice. Namely, a significant decrease in controller RT was observed across visits, F(2, 38) = 27.53, p < .001, η² = .59, and blocks on the first visit, F(7, 133) = 13.27, p < .001, η² = .41; the second visit, F(7, 133) = 7.13, p < .001, η² = .27; and the third visit, F(7, 133) = 5.75, p < .001, η² = .23. These reductions indicate that hand movements became faster as participants practiced the task throughout the entirety of the training protocol. A significant decrease in head RT was also observed across visits, F(2, 38) = 3.92, p < .05, η² = .17, but was not observed across blocks on any of the visits. Finally, trigger response time significantly decreased across visits, F(2, 38) = 4.70, p < .05, η² = .20, as participants became faster overall at performing the task.

Scalp-recorded EEG provides detailed chronometry of the neural activity associated with performance on the marksmanship task. The following sections characterize the prelaunch preparatory-spectral, visual-evoked, and shot-locked EEG responses that occurred before, during, and after each trial of the task, respectively.

To assess the relationship between prelaunch preparatory EEG and postlaunch motor performance, spectral power was calculated from channels C3, C4, and Cz using the signals extracted in the 500-msec window after the trap house turned green (prior to the target launch). Time–frequency analysis of the spectral power time-locked to the moment of trap house turning green produced spectral power estimates that generally decreased at higher frequencies and were broadly distributed over the center of the head, as illustrated in the group average spectrogram in Figures 4A and topography for beta band power in Figure 4B.

Two LMEMs were fit to explore changes in preparatory alpha and beta power, over the course of practice with the marksmanship task. Trial number, task block, and visit were included in the LMEMs as fixed-effects terms with the intercept treated as a random effect among participants. In both LMEMs, results showed that alpha and beta power tonically increased with the growth factor of trial number (p < .001), controlling for other factors. Moreover, an LMEM was fit to test the predictive power of prelaunch preparatory EEG frequency band power, task block, and visit on postlaunch motor behavior measured by controller RT and shot error, respectively, given the random effects of intercept and nonlinear term (i.e., exponential function) among participants. The LMEM run on controller RT showed reasonable fit to the data, accounting for 4%, 13%, and 52% of the variance associated with the Level 1 residual term, Level 2 intercept term, and Level 2 exponential term, respectively, given that the intraclass coefficient estimates suggested a high proportion of data variance coming from Level 2 random variables (Cohen, Cohen, West, & Aiken, 2013). Table 2 demonstrates parameter estimates for the fixed-effects part of the LMEM, revealing that preparatory beta power was a significant predictor of controller RT, t(27907.30) = −3.53, p < .001. The regression coefficient can be interpreted such that, for each dB increase in beta power, controller RT was faster by 0.34 msec, when controlling for all other factors. Results of LMEM for shot error indicated that EEG preparatory power did not correlate with shot error. Overall, this evidence implicated a functional role of beta band EEG in the marksmanship task toward predicting controller RTs. Figure 4C–4E illustrates the tonic variability pattern of beta power, controller RT, and shot error across the entire sequence of experimental trials in the three-visit protocol. Block-wise (red) and visit-wise (green) regression lines are descriptively plotted along with group means and standard deviations.

Based on the lateralization of the visual system, an important a priori hypothesis in this study was the expectation that visual cortical responses would differ in the hemisphere contralateral to left or right trajectory launches. Moreover, based on the expectation that physiological responses in the visual cortex would correlate with behavior, it was expected that this contralateralized potential would be earlier and larger on trials with better behavioral performance. As shown in Figure 5A and 5B, an early ipsilateral positive VEP began at around 100 msec, followed by the onset of a larger contralateral negative VEP that began at 110–120 msec and ended before the onset of oculomotor RT at around 194 msec. Statistical tests performed between 0 and 200 msec, with Bonferroni method to adjust for multiple comparisons, showed that significant differences existed between the VEPs for the left and right trajectory directions in the range of 98–179 msec and 111–173 msec, respectively, as indicated by the black bars below the VEPs in each plot. Figure 5C and 5D illustrate the evolution of the topographic distribution of the evoked responses for left and right trajectories, respectively, demonstrating the presence of contralateralized responses over the posterior cortex, prior to the initiation of eye movements.

In order to understand the role of the VEP in behavior, LMEMs were fit to test the predictive power of the contralateral VEP amplitude, launch direction, task block, and visit on controller RT and shot error, given the random effect of intercept and nonlinear exponential term among participants. In addition, the LMEMs included a fixed-effects factor, launch direction, to account for whether the trial observation came from channel P3 or P4. The LMEM was run with data excluding trials of central launch trajectory (to be consistent with the requirement of calculating contralateral VEP amplitude).

Overall, the LMEM showed reasonable fit to the data, explaining 6%, 15%, and 72% of the variance associated with the Level 1 residual term, Level 2 intercept term, and Level 2 exponential term, respectively. Results of the fixed-effects part in the LMEM run on controller RT are shown in Table 3. VEP amplitude was identified as a significant predictor of controller RT, regardless of channel location, t(19076.33) = 11.10, p < .001.¹ Namely, for each μV increment in the negative-going VEP amplitude, the controller RT became faster by 0.21 msec, controlling for all other factors. In addition, the results suggested that participants, on average, reacted to targets launched to the right 5.94 msec faster than to those launched to the left, t(19069.34) = −15.70, p < .001. Results of LMEM run on shot error indicated VEP amplitude did not predict shot error.

To further evaluate learning that occurred over visits, additional rANOVAs were performed on the individual daily average VEP amplitudes for channels P3 and P4. The rANOVAs revealed significant decreases in amplitude across visits in both channel P3, F(2, 38) = 4.59, p < .05, η² = .20, and channel P4, F(2, 38) = 9.18, p = .001, η² = .33. These findings are illustrated for the grand average VEPs in Figure 6A and 6B, where successive decreases in amplitudes can be seen across consecutive visits.

To obtain an overall model of the behavioral and EEG measures leading up to shot execution that predict marksmanship performance, an LMEM was run regressing controller RT on the preshot EEG-based variables (EEG power bands and VEP), task block, visit, and launch direction, while allowing parameters associated with the intercept and nonlinear exponential term to vary among participants. The LMEM was run with identical trial observations used in the previous section of testing VEP amplitude's prediction on controller RT. Overall, the LMEM showed reasonable fit to data, accounting for 6%, 17%, and 66% of the variance associated with the Level 1 residual term, Level 2 intercept term, and Level 2 exponential term, respectively. Results from the fixed-effects part of the LMEM model are shown in Table 4. In particular, the results included previous findings that both beta power, t(19073.51) = −3.14, p = .002, and VEP amplitude, t(19072.35) = 11.05, p < .001, significantly predicted controller RT. Moreover, participants, on average, reacted to targets launched to the right 5.98 msec faster than those launched to the left, t(19038.00) = −15.82, p < .001.

A final set of exploratory analyses was performed to determine if a response similar to the FRN was present during the time period immediately following the trigger pull. In the present design, the pull of the trigger instantaneously froze the display, turned the target green or red depending on shot success, and displayed the participant's shot location relative to the location of the target. Because this happened in the context of ongoing movement to intercept the target, a first objective was to assess for the presence of an FRN-like response that is frontal-centrally distributed with greater negative amplitudes for misses, relative to hits. In the presence of such a response, a second objective was to determine whether the amplitude and/or latency of the FRN changed with practice and whether or not such changes correlated with behavioral learning. Finally, based on past demonstrations that the amplitude of the FRN predicts whether participants will learn to avoid an erroneous response on subsequent attempts (van der Helden, Boksem, & Blom, 2010), sequential-trial behavioral and brain effects were then evaluated under the hypotheses that (1) behavioral measures would be faster for trials following misses than hits and (2) that greater FRN amplitudes would be followed by more successful behavior on the subsequent trials.

Analysis of the FRN revealed differences between hits (black dashed line) and misses (gray dotted line), shown graphically in Figure 7A. The miss-minus-hit difference wave (black solid line) consists of a broadly distributed negative potential that peaks at approximately 280 msec, similar to the commonly reported FRN in the literature (Cohen et al., 2007; Hajcak et al., 2006), and is relatively unchanged in distribution over visits. The FRN scalp topography is displayed in Figure 7B. As displayed in waveforms in Figures 7C and evidenced in rANOVA results in Figure 7D, the peak latency of this potential occurred earlier across visits, F(1.44, 27.24) = 17.158, p < .001, η² = .475, shifting from 303 msec, to 300 msec, to 270 msec across the three visits. Similarly, rANOVA performed on the peak amplitude extracted within ± 20 msec of this peak also demonstrate significant differences across visits, F(1.77, 33.68) = 57.08, p < .001, η² = .75, with successive reductions in the amplitude over visits.

To better understand sequential trial effects, all trials were reclassified into four conditions: hits following misses, hits following hits, misses following misses, and misses following hits. Based on past research showing that individuals exert greater control following errors (Allain, Burle, Hasbroucq, & Vidal, 2009; Rabbitt, 1966), it was expected that participants would be more likely to hit the target following miss trials relative to hit trials. However, contrary to these expectations, rANOVA comparing the proportion of hits following hits to the number of hits following misses showed that participants were more likely to hit the target following a successful trial than an unsuccessful trial, F(1, 19) = 4.85, p = .04, η² = .20. Across the three visits, participants were successful 67.5% of the time following hits, but only 65.7% of the time following misses. A 2 (Outcome: hit or miss) × 2 (Subsequent Shot: hit or miss) × 3 (Visit) rANOVA performed on the FRN amplitudes indicated that, although there was a main effect of Shot Outcome on FRN amplitudes, F(1, 18) = 25.21, p < .001, η² = .58, this did not interact with the sequential shot result, F(1, 18) = 0.07, p = .79, indicating that performance on a given trial could not be predicted based on the amplitude of the shot-locked FRN on the previous trial. As such, the current exploratory analyses provide indication of an FRN response that changes with practice and evidence of behavioral sequential trial effects (akin to the “hot hand” where a person who experiences a successful outcome has a greater chance of success in further attempts), but do not provide support that the amplitude or latency of the FRN indexes behavioral changes across trials.

In this study, a simulated trap shooting task was used to investigate the behavioral and brain processes underlying visual–motor control and learning. Repeated natural movement patterns were measured with kinematic tracking, whereas brain activity was simultaneously measured with EEG as participants practiced and became, on average, 13.03% more accurate at shooting targets. Analysis of the behavioral measures revealed that performance improvements were accompanied by faster controller RTs, lower shot errors, and earlier trigger response times, indicating that less time was needed for motor planning, execution, and error correction as training progressed.

Analysis of the EEG data reveals several markers that correlated with upcoming motor performance and changed over time with practice. First in this sequence, beta band power measured in the brief 500-msec preparatory period before the target launch correlated with RTs of the upcoming shot. Next, following the target launch, a contralateral VEP was observed over posterior cortex, indicating that cortical processing of the target occurred before the onset of eye, hand, and head movements captured by the kinematics. The amplitude of this VEP became more negative with practice and correlated with controller RT, pointing to a neural mechanism that may contribute to the development of visual–motor proficiency. Last, upon execution of the trigger pull, there was a broadly distributed potential that was more negative for misses than hits, suggestive of the commonly reported FRN (Holroyd & Coles, 2002). Although this response became significantly smaller and earlier over the course of practice, it did not differ for subsequent hits versus misses as would be expected based on reinforcement learning models that have associated this response with reward prediction. Methodologically, it was shown that an MoBI approach is feasible for recording and analyzing EEG during a simulated trap shooting event and, by extension, other tasks of similar motor demand when conducted in an immersive and interactive CAVE-like VR system.

Collectively, these findings build on our previous results demonstrating kinematic motor learning (Rao et al., 2018; Zielinski, Rao, Potter, Appelbaum, & Kopper, 2016) and biomarker identification (Clements et al., 2018), test for changes in brain activity and body movements that accompany skill learning, and establish biomarkers that can be used to infer behavioral performance. In the following discussion, we first address the rapidly growing field of MoBI then provide context for the observed learning and brain–behavior correlation findings, before closing with limitations and future directions that can help move this field forward.

Traditional EEG experiments often record with motor responses in the form of minimal movements, such as button presses. In contrast, MoBI experiments simultaneously record multiple sources of information under naturalistic contexts allowing a more complete representation of the full-body behavior (Rao et al., 2020; Makeig, 2009). The current study used such an MoBI approach to provide important new insights into the brain dynamics accompanying the interception of a moving target in a naturalistic performance context with multiple levels of kinematic and brain measurement. Kinematic tracking performed inside a CAVE-like VR environment provided the exact timing of key events in simultaneous EEG recordings, which allowed for calculating brain responses throughout the full cascade of preparatory and execution steps of the target interception task. Importantly, although it might seem counterintuitive, EEG data collection in this immersive and interactive VR environment produced notably clean recordings. This is likely because of the fact that the participants stood in the center of the chamber with image projection from outside the cave onto the fabric walls, so the participants were actually quite far from interfering electrical noise sources. The fact that expected EEG metrics could be measured with quiet baseline intervals (demonstrating characteristic 1/f + alpha power spectrum), low artifact rejection rates (< 3.5% using 35-μV rejection threshold), and significant conditional differences indicates further that the psychophysiological approach used here is conducive to identifying EEG biomarkers of behavior outcomes. Moreover, these findings may offer important information for how to integrate EEG into the emerging use of VR in motor learning applications, performance training (Ahir, Govani, Gajera, & Shah, 2020; Appelbaum & Erickson, 2018), and rehabilitation (Levac et al., 2016; Levin, Weiss, & Keshner, 2015).

One of the main objectives in this research was to identify EEG markers of motor behavior during the simulated marksmanship task. The term “biomarker” commonly refers to a measurable indicator of a specific biological state, which can be used to describe a psychophysiological process underlying behavior. In recent years, there has been an increasing interest in the identification of biomarkers to assess skill in performance-related research (Lee et al., 2017; Cheron et al., 2016; Paulus et al., 2009). Validated biomarkers have the potential to aid coaches, athletes, and researchers in the design of better training programs that can increase their effectiveness. Despite interest in research on motor skills during dynamic activities, these studies typically involve minimally active laboratory tasks where the same principles may not apply to motor skills in more complex real-world situations.

In this study, the addition of EEG recording in a VR training environment added a new dimension into the investigation of the psychophysiological biomarkers that reflects complex motor behaviors. With this framework, candidates of behavior biomarker were extracted from EEG data and tested in three different sets of multilevel models (which are capable of accounting for the natural clustering structure in the data); one focused on the preparatory spectral power, one testing contralateral VEP amplitudes, and a combined model that each give different views of the correlational structure of these variables at different start (preparation, processing, and reaction) and end points (RT, response time, and shot error).

As observed in the LMEMs, beta band power appeared as a reliable correlate of upcoming performance. This may mirror past findings demonstrating a role for beta in visual–motor integration during preparatory motor processes (Zaepffel, Trachel, Kilavik, & Brochier, 2013; Perez, Lundbye-Jensen, & Nielsen, 2006). In this model, it is thought that phase locking of neural populations within the beta band reflects greater entrainment that facilitates neural information transfer during motor planning and execution (Rueda-Delgado et al., 2014; van Wijk, Beek, & Daffertshofer, 2012). When interpreted in this context, the finding may indicate the development of an internal mental model that aids participants in shot execution (Monfared, Tenenbaum, & Folstein, 2019). Interestingly, studies comparing expert and novice marksmen have shown that experts exhibit greater alpha band power in the seconds leading up to the trigger pull (Berka et al., 2010; Janelle & Hatfield, 2008; Hatfield et al., 2004; Hillman et al., 2000). This has been interpreted as indicating that expert shooters possess greater default mode network activity and lower active cognition when executing shots (i.e., more automaticity and less top–down control). As such, contrasting experts and novices on alpha power may reflect differences in brain responses at different levels of learning. Nonetheless, because of the limited temporal window under scrutiny (500 msec) in this shooting simulation, these data are treated as tonic markers that may not have the same association as phasic changes often described in the literature. Future studies are thus encouraged to slow the pace of shooting tasks to allow better access to these important processes. Through subtle modifications to the shooting task, future studies may therefore be able to explore the connection between tonic and phasic variability in EEG spectral power and how they map to motor control and learning process.

A further finding of interest was the observation that the amplitude of the contralateralized VEP time-locked to the target launch correlated positively with the controller RT. This link suggests that, in this time-limited target-interception task, greater processing resources early in the sensory cascade, indicated by more negative-going VEP amplitude, may have led to faster motor actions to intercept the target. This evoked potential, elicited to the visual detection of the target launches, is theorized to reflect early visual and attentional processes, occurring in the contralateral occipital–parietal cortex. This is thought to be similar to the N2pc reported in other rapid attentional orienting tasks (An et al., 2012), but motor planning circuits may also contribute to this evoked potential. We therefore encourage forthcoming investigations to further clarify these relevant ERP components in similar naturalistic motor tasks.

An important question in this research concerned the mechanisms of motor skill learning during the simulated marksmanship task. In particular, this task offers an interesting view to evaluate how visual-evoked responses and error detection signals may have changed with practice and contributed to skill acquisition. Although novel in its implementation, this approach extends past skill learning research with EEG, which helps exploit the contralateralization of the visual and motor cortex, and to derive lateralized ERPs that changed through practice (van den Berg, Appelbaum, Clark, Lorist, & Woldorff, 2016; Clark, Appelbaum, van den Berg, Mitroff, & Woldorff, 2015). Based on these and other studies (e.g., An et al., 2012), it has been proposed that learning is accompanied by reorganizations at multiple stages of the neural hierarchy with dynamically interacting reorganizations at each stage (Makino, Hwang, Hedrick, & Komiyama, 2016).

One observation from the current data was that the amplitude of the contralateralized VEP became significantly more negative over the course of practice, implying that changes in the brain processes might occur through training. This change in VEP amplitude is mirrored by observations in the LMEM results, which consistently demonstrated significant effects of trial number in that participants were improving over the course of the experiment, a phenomenon descriptively illustrated in Figure 4D and 4E for controller RT and shot error, respectively.

Finally, although highly exploratory in the context of this study, the FRN, which is known to occur in EEG recordings after a participant recognizes an error, was also found to occur at earlier latencies and with lower mean amplitudes on successive days of practice. Such changes in sensory processing have been observed with learning in other domains, including perceptual learning (Appelbaum, Wade, Vildavski, Pettet, & Norcia, 2006), visual search (Arns et al., 2008; Aggarwal et al., 2006), and reward learning (dos Santos Mendes et al., 2012). As observed in Figure 7, subsequent differences in the later positive P3 component may also have indexed changes on a trial-by-trial or visit-by-visit basis. However, because these differences may arise from uncontrolled processes (e.g., lingering on the feedback or score), it is not possible to uniquely associate these with error processing. Having demonstrated here that the FRN can be captured, future studies may construct designs that can better index the multiple processes underlying this component.

The present results highlight a multiscale approach to link kinematic measurements of eye, hand, and head movements with EEG recordings during natural dynamic actions in a fully immersive and interactive VR environment. The full-body orienting task required participants to actively interact with their environments using fast, accurate movements, with evidence for both kinematic and neural learning over the course of task practice. Moreover, both beta band activity in the short preparatory period prior to target launch and contralateral VEP amplitudes in the period immediately after the target launch correlated with faster hand RTs, signaling important aspects of the neural upstream that contributes to motor performance. Taken together, the present protocol demonstrates the ability to quantify neurophysiological substrates of motor control and learning, while providing an empirical platform for the continued development of MoBI for applied uses. Future work should continue to incorporate ecologically valid measures in testing expert marksmen with the eventual goal of deriving reliable biomarkers in real time to guide motor performance and learning.

The authors would like to thank all of the participants for their time and effort during this study.

Reprint requests should be sent to Lawrence G. Appelbaum, Department of Psychiatry, Duke University, 400 Trent Dr., Durham NC 27710, or via e-mail: [email protected].

This research was funded by a grant support to L. G. A. through the United States Army Research Office [W911NF-15-1-0390].

A retrospective analysis of the citations in every article published in this journal from 2010 to 2020 has revealed a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .408, W(oman)/M = .335, M/W = .108, and W/W = .149, the comparable proportions for the articles that these authorship teams cited were M/M = .579, W/M = .243, M/W = .102, and W/W = .076 (Fulvio et al., JoCN, 33:1, pp. 3–7). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance.