Abstract

Understanding how the human brain integrates information from the environment with intrinsic brain signals to produce individual perspectives is an essential element of understanding the human mind. Brain signal complexity, measured with multiscale entropy, has been employed as a measure of information processing in the brain, and we propose that it can also be used to measure the information available from a stimulus. We can directly assess the correspondence between brain signal complexity and stimulus complexity as an indication of how well the brain reflects the content of the environment in an analysis that we term “complexity matching.” Music is an ideal stimulus because it is a multidimensional signal with a rich temporal evolution and because of its emotion- and reward-inducing potential. When participants focused on acoustic features of music, we found that EEG complexity was lower and more closely resembled the musical complexity compared to an emotional task that asked them to monitor how the music made them feel. Music-derived reward scores on the Barcelona Music Reward Questionnaire correlated with less complexity matching but higher EEG complexity. Compared with perceptual-level processing, emotional and reward responses are associated with additional internal information processes above and beyond those linked to the external stimulus. In other words, the brain adds something when judging the emotional valence of music.

INTRODUCTION

Various contemporary theories suggest that functional brain networks engage and disengage to integrate information during cognitive processes (Bressler & Kelso, 2001; McIntosh, 2000; Tononi, Sporns, & Edelman, 1994). This network activity generates variable brain signals that appear necessary for optimal function, which is also a characteristic of complex systems (Deco, Jirsa, & McIntosh, 2011; Ghosh, Rho, McIntosh, Kötter, & Jirsa, 2008; McIntosh, Kovačević, & Itier, 2008). Here, we use the term complexity formally to define a system that optimizes integration and segregation of information, rather than simply being complicated. By this token, actual measures of complexity can be used to assess the relative amount of information in a signal (Tononi et al., 1994). For example, brain signal complexity is higher during states of greater knowledge representation (e.g., Heisz, Shedden, & McIntosh, 2012) and increases following longitudinal music training (Carpentier, Moreno, & McIntosh, 2016). This study measured brain signal complexity to investigate whether a correspondence exists between information patterns in brain signals and those in the individual's environment. Furthermore, we were interested in whether this correspondence would be related to the individual's cognitive–affective state.

Music offers an ideal stimulus from which to measure information content. Sophisticated music structures are created following the application of combination rules to subordinate motifs. This makes it possible to describe a functional isomorphism (correspondence between brain signal complexity and stimulus complexity) between the brain signal and environment information by calculating music complexity and comparing it to the complexity of brain signals of the listener. We propose that this measure of comparison between EEG complexity and music complexity, or “complexity matching,” will provide a relative indication of the degree to which environmental information structure is reflected in brain signal structure.

Complexity matching is calculated as Procrustes distance (Gower, 1975) between the music and EEG multiscale entropy (MSE). MSE calculates sample entropy at multiple timescales (Costa, Goldberger, & Peng, 2002, 2005). Like brain signals, music also has structure at multiple timescales, and MSE may provide an appropriate measure of complexity for a given music passage. Procrustes distance calculates the similarity between the MSE profiles for both the music and the accompanying brain signals. A relatively high value of matching (described by a low Procrustes distance value) between lower EEG and MSE would suggest that the brain has activated the necessary information processing resources for immediate perception and little else. Conversely, a relatively low complexity matching value (described by a high Procrustes distance value) paired with high neural complexity would suggest that internal processes, in addition to those for immediate stimulus perception, dominate the neural response. This metric was inspired by the ideas in Tononi, Sporns, and Edelman (1996).

This level of brain–environment information integration may be related to subjective perspective. To examine the relationship between brain–environment matching and cognitive–affective state, we calculated complexity matching while participants performed a music perception task and a music-evoked emotion task. We expected that active attention to the acoustic features of the music during the perceptual task would be accompanied by brain signal complexity that more closely resembled these acoustic features. Comparatively, we expected the emotional task to involve additional internal processes and, therefore, to display a lower degree of matching with the environment.

We also analyzed the relationship between complexity matching and scores from the Barcelona Music Reward Questionnaire (BMRQ), a measure of an individual's musical reward and music listening habits. The ability to perceive musical structure is essential to the enjoyment of music (Huron, 2006; Meyer, 1956). It is therefore possible that a minimum quantity of complexity matching may be required for the listener to have an appreciation of the underlying “gist” or skeleton structure of a piece, which may then lead to enjoyment. Without sufficient neural integration of music signals, the listener would perhaps be unable to perceive separate noise sounds as unified, leading to decreased enjoyment. In accordance with our prediction that the emotional task would be associated with lower complexity matching values than the perceptual task, we expected that higher music reward involves internally driven, individual responses and therefore would correlate with higher neural complexity and lower complexity matching between brain signals and musical complexity.

METHODS

Participants

Eighteen healthy young adults aged 19–35 years (M = 26, 10 women) were recruited from the Greater Toronto Area to take part in the study and provided written informed consent in accordance with the joint Baycrest Centre–University of Toronto Research Ethics Committee. Prior to arriving to the lab for the experimental session, participants completed an online questionnaire about their music listening habits, and musical training was assessed as a 1–5 scale: 1 = no formal training, cannot play an instrument; 2 = can play an instrument without formal training; 3 = less than 1 year of formal music training; 4 = between 1 and 5 years of formal training; and 5 = more than 5 years of formal training.

Barcelona Music Reward Questionnaire

Music reward is highly individual, and the BMRQ was developed to describe how people experience reward from music listening (Mas-Herrero, Marco-Pallares, Lorenzo-Seva, Zatorre, & Rodriguez-Fornells, 2013). Participants are asked to indicate the level of agreement with each of 20 statements by using a 5-point Likert scale ranging from 1 = fully disagree to 5 = fully agree, with a higher score indicating that the individual experiences more music reward and a lower score indicating they do not experience music-associated rewarding feelings. These statements represent five major factors of music reward: (1) emotional evocation, (2) mood regulation, (3) musical seeking, (4) social reward, and (5) sensory–motor.

Behavior Tasks

Forty musical excerpts were selected from Western art music to describe a range of emotional reactions and perceptual features. Pieces included a range of instrumentation including solo instrumental, orchestral, chamber, and operatic works, and we aimed to induce a broad range of emotional and perceptual responses. Segment lengths ranged between 0:40 and 1:17 min. This choice was made to allow each segment to conclude naturally at the end of a musical phrase, rather than ending abruptly in the middle. Thirty pieces were selected for the emotional task, and 10 pieces were selected for the perceptual task (Table 1).

Table 1. 
List of Songs for Each Task
Emotional Judgment Songs
• Adams “Nixon in China, ‘Beginning’” • Penderecki “Threnody to the Victims of Hiroshima” 
• Adams “Disappointment Lake” • Puccini “O soave fanciulla” 
• Bach “No. 3 Aria ‘Es Ist Vollbracht’” • Rameau “Entrée de Polymnie” 
• Barber “Adagio for Strings” • Richter “Vivaldi's Summer” 
• Brahms “Intermezzo No. 2 in a Major, Op. 118” • Rossini “Barbiere di Siviglia: Largo Al Factotum” 
• Delibes “Lakmé/Flower Duet” • Schroer “Field of Stars” 
• Elgar “Variation IX (Adagio) ‘Nimrod’” • Schumann-Liszt “Liebeslied (Widmung)” 
• Galvany “Oh My Son” • Staniland “Solstice Songs No. 2 Interlude” 
• Gluck “Armide Act Iv Air Sicilien” • Stravinsky “Glorification of the Chosen One” 
• Goodall “Belief” • Tarrega “Recuerdos De La Alhambra” 
• Ives “Three Places in New England Orchestral Set No.1” • Verdi “Messa Da Requiem: Dies Irae-Tuba Mirum Part 1” 
• Liszt “Totentanz” • Verdi “Messa Da Requiem: Dies Irae-Tuba Mirum Part 2” 
• Monteverdi “Zefiro Torna” • Wagner “Die Walkurie, Act 3: Ride of the Valkyries” 
• Mozart “Cosi fan tutte” • Wagner “Tristan Und Isolde/Act 2–Prelude” 
• Mozetich “The Passion of Angels”   
• Part “Spiegel im Spiegel”   
  
Perceptual Songs
• Beethoven “Sonata in A Major Op. 69” • Strauss–Der Rosenkavalier Act III/Duet-Denouement and Grand Waltz–Coda” 
• Brahms “Violin Concerto in D, Op. 77-3” • Strauss “September” 
• Glass “Glassworks Opening” • Vivaldi “Concerto for Violin, Stings and Harpsichord in G” 
• Haydn “Cello Concerto in D Major” • Vivaldi “Stabat Mater” 
• Mozart “Symphony No. 40 in G-minor, K. 550, Finale”   
• Praetorious “Praeambulum”   
Emotional Judgment Songs
• Adams “Nixon in China, ‘Beginning’” • Penderecki “Threnody to the Victims of Hiroshima” 
• Adams “Disappointment Lake” • Puccini “O soave fanciulla” 
• Bach “No. 3 Aria ‘Es Ist Vollbracht’” • Rameau “Entrée de Polymnie” 
• Barber “Adagio for Strings” • Richter “Vivaldi's Summer” 
• Brahms “Intermezzo No. 2 in a Major, Op. 118” • Rossini “Barbiere di Siviglia: Largo Al Factotum” 
• Delibes “Lakmé/Flower Duet” • Schroer “Field of Stars” 
• Elgar “Variation IX (Adagio) ‘Nimrod’” • Schumann-Liszt “Liebeslied (Widmung)” 
• Galvany “Oh My Son” • Staniland “Solstice Songs No. 2 Interlude” 
• Gluck “Armide Act Iv Air Sicilien” • Stravinsky “Glorification of the Chosen One” 
• Goodall “Belief” • Tarrega “Recuerdos De La Alhambra” 
• Ives “Three Places in New England Orchestral Set No.1” • Verdi “Messa Da Requiem: Dies Irae-Tuba Mirum Part 1” 
• Liszt “Totentanz” • Verdi “Messa Da Requiem: Dies Irae-Tuba Mirum Part 2” 
• Monteverdi “Zefiro Torna” • Wagner “Die Walkurie, Act 3: Ride of the Valkyries” 
• Mozart “Cosi fan tutte” • Wagner “Tristan Und Isolde/Act 2–Prelude” 
• Mozetich “The Passion of Angels”   
• Part “Spiegel im Spiegel”   
  
Perceptual Songs
• Beethoven “Sonata in A Major Op. 69” • Strauss–Der Rosenkavalier Act III/Duet-Denouement and Grand Waltz–Coda” 
• Brahms “Violin Concerto in D, Op. 77-3” • Strauss “September” 
• Glass “Glassworks Opening” • Vivaldi “Concerto for Violin, Stings and Harpsichord in G” 
• Haydn “Cello Concerto in D Major” • Vivaldi “Stabat Mater” 
• Mozart “Symphony No. 40 in G-minor, K. 550, Finale”   
• Praetorious “Praeambulum”   

During the emotional task, a computer screen in front of participants showed four quadrants marked on two dimensions: stimulating–relaxing and pleasant–unpleasant (Figure 1). Participants were asked to move a mouse around the quadrant space in a continuous manner during each song based on “how the music made them feel” on the two dimensions. Participants were explicitly instructed to report their own feelings during music listening and not the alternative of reporting on what emotions they believe are expressed in the music (emotional conveyance). The task design was based on Hunter and Schellenberg's (2010) valence–arousal model. In this model, two dimensions describe arousal (high to low) and valence (positive to negative). We modified the labels to energy and pleasantness in this study to render the task more intuitive for participants. This valence–arousal model is designed to capture a wide range of emotions. In their study, difference valence and arousal combinations were associated with multiple different emotions. For example, high arousal/negative valence was correlated with distress, fear, and anger; whereas low arousal/positive valence was associated with feelings of peace, contentment, and relaxation. Participants from our pilot sample gave similar reports. In this way, it is possible to capture a larger range of emotions without limiting responses to more specific emotions.

Figure 1. 

Participants viewed screens with each of the above quadrants during each task. They were asked to move a mouse continuously around the quadrant space depending on how the music was making them feel in that moment (emotional task) or based on their judgments of pitch and tempo (perceptual task).

Figure 1. 

Participants viewed screens with each of the above quadrants during each task. They were asked to move a mouse continuously around the quadrant space depending on how the music was making them feel in that moment (emotional task) or based on their judgments of pitch and tempo (perceptual task).

The perceptual task mimicked the emotional task, with participants assessing pitch and tempo for each song (Figure 1). Once again, a screen displayed four quadrants with two dimensions (high–low pitch and fast–slow tempo), and participants moved a mouse in a continuous manner on the screen based on the pitch and tempo of each song.

Experimental Procedure

All stimuli were presented through ER 3A insert earphones (Etymotic Research) while participants were seated in a soundproof room. The experimental session began with five perceptual task songs, followed by 30 emotional songs, and concluded with the remaining five perceptual songs. Perceptual songs were always presented in the same order. Emotional songs were presented in one of two counterbalance orders. Pieces in the first order were curated to have a sense of flow between them and avoid jarring transitions from one song to the next that may disrupt emotional experiences. The second order was the reverse of the first. There was no significant effect of counterbalance order on any of our measures.

EEG Recording and Preprocessing

EEG was recorded using a 64 + 10 channel Biosemi Active Two System at a sampling rate of 512 Hz. Continuous EEG recordings were bandpass filtered at 0.5–90 Hz, with a notch filter at 55–65 Hz for line noise. EEG data for each song were segmented into 4 × 10 sec epochs in accordance with the shortest excerpt (40 sec), and each was baseline corrected based on a 200-msec prestimulus interval. Trials with excessive signal amplitude were rejected. Ocular and muscle artifact removal was performed on the remaining concatenated trials using independent component analysis implemented in EEGLAB (Delorme & Makeig, 2004). The highest number of trials lost for any participant was 8 of 40, seven participants retained all trials, and the average number of rejected trials from the remaining participants was 2.67, with no difference in trial rejection between conditions.

We performed source estimation at the 68 ROIs of the Desikan–Killiany Atlas (Desikan et al., 2006) (see Table 2), using sLORETTA (Pascual-Marqui, 2002) as implemented in Brainstorm (neuroimage.usc.edu/brainstorm; Tadel, Baillet, Mosher, Pantazis, & Leahy, 2011). Source reconstruction was constrained to the cortical mantle of the brain template MNI/Colin27 defined by the Montreal Neurological Institute (Holmes et al., 1998). Current density for one source orientation (X component) was estimated for 15,768 equally spaced vertices, and the source waveform was mapped at the 68 brain regions of interest as an average taken over all vertices in each region. MSE was calculated on the source waveform at each ROI for each participant as a measure of brain signal complexity.

Table 2. 
List of Abbreviations for Regional Sources
BankST: Bank of the Superior Temporal Sulcus 
cACC: caudal anterior cingulate cortex 
cMF: caudal middle frontal gyrus 
Cun: cuneus 
Ent: entorhinal cortex 
FP: frontal pole 
Fus: fusiform gyrus 
IP: inferior parietal lobe 
IT: inferior temporal gyrus 
Isth: isthmus of the cingulate gyrus 
lOcc: lateral occipital cortex 
lOFC: lateral orbital frontal cortex 
Ling: lingual gyrus 
mOFC: medial orbital frontal cortex 
MT: middle temporal gyrus 
ParC: paracingulate cortex 
ParH: parahippocampal gyrus 
P Op: prefrontal pars opercularis (area 44) 
P Orb: prefrontal pars orbitalis (area 47) 
P Tri: prefrontal pars triangularis (area 45) 
PCal: precalcarine 
PostC: postcentral gyrus 
PCC: posterior cingulate cortex 
PreC: precentral gyrus 
PreCun: precuneus 
rACC: rostral anterior cingulate cortex 
rMF: rostral middle frontal gyrus 
SF: superior frontal gyrus 
SP: superparietal cortex 
ST: superior temporal gyrus 
SMG: supramarginal gyrus 
TP: temporal pole 
TT: transverse temporal cortex 
BankST: Bank of the Superior Temporal Sulcus 
cACC: caudal anterior cingulate cortex 
cMF: caudal middle frontal gyrus 
Cun: cuneus 
Ent: entorhinal cortex 
FP: frontal pole 
Fus: fusiform gyrus 
IP: inferior parietal lobe 
IT: inferior temporal gyrus 
Isth: isthmus of the cingulate gyrus 
lOcc: lateral occipital cortex 
lOFC: lateral orbital frontal cortex 
Ling: lingual gyrus 
mOFC: medial orbital frontal cortex 
MT: middle temporal gyrus 
ParC: paracingulate cortex 
ParH: parahippocampal gyrus 
P Op: prefrontal pars opercularis (area 44) 
P Orb: prefrontal pars orbitalis (area 47) 
P Tri: prefrontal pars triangularis (area 45) 
PCal: precalcarine 
PostC: postcentral gyrus 
PCC: posterior cingulate cortex 
PreC: precentral gyrus 
PreCun: precuneus 
rACC: rostral anterior cingulate cortex 
rMF: rostral middle frontal gyrus 
SF: superior frontal gyrus 
SP: superparietal cortex 
ST: superior temporal gyrus 
SMG: supramarginal gyrus 
TP: temporal pole 
TT: transverse temporal cortex 

Data Analyses

Multiscale Entropy

MSE has been previously validated as a measure of brain signal complexity (Catarino, Churches, Baron-Cohen, Andrade, & Ring, 2011; Mišić, Mills, Taylor, & McIntosh, 2010; McIntosh et al., 2008; for details on the use for neuroscience, see Courtiol et al., 2016). We calculated MSE in two steps using the algorithm available at www.physionet.org/physiotools/mse. First, the source EEG and music signals were progressively down-sampled into multiple coarse-grained timescales where, for scale τ, the time series is constructed by averaging the data points with nonoverlapping windows of length τ. Each element of the coarse-grained time series, yj(τ), is calculated according to Equation 1,
yjτ=1τi=j1τ+1jτxi,1Nτ
(1)

The number of scales is determined by a function of the number of data points in the signal, and MSE was calculated for 100 timescales (sampling rate [512 Hz] × epoch [10,000 msec]/50 time points per epoch = maximum of 102.4 scales).

Second, the algorithm calculates the sample entropy (SE) for each coarse-grained time series yj(τ),
SEmrN=lni=1Nmnimi=1Nmnim+1
(2)

Sample entropy quantifies the predictability of a time series by calculating the conditional probability that any two sequences of m consecutive data points that are similar to each other within a certain similarity criterion (r) will remain similar at the next point (m + 1) in the data set (N), where N is the length of the time series (Richman & Moorman, 2000). In this study, MSE was calculated with pattern length set to m = 2, and a similarity criterion of r = .5. The value r is defined as a proportion of the standard deviation of the original data (Costa et al., 2005; Richman & Moorman, 2000). MSE estimates were obtained for each participant's EEG source time series as a mean across single-trial entropy measures for each timescale per source.

Music pieces were imported into Matlab using the wavread function at a sampling rate of 11.25 kHz (MathWorks, Inc., Release 2011b). Music auditory signal MSE was subsequently calculated with the same parameter values and the same number of timescales as the EEG source MSE.

Complexity Matching

Complexity matching applies Procrustes analysis to measure the equivalence of the MSE curve for the auditory signal of a song (X1) and the MSE curve of the EEG source time series of a participant listening to that song (X2j), for all j ROIs individually (Gower, 1975). It minimizes the sum of the squared deviations between matching corresponding points (landmarks) from each of the two data sets (MSE curves), allowing for scaling, translation, and orthogonal rotation of X1 to fit X2j, where choice of label X1 or X2 is arbitrary. X1 and X2j must have the same number of i sample points or “landmarks,” and Procrustes matches X1i to X2ij. In our simple case of two vectors, the rotation matrix T such that X1 best fits X2j is given as T = VU from the singular value decomposition X1X2j = USV. Dissimilarity of X1 and X2j is given as the Procrustes distance,
distance=i=1X1TX2j2
(3)
A smaller distance value denotes greater similarity between the two curves or a closer match between them. The analysis returns a distance value for each ROI for each participant. Procrustes distance was calculated using the Matlab function procrustes (MathWorks, Inc., Release 2011b). Figure 2 presents a conceptual depiction of our implementation of “complexity matching.”
Figure 2. 

Complexity matching applies Procrustes analysis to determine a linear transformation (scaling, translation, and orthogonal rotation) of the points in x2 (EEG source MSE for a given ROI) to best match the points in x1 (song MSE). The goodness-of-fit criterion is the sum of squared errors, and Procrustes distance is the minimized value of this dissimilarity measure. Distance is standardized by a measure of the scale of x1.

Figure 2. 

Complexity matching applies Procrustes analysis to determine a linear transformation (scaling, translation, and orthogonal rotation) of the points in x2 (EEG source MSE for a given ROI) to best match the points in x1 (song MSE). The goodness-of-fit criterion is the sum of squared errors, and Procrustes distance is the minimized value of this dissimilarity measure. Distance is standardized by a measure of the scale of x1.

Partial Least Squares

Task partial least squares (PLS) analysis was used to statistically assess task- and epoch-related effects. Task PLS is a multivariate statistical technique similar to canonical correlation that employs singular value decomposition to extract mutually orthogonal latent variables (LVs) that capture the maximum covariance between the task design and neural activity. Each LV consisted of (1) a singular vector of design contrast, (2) a singular vector of weights showing the pattern across brain regions and sampling scales for the contrast, and (3) a singular value (s) representing the covariance between the design contrast and the brain pattern (McIntosh & Lobaugh, 2004; McIntosh, Bookstein, Haxby, & Grady, 1996).

The statistical significance of each LV was determined using permutation testing (McIntosh & Lobaugh, 2004; Good, 2000). The reliability of each statistical effect was assessed through bootstrap estimation of standard error and confidence intervals (CIs) of the singular vector weights in each LV (Efron & Tibshirani, 1986).

Finally, the dot product of an individual's raw MSE data and the singular image from the LV was used to produce a brain score. The brain score is similar to a factor score that indicates how strongly an individual expresses the patterns of the LV and allowed us to estimate 95% CIs for the effects in each group and task condition using the bootstrap procedure described above.

Behavioral PLS is a variation on task PLS for analyzing the relationship between brain measures and behavior (Krishnan, Williams, McIntosh, & Abdi, 2011; McIntosh & Lobaugh, 2004). Similar to task PLS, the singular value decomposition returns mutually orthogonal LVs, where each LV contains (1) a singular vector of weights for the behavioral measures, (2) a singular vector of for brain activity, and (3) a singular value (s) representing the covariance between the behavior and the brain. Permutation and bootstrap resampling were used for significance and reliability assessment as described for task PLS.

RESULTS

One participant was excluded from the study for mild hearing loss determined by audiogram, and one other was removed for excessive motion during EEG recording, leaving n = 16 participants. Two participants did not complete the music training questionnaire, leaving 14 participants. For music training, of the n = 14 participants, seven reported “no formal training, cannot play an instrument,” one reported “can play an instrument without formal training,” one reported “less than 1 year of formal music training,” two reported “between 1 and 5 years of formal training,” and four reported “more than 5 years of formal training.” Where we did not capture musical training in years, a sufficient number of participants who fulfill the common requirements for musicianship was not established (e.g., at least 10 years of formal music training; Fujioka, Trainor, Large, & Ross, 2009), and we did not proceed with analysis of the effects of music training.

MSE curves of sound signals from a sample of the songs are visualized in Figure 3 for illustration purposes.

Figure 3. 

MSE curves obtained from the sound signals of a 10-sec sample of the songs from each task. Temporal scale (in msec) refers to the size of nonoverlapping windows within which data points were averaged; hence, the left most values represent fine temporal scales and right more coarse scales.

Figure 3. 

MSE curves obtained from the sound signals of a 10-sec sample of the songs from each task. Temporal scale (in msec) refers to the size of nonoverlapping windows within which data points were averaged; hence, the left most values represent fine temporal scales and right more coarse scales.

Importantly, there was no difference in the music complexity (MSE) between the perceptual and emotional judgment tasks. We also analyzed musical excerpts using the MIRtoolbox Version 1.7.2 (Lartillot & Toiviainen, 2007). Eleven features were extracted to describe low- and high-level aspects of the music. Low-level features describe the quality of the sound (timbre, amplitude, texture) and may shift rapidly. High-level features describe the content of the musical excerpt and require context before shifts are perceived. As with MSE, these MIR features did not differ between perceptual and emotional judgment tasks.

There were no within-task effects of emotional (e.g., stimulating compared to relaxing) or pitch/tempo (fast compared to slow) dimension ratings on any of our brain measures (MSE, Procrustes distance or PSD; all PLS p > .10). This may be due to the high level of variance between individuals' emotional responses (Figure 4) or because the continuous nature of the behavior ratings is not well suited to the dichotomization necessary for the present analyses.

Figure 4. 

Example single-participant, single-song behavior heat maps for the emotional judgment task. Plots indicate the proportion of time the cursor controlled by a computer mouse was in a location. Note the variation between individuals in both the subjective arousal and valence and the variability within each participant and song (i.e., some participants were stable in one quadrant, whereas some participants felt a greater range during the same song).

Figure 4. 

Example single-participant, single-song behavior heat maps for the emotional judgment task. Plots indicate the proportion of time the cursor controlled by a computer mouse was in a location. Note the variation between individuals in both the subjective arousal and valence and the variability within each participant and song (i.e., some participants were stable in one quadrant, whereas some participants felt a greater range during the same song).

Examining both tasks across all four epochs, the emotional task was generally associated with higher EEG source MSE at finer time scales, compared with the perceptual task that showed higher MSE at coarser timescales (>40; LV = 1, p ≪ 0, singular value = 1.81, 31.6% cross-block covariance; Figures 5 and 6). Both tasks showed an increase in MSE at finer timescales (<20) and a decrease in coarse scale MSE across epochs from the beginning to the end of the piece of music (bar graph, Figure 5). The spatial distribution of these effects was such that the emotional task was associated with higher MSE in finer timescales (Figure 5A) in the bilateral bank of the superior temporal sulcus (bankSTS) and inferior parietal cortex; left hemisphere caudal and rostral middle frontal, and precentral regions; right medial pFC, paracentral, pars triangularis, rostral ACC, precuneus, SP MT, and ST. MSE was reliably higher for the perceptual task at coarse scales (Figure 5B) in bilateral insula, cingulate, ST, PCC, cuneus, and pericalcarine; left FG, mOFC, SF, parahippocampal, PCC, precuneus, and lingual; and right hemisphere MF, FP, OFC, IFG, postcentral, entorhinal, and temporal cortex.

Figure 5. 

PLS first LV for the examination of effects of between-tasks and within-task epochs on MSE. (A) The bar graph (95% CI) depicts the data-driven contrast highlighting higher MSE on all epochs of the emotional task compared to the perceptual task, as well as epoch effects within each task. (B) Cortical regions at which the contrast was most stable as determined by bootstrapping. Values represent the ratio of the parameter estimate for the source divided by the bootstrap-derived standard error (roughly z scores). Regions are listed for left and then right hemispheres using the abbreviations in Table 2. (C) Cortical visualization of stable bootstrap values for fine (left) and coarse (right) scales.

Figure 5. 

PLS first LV for the examination of effects of between-tasks and within-task epochs on MSE. (A) The bar graph (95% CI) depicts the data-driven contrast highlighting higher MSE on all epochs of the emotional task compared to the perceptual task, as well as epoch effects within each task. (B) Cortical regions at which the contrast was most stable as determined by bootstrapping. Values represent the ratio of the parameter estimate for the source divided by the bootstrap-derived standard error (roughly z scores). Regions are listed for left and then right hemispheres using the abbreviations in Table 2. (C) Cortical visualization of stable bootstrap values for fine (left) and coarse (right) scales.

Figure 6. 

Mean MSE across participants for right superior temporal cortex ROI. (A) All temporal scales, emphasizing higher MSE in the perceptual versus emotional task. (B) Zoomed into visualize higher MSE for the emotional task at fine scales relative to the perceptual task (<39.1 msec).

Figure 6. 

Mean MSE across participants for right superior temporal cortex ROI. (A) All temporal scales, emphasizing higher MSE in the perceptual versus emotional task. (B) Zoomed into visualize higher MSE for the emotional task at fine scales relative to the perceptual task (<39.1 msec).

Procrustes distance was larger (i.e., lower complexity matching with the music clips) during the emotional task than the perceptual task in most brain sources (LV = 1, p = 0, singular value = 0.81, 54.1% of the cross-block covariance; Figure 7). Most brain regions showed lower complexity matching on the emotional task except for bilateral entorhinal, FP, IT, parahippocampal, TP; left bankSTS MT, pars opercularis, ST; and right FG.

Figure 7. 

First significant PLS result for the examination of effects of between-tasks and within-task epochs on Procrustes distance. (A) The bar graph (95% CI) depicts the data-driven contrast highlighting greater distance on all epochs of the emotional task compared to the perceptual task, as well epoch effects within each task, significantly expressed across the entire data set, as determined by permutation tests (p = 0). (B) Cortical regions at which the contrast was most stable as determined by bootstrapping. Values represent the ratio of the parameter estimate for the source divided by the bootstrap-derived standard error (roughly z scores).

Figure 7. 

First significant PLS result for the examination of effects of between-tasks and within-task epochs on Procrustes distance. (A) The bar graph (95% CI) depicts the data-driven contrast highlighting greater distance on all epochs of the emotional task compared to the perceptual task, as well epoch effects within each task, significantly expressed across the entire data set, as determined by permutation tests (p = 0). (B) Cortical regions at which the contrast was most stable as determined by bootstrapping. Values represent the ratio of the parameter estimate for the source divided by the bootstrap-derived standard error (roughly z scores).

Music Reward

With the relatively small sample size (n = 16), we decided to use an average score across all BMRQ subfactors as the overall measure of music reward. We focused our behavioral PLS on the correlation of the participant reward scores and Procrustes distance.

A significant positive correlation was observed between distance and reward during the emotional task for all epochs (PLS LV1 p = .012, r2 = .13, singular value = 4.29, 72.9% of cross-block covariance; Figure 8; perceptual task p > .10). This effect was localized to the right hemisphere frontal regions, right ACC, IP, inferior and middle temporal, and lOcc. No significant correlations were observed between distance and reward for the control task (p > .10).

Figure 8. 

Behavior PLS result examining the correlation between complexity distance on all epochs of the emotional task and BMRQ music reward score. (A) Brain scores correlations (95% CI) with BMRQ total across epochs. (B) Brain regions at which the relationship was most stable as determined by bootstrapping. Together A and B indicate a positive correlation between distance and BMRQ score in the highlighted regions. (C) Scatterplot of the brain scores from the first epoch with BMRQ reward scores further displays this positive relationship (r2 = .13).

Figure 8. 

Behavior PLS result examining the correlation between complexity distance on all epochs of the emotional task and BMRQ music reward score. (A) Brain scores correlations (95% CI) with BMRQ total across epochs. (B) Brain regions at which the relationship was most stable as determined by bootstrapping. Together A and B indicate a positive correlation between distance and BMRQ score in the highlighted regions. (C) Scatterplot of the brain scores from the first epoch with BMRQ reward scores further displays this positive relationship (r2 = .13).

DISCUSSION

We found higher complexity matching in widespread brain regions during a perceptual music listening task compared to an emotional music listening task using Procrustes distance to compare the MSE of EEG signals to the MSE of the music itself. This indicates that brain signal complexity more closely resembles the complexity of the music when participants attend to the acoustic features of a piece of music compared to when they monitor their emotional response to the music. These results suggest that the level of neural “mirroring” can be related to the type of cognitive processing conducted.

Analysis of the EEG MSE values alone found that MSE was higher in relatively finer timescales (<48.8 msec) during the emotional task compared to the perceptual task and that both groups showed an increase of this pattern as the music progressed. The MSE results and the complexity matching results together suggest that the emotional task engaged additional processes above and beyond those employed in bottom-up stimulus perception. The brain regions that displayed this MSE effect are frequently linked to music cognition (e.g., right temporal, inferior frontal gyrus; Zatorre, Belin, & Penhune, 2002) and self-referential emotional processing (e.g., medial PFC; Denny, Kober, Wager, & Ochsner, 2012; Amodio & Frith, 2006; Ochsner et al., 2004). The effect was also observed in regions that operate as network hubs (e.g., medial parietal) that are densely connected to neighboring regions and have long-range interconnections, enabling efficient global integration of information (van den Heuvel & Sporns, 2011; Zamora-López, Zhou, & Kurths, 2010; Hagmann et al., 2008). This effect is spatially and temporally (<48.8 msec) similar to increased complexity associated with musical training (Carpentier et al., 2016) and in other studies where higher MSE has been linked to performance on cognitive tasks that require higher information processing (Heisz et al., 2012; Mišić et al., 2010). Additionally, because of the longer stimulus presentation, the current study opens up the analysis to coarser scales, up to 195.3 msec, well beyond what has been typically used in our previous studies and other empirical studies of cognition and brain MSE. We observed that MSE was initially higher in the perceptual task and decreased over temporal scale range of 78.1–195.3 msec. We suggest the possibility that complexity at these slow timescales is indicative of an initial distributed and reentrant processing of the fine acoustic details of the music, followed by a decrease at coarse scales as there is no need to integrate stimulus information over the longer time window. However, further investigation of the link between perception, cognition, and complexity at these scales is required.

Higher music reward scores on the BMRQ were associated with lower complexity matching, and this correlation was strongest in the emotional task. The complementary MSE and matching results suggest that music reward engages a brain state that is quantitatively and qualitatively different than music sound perception alone. Evaluation of music reward is associated with the activation of multiple different intrinsic processes and high information integration. This makes intuitive sense: Music reward is a product of what is added subjectively to perception of the immediate sensory events.

The observed relationship between higher neural information generation and music reward may relate to the direct reward experience itself, since pleasurable responses to music are associated with particular patterns of cortical and subcortical activity not observed during neutral music perception. Multiple studies have reported correlations between music reward and BOLD activity in ventromedial pFC and OFC, and also IFG, ACC and sensory motor areas (Salimpoor et al., 2013; Blood & Zatorre, 2001). The relationship between complexity and reward in this study was observed in temporal regions, as well as paralimbic and cortical regions involved in emotional processing (e.g., OFC, insula). Another proposal, not mutually exclusive to the first, is that the effects capture intermediate internal states that are important to generating the reward response. For example, the activity may reflect processes related to the BMRQ factors, such as musical knowledge, or other factors not directly measured by the BMRQ, like visual imagery or episodic memories evoked by the music (Vuoskoski & Eerola, 2012; Juslin & Västfjäll, 2008). Consistent with this notion, the spatial reliability of the correlation between complexity matching and music reward suggest that frontal regions (superior, middle, inferior frontal) were involved with processing internally generated information patterns.

Integration may be a requirement for the commonly highlighted role of expectancy in music reward. The theory that rewarding emotional responses to music are derived from expectations and anticipation during music listening was first extensively described by Meyer (1956; see also Huron, 2006). It explains that expectations are generated from explicit and implicit knowledge of music structure and patterns, and composers evoke emotional arousal by playing with “tension and release.” Anticipation of a familiar rewarding segment of music has been linked to caudate dopamine release and BOLD activity increases prior to nucleus accumbens dopamine activity at peak reward response (Salimpoor, Benovoy, Larcher, Dagher, & Zatorre, 2011). Music expectations and familiarity with the stimuli were not behaviorally evaluated in this study, but there is a logical link between them and brain signal complexity. Generation of expectations requires sufficient understanding of the structure and patterns in the music. Therefore, it would be expected that enjoyment of a piece of music needs to be associated with a minimum amount of information processing that would allow the listener to appreciate the music as a coherent whole, rather as a sequence of individual notes.

We do note that the songs were not randomized across tasks, which could be a source of confound. We opted to not use a complete random counterbalancing to prevent jarring transitions between songs. Moreover, the average features of the music pieces (e.g., MSE, low-level perceptual features, and high-level compositional features) were not different between tasks. Nevertheless, the lack of complete counterbalancing does add a caution to the interpretation of our results. Future studies where music features most related to induced emotional responses parametrically mapped should be considered.

Acknowledgments

The authors wish to thank Natasa Kovačević for her work on the EEG preprocessing pipeline, as well as essential input at multiple levels of the Matlab scripting process. This work was supported by Canadian NSERC grant RGPIN-2018-04457 (A. R. M.).

Reprint requests should be sent to Anthony R. McIntosh, 3560 Bathurst Street, Rotman Research Institute, Baycrest Centre, University of Toronto, Ontario M6A 2E1, Canada, or via e-mail: rmcintosh@research.baycrest.org.

REFERENCES

Amodio
,
D. M.
, &
Frith
,
C. D.
(
2006
).
Meeting of minds: The medial frontal cortex and social cognition
.
Nature Reviews Neuroscience
,
7
,
268
277
.
Blood
,
A. J.
, &
Zatorre
,
R. J.
(
2001
).
Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion
.
Proceedings of the National Academy of Sciences, U.S.A.
,
98
,
11818
11823
.
Bressler
,
S. L.
, &
Kelso
,
J. A.
(
2001
).
Cortical coordination dynamics and cognition
.
Trends in Cognitive Sciences
,
5
,
26
36
.
Carpentier
,
S. M.
,
Moreno
,
S.
, &
McIntosh
,
A. R.
(
2016
).
Short-term music training enhances complex, distributed neural communication during music and linguistic tasks
.
Journal of Cognitive Neuroscience
,
28
,
1603
1612
.
Catarino
,
A.
,
Churches
,
O.
,
Baron-Cohen
,
S.
,
Andrade
,
A.
, &
Ring
,
H.
(
2011
).
Atypical EEG complexity in autism spectrum conditions: A multiscale entropy analysis
.
Clinical Neurophysiology
,
122
,
2375
2383
.
Costa
,
M.
,
Goldberger
,
A. L.
, &
Peng
,
C. K.
(
2002
).
Multiscale entropy analysis of complex physiologic time series
.
Physical Review Letters
,
89
,
6
9
.
Costa
,
M.
,
Goldberger
,
A. L.
, &
Peng
,
C. K.
(
2005
).
Multiscale entropy analysis of biological signals
.
Physical Review E, Statistical, Nonlinear, and Soft Matter Physics
,
71
,
18
.
Courtiol
,
J.
,
Perdikis
,
D.
,
Petkoski
,
S.
,
Müller
,
V.
,
Huys
,
R.
,
Sleimen-Malkoun
,
R.
, et al
(
2016
).
The multiscale entropy: Guidelines for use and interpretation in brain signal analysis
.
Journal of Neuroscience Methods
,
273
,
175
190
.
Deco
,
G.
,
Jirsa
,
V. K.
, &
McIntosh
,
A. R.
(
2011
).
Emerging concepts for the dynamical organization of resting-state activity in the brain
.
Nature Reviews Neuroscience
,
12
,
43
56
.
Delorme
,
A.
, &
Makeig
,
S.
(
2004
).
EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis
.
Journal of Neuroscience Methods
,
134
,
9
21
.
Denny
,
B. T.
,
Kober
,
H.
,
Wager
,
T. D.
, &
Ochsner
,
K. N.
(
2012
).
A meta-analysis of functional neuroimaging studies of self- and other judgments reveals a spatial gradient for mentalizing in medial prefrontal cortex
.
Journal of Cognitive Neuroscience
,
24
,
1742
1752
.
Desikan
,
R. S.
,
Ségonne
,
F.
,
Fischl
,
B.
,
Quinn
,
B. T.
,
Dickerson
,
B. C.
,
Blacker
,
D.
, et al
(
2006
).
An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest
.
Neuroimage
,
31
,
968
980
.
Efron
,
B.
, &
Tibshirani
,
R.
(
1986
).
Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy
.
Statistical Science
,
1
,
54
75
.
Fujioka
,
T.
,
Trainor
,
L. J.
,
Large
,
E. W.
, &
Ross
,
B.
(
2009
).
Beta and gamma rhythms in human auditory cortex during musical beat processing
.
Annals of the New York Academy of Sciences
,
1169
,
89
92
.
Ghosh
,
A.
,
Rho
,
Y.
,
McIntosh
,
A. R.
,
Kötter
,
R.
, &
Jirsa
,
V. K.
(
2008
).
Noise during rest enables the exploration of the brain's dynamic repertoire
.
PLoS Computational Biology
,
4
,
e1000196
.
Good
,
P.
(
2000
).
Permutation, parametric and bootstrap tests of hypotheses
.
Huntington Beach, CA
:
Springer Science + Business Media, Inc
.
Gower
,
J. C.
(
1975
).
Generalized procrustes analysis
.
Psychometrika
,
40
,
33
51
.
Hagmann
,
P.
,
Cammoun
,
L.
,
Gigandet
,
X.
,
Meuli
,
R.
,
Honey
,
C. J.
,
Wedeen
,
V. J.
, et al
(
2008
).
Mapping the structural core of human cerebral cortex
.
PLoS Biology
,
6
,
e159
.
Heisz
,
J. J.
,
Shedden
,
J. M.
, &
McIntosh
,
A. R.
(
2012
).
Relating brain signal variability to knowledge representation
.
Neuroimage
,
63
,
1384
1392
.
Holmes
,
C. J.
,
Hoge
,
R.
,
Collins
,
D. L.
,
Woods
,
R.
,
Toda
,
A. W.
, &
Evans
,
A. C.
(
1998
).
Enhancement of MR images using registration for signal averaging
.
Journal of Computer Assisted Tomography
,
22
,
324
333
.
Hunter
,
P. G.
, &
Schellenberg
,
E. G.
(
2010
).
Music perception
. In
M. R.
Jones
,
R. R.
Fay
, &
A. N.
Popper
(Eds.),
Springer handbook of auditory research
(pp.
129
164
).
New York
:
Springer
.
Huron
,
D.
(
2006
).
Sweet anticipation: Music and the psychology of expectation
.
Cambridge, MA
:
MIT Press
.
Juslin
,
P. N.
, &
Västfjäll
,
D.
(
2008
).
Emotional responses to music: The need to consider underlying mechanisms
.
Behavioral and Brain Sciences
,
31
,
559
621
.
Krishnan
,
A.
,
Williams
,
L. J.
,
McIntosh
,
A. R.
, &
Abdi
,
H.
(
2011
).
Partial least squares (PLS) methods for neuroimaging: A tutorial and review
.
Neuroimage
,
56
,
455
475
.
Lartillot
,
O.
, &
Toiviainen
,
P.
(
2007
).
A Matlab toolbox for musical feature extraction from audio
. In
C.
Preisach
,
H.
Burkhardt
,
L.
Schmidt-Thieme
, &
R.
Decker
(Eds.),
Data analysis, machine learning and applications: Studies in classification, data analysis, and knowledge organization
(pp.
261
268
).
Berlin, Heidelberg
:
Springer
.
Mas-Herrero
,
E.
,
Marco-Pallares
,
J.
,
Lorenzo-Seva
,
U.
,
Zatorre
,
R. J.
, &
Rodriguez-Fornells
,
A.
(
2013
).
Individual differences in music reward experience
.
Music Perception
,
31
,
118
138
.
McIntosh
,
A. R.
(
2000
).
Towards a network theory of cognition
.
Neural Networks
,
13
,
861
870
.
McIntosh
,
A. R.
,
Bookstein
,
F. L.
,
Haxby
,
J. V.
, &
Grady
,
C. L.
(
1996
).
Spatial pattern analysis of functional brain images using partial least squares
.
Neuroimage
,
3
,
143
157
.
McIntosh
,
A. R.
,
Kovačević
,
N.
, &
Itier
,
R. J.
(
2008
).
Increased brain signal variability accompanies lower behavioral variability in development
.
PLoS Computational Biology
,
4
,
e1000106
.
McIntosh
,
A. R.
, &
Lobaugh
,
N. J.
(
2004
).
Partial least squares analysis of neuroimaging data: Applications and advances
.
Neuroimage
,
23(Suppl. 1)
,
S250
S263
.
Meyer
,
L. B.
(
1956
).
Emotion and meaning in music
.
Chicago
:
The University of Chicago Press
.
Mišić
,
B.
,
Mills
,
T.
,
Taylor
,
M. J.
, &
McIntosh
,
A. R.
(
2010
).
Brain noise is task dependent and region specific
.
Journal of Neurophysiology
,
104
,
2667
2676
.
Ochsner
,
K. N.
,
Knierim
,
K.
,
Ludlow
,
D. H.
,
Hanelin
,
J.
,
Ramachandran
,
T.
,
Glover
,
G.
, et al
(
2004
).
Reflecting upon feelings: An fMRI study of neural systems supporting the attribution of emotion to self and other
.
Journal of Cognitive Neuroscience
,
16
,
1746
1772
.
Pascual-Marqui
,
R. D.
(
2002
).
Standardized low resolution brain electromagnetic tomography (sLORETTA): Technical details
.
Methods and Findings in Experimental and Clinical Pharmacology
,
24(Suppl. D)
,
5
12
.
Richman
,
J. S.
, &
Moorman
,
J. R.
(
2000
).
Physiological time-series analysis using approximate entropy and sample entropy
.
American Journal of Physiology—Heart and Circulatory Physiology
,
278
,
H2039
H2049
.
Salimpoor
,
V. N.
,
Benovoy
,
M.
,
Larcher
,
K.
,
Dagher
,
A.
, &
Zatorre
,
R. J.
(
2011
).
Anatomically distinct dopamine release during anticipation and experience of peak emotion to music
.
Nature Neuroscience
,
14
,
257
262
.
Salimpoor
,
V. N.
,
van den Bosch
,
I.
,
Kovacevic
,
N.
,
McIntosh
,
A. R.
,
Dagher
,
A.
, &
Zatorre
,
R. J.
(
2013
).
Interactions between the nucleus accumbens and auditory cortices predict music reward value
.
Science
,
340
,
216
219
.
Tadel
,
F.
,
Baillet
,
S.
,
Mosher
,
J. C.
,
Pantazis
,
D.
, &
Leahy
,
R. M.
(
2011
).
Brainstorm: A user-friendly application for MEG/EEG analysis
.
Computational Intelligence and Neuroscience
,
2011
,
879716
.
Tononi
,
G.
,
Sporns
,
O.
, &
Edelman
,
G. M.
(
1994
).
A measure for brain complexity: Relating functional segregation and integration in the nervous system
.
Proceedings of the National Academy of Sciences, U.S.A.
,
91
,
5033
5037
.
Tononi
,
G.
,
Sporns
,
O.
, &
Edelman
,
G. M.
(
1996
).
A complexity measure for selective matching of signals by the brain
.
Proceedings of the National Academy of Sciences, U.S.A.
,
93
,
3422
3427
.
van den Heuvel
,
M. P.
, &
Sporns
,
O.
(
2011
).
Rich-club organization of the human connectome
.
Journal of Neuroscience
,
31
,
15775
15786
.
Vuoskoski
,
J. K.
, &
Eerola
,
T.
(
2012
).
Can sad music really make you sad? Indirect measures of affective states induced by music and autobiographical memories
.
Psychology of Aesthetics Creativity and the Arts
,
6
,
204
213
.
Zamora-López
,
G.
,
Zhou
,
C.
, &
Kurths
,
J.
(
2010
).
Cortical hubs form a module for multisensory integration on top of the hierarchy of cortical networks
.
Frontiers in Neuroinformatics
,
4
,
1
.
Zatorre
,
R. J.
,
Belin
,
P.
, &
Penhune
,
V. B.
(
2002
).
Structure and function of auditory cortex: Music and speech
.
Trends in Cognitive Sciences
,
6
,
37
46
.