Task-modulated Sensitivity to Vocal Pitch in the Dorsal Premotor Cortex during Multitalker Speech Recognition

Regions of the left premotor cortex involved in speech production are also activated during listening to speech (Skipper, Devlin, & Lametti, 2017; Buchsbaum et al., 2011; Pulvermuller et al., 2006; Wilson, Saygin, Sereno, & Iacoboni, 2004; Buchsbaum, Hickok, & Humphries, 2001). While numerous authors have suggested this supports a role for motor mechanisms in speech perception (Skipper et al., 2017; Schwartz, Basirat, Ménard, & Sato, 2012; Tremblay & Small, 2011; Pulvermuller & Fadiga, 2010; Rizzolatti & Craighero, 2004), several recent studies show that left premotor cortex responds to auditory speech features including the spectrotemporal envelope, pitch contour, rhythmic phrasal structure, and auditory-phonetic features (Berezutskaya, Baratin, Freudenburg, & Ramsey, 2020; Keitel, Gross, & Kayser, 2018; Cheung, Hamilton, Johnson, & Chang, 2016). This raises the possibility that at least some portion of the left premotor cortex processes speech in auditory rather than (or in addition to) articulatory-phonetic terms. This most likely reflects the fact that, for some speech motor circuits, (i) the sensory “targets” for production (Hickok, Houde, & Rong, 2011; Tourville, Reilly, & Guenther, 2008) are primarily auditory (i.e., as opposed to visual, somatosensory, or multimodal) and (ii) these “targets” can be understood in terms of simple acoustic properties, without requiring a phonological abstraction (Indefrey & Levelt, 2004) or nonlinear transformation to motor coordinates (Parrell, Ramanarayanan, Nagarajan, & Houde, 2019).

Recent studies showing premotor responses to acoustic speech features have consistently identified a more dorsal region of the precentral gyrus adjacent to laryngeal motor cortex (cf. Belyk et al., 2021; Correia, Caballero-Gaudes, Guediche, & Carreiras, 2020; Eichert, Papp, Mars, & Watkins, 2020; Simonyan, 2014; Brown, Ngan, & Liotti, 2008) rather than more ventral precentral regions associated with the orofacial articulators (e.g., lips and tongue). Studies showing motor activation during speech perception tasks, including somatotopic effects and potential contributions to task performance, have focused on these more ventral regions (e.g., lips and tongue; Schomers, Kirilina, Weigand, Bajbouj, & Pulvermuller, 2015; D'Ausilio et al., 2009; Mottonen & Watkins, 2009; Meister, Wilson, Deblieck, Wu, & Iacoboni, 2007; Pulvermuller et al., 2006; Watkins & Paus, 2004; Watkins, Strafella, & Paus, 2003; Fadiga, Craighero, Buccino, & Rizzolatti, 2002). Several groups, including our own, have suggested the orofacial motor regions may be recruited to process speech that is embedded in background noise or otherwise degraded, or when the listening task involves phonological working memory or decision-related processes (Nuttall, Kennedy-Higgins, Hogan, Devlin, & Adank, 2016; Du, Buchsbaum, Grady, & Alain, 2014; Mottonen, van de Ven, & Watkins, 2014; Krieger-Redwood, Gaskell, Lindsay, & Jefferies, 2013; D'Ausilio, Bufalari, Salmas, & Fadiga, 2012; Hervais-Adelman, Carlyon, Johnsrude, & Davis, 2012; Osnes, Hugdahl, & Specht, 2011). In contrast, perhaps a defining feature of the more dorsal premotor site is its tendency to activate during passive listening when no explicit response or motor planning is required (Chen, Penhune, & Zatorre, 2009; Wilson et al., 2004; but see also Panouillères, Boyles, Chesters, Watkins, & Möttönen, 2018) and, as in temporal lobe speech regions, when speech is more rather than less intelligible (Okada et al., 2010). One possibility, as noted above, is that laryngeal motor circuits are modulated by rhythmic and/or prosodic acoustic cues, many of which are associated with voicing and vocal pitch (Dichter, Breshears, Leonard, & Chang, 2018; D'Ausilio, Bufalari, Salmas, Busan, & Fadiga, 2011). Indeed, recent work suggests that articulatory-phonetic information (e.g., place of articulation) is not encoded in dorsal or ventral premotor activations during listening to speech (Arsenault & Buchsbaum, 2016; Evans & Davis, 2015), which supports the notion that these regions function differently when speech is processed in auditory terms. Notably, the acoustic-to-articulatory mapping for pitch is likely much simpler than that for phonetic cues (Parrell et al., 2019), perhaps facilitating auditory–motor interactions in dorsal laryngeal circuits during listening.

Using fMRI, we recently identified a region in the left dorsal premotor cortex (dPM) that responds preferentially to intelligible speech, encodes vocal pitch, is functionally connected with early auditory regions, and is proximal to the dorsal laryngeal motor cortex (Venezia, Richards, & Hickok, 2021). Using a single-talker, yes–no intelligibility judgment task with a spectrotemporal modulation filtering paradigm that independently degraded acoustic features related to phonetic content versus vocal pitch, we further showed that, paradoxically, left dPM and pitch-sensitive regions of auditory cortex (AC) responded to vocal pitch only when speech was rated as unintelligible by listeners. The dominant response in left dPM and pitch-sensitive auditory regions was to phonetic content, which was also strongly associated with intelligibility judgments, whereas vocal pitch was not. Therefore, by definition, activation was driven largely by the categorical difference between intelligible and unintelligible speech. We speculated that left dPM and pitch-sensitive auditory regions responded to pitch only during unintelligible trials because pitch information was irrelevant to the task when phonetic content was also present—namely, because intelligibility judgments relied only on phonetic content and not vocal pitch, brain regions involved in processing vocal pitch activated to pitch-related features only when phonetic content was unavailable to the listener.

This dual response profile, dominated by a response to phonetic content with a secondary response to pitch, was somewhat unsurprising given that previous work, including our own, had shown a similar response profile in left dPM and pitch-sensitive auditory regions (Hamilton, Oganian, Hall, & Chang, 2021; Venezia, Martin, Hickok, & Richards, 2019; Cheung et al., 2016). Nonetheless, we assumed that the response to pitch was a defining characteristic of these regions given that no such response was observed in other temporal and frontal lobe regions that responded robustly to phonetic content. An extension of this assumption, related to the discussion above, is that pitch-sensitive regions in left dPM and AC comprise an auditory–motor network driven primarily by pitch and related low-level auditory cues, whereas other regions in the lateral superior temporal lobe and inferior frontal cortex form a speech-motor network driven primarily by phonetic content and other high-level speech information (e.g., lexical representations). However, an alternative possibility is that the response to vocal pitch in left dPM and/or pitch-sensitive auditory regions is truly secondary, occurring only when phonetic information is unavailable and therefore speech is not processed qua speech. Stated in terms of motor control, this alternative would imply that the auditory-cortex–dPM circuit is dedicated to speech production, as opposed to a more general auditory–vocal control function that supports speech. We hypothesized that for a listening task that requires extraction of vocal pitch, dPM and pitch-sensitive auditory regions would respond to pitch-related features regardless of the presence of phonetic content (i.e., overall speech intelligibility), consistent with a primary or general (although not necessarily exclusive) role in processing pitch and related auditory cues.

Here, we use fMRI to test that hypothesis. We again employ our spectrotemporal modulation filtering technique, termed Auditory Bubbles (Venezia, Hickok, & Richards, 2016), to independently degrade spectrotemporal features related to phonetic content versus vocal pitch. However, whereas our previous study used a single talker in a quiet background, this study uses a competing speech task (Bolia, Nelson, Ericson, & Simpson, 2000) with target female and competing male talkers (see Venezia, Leek, & Lindeman, 2020). The male and female talkers are generated synthetically from the same original waveforms using a vocal gender-morph procedure, thus allowing two conditions: Competing, in which the female and male talkers generate similar utterances that differ by several keywords, and Unison, in which the female and male talkers generate the exact same utterance. The crucial difference is that only the Competing condition requires the listener to segregate the talkers using vocal pitch.

Twenty-five listeners completed 400 trials of a speeded, three-alternative forced-choice (3-AFC) speech recognition task in the Competing and Unison conditions during fMRI scanning. A hierarchical Bayesian drift diffusion model (HDDM; Vandekerckhove, Tuerlinckx, & Lee, 2011) was fitted to RT data to predict trial-by-trial fluctuations in speech recognition performance (the “drift rate” parameter of the HDDM) from the pattern of spectrotemporal distortion imposed via Auditory Bubbles on each trial. The model confirmed that drift rate was more strongly modulated by vocal pitch for Competing versus Unison and phonetic content for Unison versus Competing. A second hierarchical Bayesian model was then used to examine the relation between HDDM-predicted, trial-wise drift rate and trial-wise fMRI activation. Two versions of trial-wise drift rate were generated: overall drift rate and “pitch-restricted” drift rate, where the latter reflected only the pitch-related features that modulated task performance and the former reflected any/all acoustic features (pitch and/or phonetic content) that modulated task performance.

Crucially, trial-wise drift rates represented only the relative presence or absence, on a given trial, of the spectrotemporal modulations associated with drift rate across trials, thus capturing trial-by-trial variance in behaviorally relevant stimulus features. Also crucial was that the majority of trials were intelligible in both the Competing (67% correct) and Unison (> 80% correct) conditions. Therefore, we hypothesized that: (i) overall drift rate, which captures stimulus features related to overall intelligibility, would be correlated with fMRI activation in dPM and auditory speech regions in the Competing and Unison conditions and (ii) pitch-restricted drift rate, which captures stimulus features related to vocal pitch regardless of overall intelligibility, would be correlated with fMRI activation in dPM and pitch-sensitive auditory regions only in the Competing condition. The logic behind the latter hypothesis was that, following our previous work (Venezia et al., 2021), pitch-responsive regions should be activated by pitch only for unintelligible trials in the Unison condition. Because only a small minority of trials (< 20%) were incorrect in the Unison condition, a correlation across all trials between pitch-restricted drift rate and fMRI activation should not be observed in this condition. Conversely, in the Competing condition, we assumed that, given the essential role of pitch in performing the Competing task, pitch-sensitive regions would be activated by pitch for both intelligible and unintelligible trials, and thus, a correlation across all trials between pitch-restricted drift rate and fMRI activation should be observed in this condition.

In fact, Hypothesis (i) was supported by the data for both dPM and auditory-cortical regions. Hypothesis (ii) was supported by the data only for dPM; counter to expectations, activation in pitch-sensitive auditory regions was correlated with pitch-restricted drift rate in the Competing and Unison conditions. We speculate as to the explanation for this below. Post hoc analyses were then conducted to (i) confirm that the observed pattern of results could not be explained by trial-wise fluctuations in task difficulty that might be correlated with trial-wise, pitch-restricted drift rate, and (ii) examine how drift rate correlations differed by response category (correct, incorrect) across conditions (Competing, Unison).

The data in this study were reported previously by Herrera et al. (2021). Some general methodological details of that study are recapitulated here. This study is a re-analysis of the same data to answer a different experimental question.

Participants were previously deployed veterans between the ages of 18 and 60 years, with no restrictions on gender, ethnicity, or race. All participants were screened via phone and/or medical record review to verify: (1) age, (2) history of deployment, (3) no history of auditory conductive pathology including ear infections or ear surgeries, (4) English as a first language, and (5) no history of major neurological disorder such as stroke, tumor, epilepsy, or moderate-to-severe traumatic brain injury. Once screened, potential participants underwent a full in-person audiologic evaluation to verify: (1) air-conduction pure-tone average at 0.5, 1, 2, and 4 kHz was 35 dB HL or better in both ears; (2) air-conduction pure-tone thresholds were no worse than 40 dB HL at any two audiometric frequencies below 8 kHz; (3) air-conduction pure-tone thresholds did not exceed 10 dB HL asymmetry between the ears at more than one audiometric frequency below 4 kHz; and (4) no evidence of conductive pathology. Participants meeting these audiometric criteria can be described as having at least “near normal” hearing—that is, normal hearing in the speech frequency range and no more than mild hearing loss at higher frequencies, without significant inter-ear asymmetries or evidence of damage to the middle ear. In addition, all participants were required to score at least 25 on the Mini-Mental State Examination. No participants were excluded for active or historical psychiatric diagnoses or use of psychotropic medications, so long as these did not preclude the participant from completing the task.

Twenty-five veterans (22 men, 3 women) completed the study. The average age of the participants was 49.4 years (SD = 9.2 years, range = 30–60 years). Sixteen participants had clinically significant symptoms of post-traumatic stress disorder (PTSD) (via medical record review or the PTSD checklist—military version; Wilkins, Lang, & Norman, 2011), eight had a history of mild traumatic brain injury (via medical record review or subject report), and 16 were diagnosed with another psychiatric disorder such as depression, anxiety, or substance use disorder (medical record review). Potential participants with major neurological (e.g., stroke, tumor, epilepsy, or moderate-to-severe brain injuries) or psychiatric (e.g., schizophrenia, antisocial personality disorder, or any disorders resulting in manifestly violent or aggressive behavior) were not enrolled in the study. Notably, the prevalence of psychiatric disorders in our sample (64%) is not drastically different from the prevalence among college-aged adults in the United States (50%; Blanco et al., 2008), who commonly serve as research participants in studies on “neurotypical” brain function.

Our previous study showed that dPM responded significantly to spectrotremporal modulations in the vocal pitch region of the speech modulation power spectrum (MPS; Venezia et al., 2021). Using data from that study, we averaged spectrotemporal receptive field coefficients from each participant (n = 10) across all significant voxels in dPM and all MPS pixels likely to contain vocal pitch. The standardized effect size (one-sample t test) was 1.15. We then simulated 10,000 data sets in which each “participant's” dPM response was drawn from a normal distribution with mean = 1.15 and standard deviation = 1. For different sample sizes, we calculated the proportion of simulated data sets for which the Bayes Factor (one-sample t test; prior width = 0.707) was greater than 100 (see Trial-Wise Predicted Drift Rate section below). A sample size of at least n = 23 was required to achieve 80% power. Our sample size is n = 25 (estimated power = 89.7%).

This study, all documentation, and procedures were reviewed and approved by the VA Loma Linda Healthcare System (VALLHCS) institutional review board. Before enrollment, all participants provided their written informed consent. All study procedures were completed at either the VALLHCS Auditory Research Lab or at the Loma Linda University East Campus Radiology Department. All participants received monetary compensation for their participation.

Audiologic case history, otoscopy, middle ear immittance testing including tympanometry and ipsilateral acoustic reflex testing at 1 kHz, air-conduction thresholds from 0.25 to 8 kHz, and bone conduction thresholds from 0.25 to 4 kHz were acquired. Equipment for testing included the Madsen Astera2 audiometer by Natus, Inc. for audiometric thresholds and the Tympstar from Grason-Stadler Inc. for immittance testing. Conductive pathology was ruled out combining tympanometry, acoustic reflex testing, and restricting audiometric air- and bone-conduction gaps to 10 dB or less at a given frequency.

Items from the Coordinate Response Measure (CRM) speech corpus were obtained from source audio files (20-kHz sampling rate, 16-bit quantization) with a single male talker (Talker 0). Two versions of the items were derived from these audio files using a vocal-gender shifting procedure described by Venezia et al. (2020): (1) items spoken by a synthetic female talker with a mean fundamental frequency scaled by 1.7 and vocal tract length scaled by 0.84 relative to the original values, and (2) items spoken by the original (male) talker processed through the gender shift algorithm with mean fundamental frequency and vocal tract length each scaled by 1.0. Vocal-gender shifting was accomplished using the pitch-synchronous overlap-add procedure as implemented in Praat (Boersma, 2002), ensuring that synthetic female and male talkers would differ only in fundamental frequency and vocal tract length and not by other talker-specific speech patterns (i.e., speech rate or intonation contour). The difference in vocal pitch between the synthetic female and male talkers can be visualized on the MPS (Figure 1), which is obtained as the modulus of the 2-D Fourier transform of the log-magnitude speech spectrogram. On the MPS, the synthetic female talker appears at lower spectral modulation rates (higher pitch; ∼5–7.5 cyc/kHz) and the male talker appears at higher spectral modulation rates (lower pitch; ∼10–12.5 cyc/kHz). Phonetic speech content (formant-scale spectral components) shared across the two talkers appears at the lowest spectral modulation rates (< 2.5 cyc/kHz). Figure 1 can thus be used as a reference to the acoustic feature space underlying later plots showing the relative contributions of these features to task performance (see Figure 3); the figure further shows that the long-term average MPS is well matched across conditions (Unison, Competing).

In the CRM corpus, each item takes the form “Ready [call sign] go to [color] [number] now.” There are eight possible call signs (arrow, baron, charlie, eagle, hopper, laker, ringo, and tiger), four possible colors (blue, green, red, and white), and eight possible numbers (1–8), yielding 256 possible items. The color and the number are the targets for speech recognition, yielding 32 possible color/number targets. Here, we present two-talker (synthetic female and male) mixtures at a + 3-dB target-to-masker ratio favoring the female talker. We further employ two conditions: Competing, in which the synthetic female and male talkers utter sentences with different call signs, colors, and numbers; and Unison, in which the synthetic female and male talkers utter sentences with the exact same call sign, color, and number (i.e., identical waveforms aside from the vocal gender-shift procedure). Listeners are asked to identify the color/number spoken by the female talker in the Competing condition and either/both talkers in the Unison condition. Crucially, only the Competing condition requires segregation of the target (synthetic female) talker from a competing (male) talker.

During fMRI scanning, two-talker mixtures in the Unison and Competing conditions were further modified through a spectrotemporal modulation filtering procedure called Auditory Bubbles (Venezia et al., 2016). Briefly, in Auditory Bubbles, a quasirandomly generated, binary multiplicative filter is applied to the MPS of a two-talker mixture such that some spectrotemporal patterns are retained in the signal whereas others are removed. Here, filters were applied only to MPS regions with spectral modulations ≤ 15 cyc/kHz and temporal modulations ≤ 50 Hz. The bubbles filters were created by: (1) generating an all-zero image of dimensions equal to the MPS, (2) setting a randomly-selected subset of pixels to a value of one (where the number of pixels is the number of “bubbles” in the filter), (3) applying a 2-D Gaussian filter with standard deviation 0.5 cyc/kHz in the spectral modulation dimension and 2 Hz in the temporal modulation dimension, and (4) binarizing the resulting image with a cutoff value of 0.1. The filters were applied to individual CRM mixtures as follows: (1) obtain a log-magnitude spectrogram with Gaussian windows, (2) perform a two-dimensional fast Fourier transform on the spectrogram, (3) multiply the bubbles filter with the modulus of the 2-D fast Fourier transform, (4) invert the modified 2-D Fourier spectrum to generate a filtered magnitude spectrogram, and (5) invert the magnitude spectrogram via phase gradient heap integration to obtain a filtered waveform (Venezia et al., 2020).

In the scanner, each listener performed a modified 3-AFC version of the CRM task. After hearing a filtered two-talker mixture (see the sections Speech Stimuli and Spectrotemporal Modulation Filtering), listeners were presented immediately upon sound offset with three possible color/number targets displayed on a computer screen in a horizontal array (left, center, right) as filled squares (blue, green, red, or white) with a single digit in the center (1–8 in black font) with a uniform gray background. Of the three possible responses, one was always the color/number spoken by the target female talker. In the Competing condition, a second color/number was that spoken by the male talker and the third color/number was selected from one of five categories with equal probability: (i) same color as the female talker; (ii) same color as the male talker; (iii) same number as the female talker; (iv) same number as the male talker; or (v) neither the same color nor the same number as the female/male talkers. In the Unison condition, the second alternative was selected as if the male talker had uttered a different color/number than the female talker and the third alternative was chosen as described for the third alternative in the Competing condition.

Each trial was synchronized to the onset of an image acquisition period (repetition time [TR]). The auditory stimulus was presented 200 msec after TR onset and lasted 1.96 sec. The participant then had 2.5 sec to choose a response via button pad with the dominant hand. The next trial began 140 msec after the response period, yielding a total trial duration of 4.8 sec (four TRs). The correctness of the response and the RT were logged for each trial. Each listener completed 400 trials in the Competing condition on a first day of scanning and 400 trials in the Unison condition on a second day of scanning. One listener completed the Competing and then the Unison trials in separate sessions on the same day with a 2-hr break in between.

Listeners completed eight blocks of 50 trials in each condition with a break at the end of each block. In the Competing condition, the number of bubbles (see the section Spectrotemporal Modulation Filtering) applied on each trial was adjusted using a weighted staircase procedure (Kaernbach, 1991) targeting 66.67% correct performance. Two interleaved staircases with starting values of 50 and 150 bubbles, respectively, proceeded without being restarted at the beginning of each block. The number of bubbles was decreased by five (more acoustic distortion) with a correct response and increased by 10 (less acoustic distortion) with an incorrect response. In the Unison condition, the number of bubbles was set to exactly match the trial-by-trial trajectory in the Competing condition, thus ensuring an equivalent amount of overall acoustic distortion between the conditions. However, given the Unison task is easier than the Competing task at baseline, this also yielded better overall performance in the Unison condition (mean = 84.3% correct, SD = 4.4%) than the Competing condition (mean = 66.8% correct, SD = 1.3%).

At the scanner, speech signals were delivered diotically via an external sound card (Scarlett 2i2, Focusrite, Inc.), amplifier (AMP100VS, AudioSource, Inc.), and MRI-compatible insert earphones (Model S14, Sensimetrics, Inc.). Each stimulus was upsampled to 44100 Hz, processed with a digital equalization filter provided by Sensimetrics to produce a flat frequency response, attenuated to an overall level of 85 dB SPL, and gated on and off with a 10-msec raised cosine ramp. Listeners were fitted with isolating foam eartips (Comply Canal Tips, Hearing Components Inc.) and silicone-backed foam earmuffs (Hibermate, Inc.) or custom, low-profile earmuffs (foam backed with engineered wood). Visual stimuli were presented on an MRI-compatible digital display (SensaVue, Philips, Inc.) and viewed on an angled-front-surface mirror mounted on the head coil. Responses were collected with a four-button MRI-compatible response pad (LS-LINE, Cedrus, Inc.) and button response unit (Lumina, Cedrus Inc.). Psychtoolbox v3 (Kleiner et al., 2007) was used to control stimulus presentation, timing, scanner synchronization, and delivery of instructions.

Each listener also completed a practice session outside the scanner including verbal and written instructions, a single CRM trial with no bubbles and an unlimited response period, 10 CRM trials with no bubbles and a 2.5-sec response period, and 50 trials otherwise identical to the trial blocks performed inside the scanner including use of the same adaptive staircase procedure described above for the in-scanner task. Recorded MRI-scanner noise was played in the background to simulate the acoustic environment in the scanner. At the practice session, acoustic stimuli were delivered diotically via the internal sound card (C610/X99 chipset, Intel, Inc.) of a Linux desktop (Silverback Workstation, System76, Inc.) to insert earphones (ER-4, Etymotic, Inc.). Visual stimuli were displayed on an LCD monitor (VX2457-MHD, ViewSonic, Inc.), and responses were collected via mechanical keyboard (Dell, Inc.). The practice trials were controlled via Psychtoolbox v3.

Images were acquired on a 3 T MAGNETOM Skyra (Siemens, Inc.) with a 32-channel head coil located at the Loma Linda University East Campus Radiology Center. At each scan session, an anatomical scout, T1 anatomical volume, field map scan, and eight runs of BOLD functional volumes were acquired in that order. The functional volumes were acquired with a simultaneous multislice (SMS) EPI sequence without partial Fourier or in-plain acceleration (Smith et al., 2013) and with a 10% slice gap to minimize interslice signal leakage (Cauley, Polimeni, Bhat, Wald, & Setsompop, 2014); there were 56 axial slices oriented parallel to the anterior commissure-posterior commissure axis with interleaved slice order, multiband acceleration factor = 4, TR = 1.2 sec, echo time (TE) = 33 msec, flip angle = 45°, echo spacing = 0.61 msec, phase encoding direction = anterior-posterior, bandwidth = 2055 Hz/Px, voxel size = 2.5 × 2.5 × 2.75 mm (including 10% gap), and matrix = 84 × 84. In each functional run, 210 volumes were acquired to allow for 50 CRM trials (four volumes per trial) plus 10 additional volumes to capture the hemodynamic response to the final trials. Thus, each scan session produced 1680 functional volumes (400 total CRM trials). The field map scan parameters were TR = 0.557 sec, TE1 = 4.92 msec, TE2 = 7.38 msec, flip angle = 60°, and bandwidth = 510 Hz/Px, with the same slice prescription, slice thickness, voxel size, and matrix size as the functional volumes. The T1 anatomical volume was acquired with a magnetization prepared rapid gradient echo sequence with 176 sagittal slices, TR = 1.95 sec, TE = 2.32 msec, TI = 917 msec, flip angle = 8°, generalized autocalibrating partially parallel acquisitions acceleration factor = 2, bandwidth = 200 Hz/Px, voxel size = 0.9 × 0.938 × 0.938 mm, and matrix = 256 × 256.

Automated cortical surface reconstruction from the T1 anatomical volume collected during the Competing scan session was performed in Freesurfer v6.0 (https://surfer.nmr.mgh.harvard.edu). Right and left hemisphere cortical surface meshes were then converted to AFNI/SUMA (https://afni.nimh.nih.gov/Suma) format, co-registered to the native-space T1 volume, resampled to a standard topology via linear icosahedral tessellation with 64 edge divides, and merged into a single surface containing 81,924 nodes using the prep_afni_surf.py function of the surfing toolbox v0.6 (https://github.com/nno/surfing; Oosterhof, Wiestler, Downing, & Diedrichsen, 2011). A standard topology surface mesh was also generated from the Colin 27 template brain. The Unison T1 volume was aligned to the Competing T1 volume using a six-parameter rigid body transformation (antsRegsistrationSyn.sh) in ANTs v 3.0.0 (https://stnava.github.io/ANTs/). The preprocessed functional volumes from the Unison scan session were brought into alignment with the Competing data using the Unison-to-Competing T1 transformation matrix, and then co-registered to the Competing T1 volume to allow mapping to the standard topology cortical surface.

The images from the Competing and Unison scan sessions were otherwise preprocessed identically. Following conversion to NIFTI format using dcm2niix v1.0.2 (https://github.com/rordenlab/dcm2niix), images were slice-timing corrected in SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12/) using the nii_SliceTime.m MATLAB script (https://www.mccauslandcenter.sc.edu/crnl/tools/stc). Motion and distortion (field map) correction were then performed in SPM12 via the CONN toolbox (RRID:SCR_009550 www.nitrc.org/projects/conn) release 18.b (Whitfield-Gabrieli & Nieto-Castanon, 2012). Automated tissue segmentation of the Competing T1 volume was performed in CONN, and the Unison data were aligned with the Competing data as described above. Co-registration (12-parameter affine) to the Competing T1 volume was then performed in SPM12 via CONN followed by regression-based denoising with quadratic detrending and the following covariates: six parameter rigid body motion-correction parameters and their temporal derivatives, five principle components extracted from the signals within each of the white matter and cerebrospinal fluid regions of interest defined during tissue segmentation, and scrubbing covariates determined from the CONN Artifact Rejection Tool with the liberal setting. Next, the preprocessed functional data were mapped to the standard topology surface mesh and smoothed to a target level of 4 mm FWHM using AFNI v19.0 (https://afni.nimh.nih.gov/). Finally, trial-wise beta activation time series were generated separately for the Unison and Competing conditions using the Least Squares-Separate (LSS; Mumford, Turner, Ashby, & Poldrack, 2012) algorithm as implemented in AFNI (3dLSS). The final product was two length-400 time series volumes in the cortical surface domain, one for Unison and one for Competing.

The Wiener diffusion model, also known as the drift diffusion model, aims to characterize the processes underlying speeded binary decisions (e.g., a 2-AFC task). In a binary choice task, the model assumes that evidence is accumulated over time until a decision is made when an upper (Choice 1) or lower (Choice 2) decision boundary is reached. Two-choice RTs can thus be modeled using four parameters (Figure 2, left): β, the initial bias toward the upper or lower boundary; τ, the non-decision time capturing basic sensorimotor processes unrelated to the decision itself; α, the degree of separation between the upper and lower boundaries; and δ, the drift rate or the average rate and direction (i.e., slope) of evidence accumulation. The drift rate, δ, is somewhat analogous to the signal detection measure, d′, whereas α and β are somewhat analogous to the signal detection measures of decision criterion and response bias, respectively. In the context of speeded responses, α reflects the speed-accuracy tradeoff, where larger α will lead to more accurate but slower responding. Notably, the model is extensible to speeded decision tasks with more than two choices if the responses are reduced to two alternatives, namely, “correct” and “incorrect.” Referred to as accuracy coding, this will be employed in the present study to model 3-AFC responses (cf. Muraskin et al., 2018). In general, it is valid to employ accuracy coding with more than two alternatives when it can be assumed that the same processes underlie the different responses and when performance (RT, accuracy) is similar across responses (Voss, Nagler, & Lerche, 2013). In our case, the responses options are defined by location on the screen (left, center, right), and there is no reason to expect location to influence speech processing. Even if we assume a difference in difficulty between responses (e.g., center easier than left/right) or a prepotent response bias (preferred location-button mapping), it seems sensible to “average out” such effects by accuracy coding the responses. Moreover, it is useful here to recall that target responses and each type of nontarget response (counterbalanced across trials) were equally likely to appear in each response position. Thus, we argue that accuracy coding is justifiable for the present experiment. In the accuracy-coded model, the β parameter must be fixed to 0.5 (midway between the upper and lower boundaries; Voss, Voss, & Lerche, 2015).

Our goal is to predict the HDDM parameters, namely, α and δ, from trial-wise patterns of idiosyncratic acoustic distortion introduced by Auditory Bubbles (see the Spectrotemporal Modulation Filtering section; Figure 2, “HDDM”). The result is a “classification image” (Venezia et al., 2016, 2020; Venezia, Martin, et al., 2019) for each parameter showing how the parameter changes with the relative presence or absence of acoustic energy at different locations in the MPS (Figure 2, “Drift Rate Classification Image”; see also Figure 1). We produce these classification images separately in the Unison and Competing conditions. We then later use trial-wise posterior predictions of the HDDM parameters to predict trial-wise fMRI activation time series in the Unison and Competing conditions (see the sections Image Preprocessing and Regression of Trial-Wise Drift Rate on Trial-Wise Brain Response). To ease the computational burden of generating HDDM-based classification images, a set of 48 continuous “bubbles” predictors was generated by down-sampling and vectorizing the binary, 2-D bubbles filters presented to each listener on each trial. Before vectorizing, the down-sampled bubbles filters were 12 pixels in the spectral modulation dimension (0–12 cyc/kHz, 1 cyc/kHz/pixel) and 4 pixels in the temporal modulation dimension (0–20 Hz, 5 Hz/pixel). Below, we describe individual HDDMs using R model notation shorthand, wherein fixed and random effects are specifically enumerated; we refer to the length-48 bubbles feature vector as “Bub.”

In addition to the full model, “drift rate” and “threshold” models were generated in which only the drift rate and threshold parameters, respectively, were allowed to be modulated by bubbles features. This allowed for the selection of the most parsimonious model via formal model comparison. The “drift rate” model was generated according to Equation 2, but the α₀ term was removed from the model entirely. Thus, a bubbles classification image was only generated for the δ parameter. Similarly, a “threshold model” was generated according to Equation 2, but the Cond:Bub term was removed from the δ component and the fixed and random intercepts were suppressed such that the remaining Cond term was estimated as condition-specific intercepts for Unison and Competing; the Cauchy(0, 5) prior was applied to the fixed intercept for both conditions. Thus, in the threshold model, a bubbles classification image was only generated for the α parameter. Finally, a “null model” was generated according to Equation 2, but here, the α₀ term was removed and the Cond:Bub term was removed from the δ component as just described for the drift rate and threshold models. Thus, the null model estimated only condition means for the δ and α parameters and did not allow these parameters to vary from trial to trial according to the acoustic distortion patterns imposed by bubbles filtering. The null model was included to test whether inclusion of bubbles predictor in the full, drift rate, and threshold models considerably improved the model fit relative to a means-only model.

Model fitting was carried out in R using brms with the cmdstanr backend (v0.3.0.9000) and CmdStan v2.26.1 (Carpenter et al., 2017). Model comparison was carried out in R using brms to call the loo package v2.3.1 (Vehtari, Gelman, & Gabry, 2017). For all models (full, drift rate, threshold, null), the No-U-Turn sampling procedure (Hoffman & Gelman, 2014) was used to estimate the posterior distribution of model parameters, and 10 independent sampling chains were obtained with 1000 warmup samples and 3000 postwarmup samples, resulting in 30,000 total draws from the posterior. Such a high number of posterior samples was obtained because of high autocorrelation in the sampling chains for a subset of the parameters (in particular, fixed effects on α₀ and τ). The values of adapt_delta and max_treedepth were set to 0.99 and 12, respectively, and all parameters were initialized to a randomly selected value within a manually defined range optimized for each individual parameter. Across all models, no divergent transitions were observed and no samples exceeded the maximum tree depth. Following Houpt and Bittner (2018), two criteria were used to ensure the samples adequately represented the posterior distribution: (1) Visual checks indicated good convergence and good mixing, and the Gelman-Rubin $Rˆ$ was less than 1.01 for all parameters; and (2) the smallest bulk and tail effective sample sizes (Carpenter et al., 2017) across all parameters were still adequately large to characterize the posterior density.

The best fitting model was determined via approximate leave-one-out (LOO) cross-validation with exact LOO subsampling (Magnusson, Vehtari, Jonasson, & Andersen, 2020). A random sample of 1000 trials was drawn for LOO subsampling, and the same trial indices were maintained across all models to ensure a fair comparison. This method identifies the best fitting model as that with the maximum expected log predictive density (elpd_loo), which is to say the model with the best predictive performance and greatest ability to generalize to new, unseen data. A given model was taken to have significantly better predictive performance if its elpd_loo exceeded that of another model by more than four standard errors of the estimated difference in elpd_loo.

Upon model comparison, the drift rate model was identified as the best model considering the combination of predictive performance and number of free parameters. An estimate of drift rate for each trial in each condition and each listener was then obtained from the model as the mean of the posterior samples of the expected value of the posterior predictive distribution (Figure 2, “Posterior Predict”). Notably, the only predictors that would cause the predicted drift rate to vary from trial to trial were the elements of the Cond:Bub term (Equation 2, δ), which captures the effect of the 48 “bubbles” acoustic distortion features (see section Behavioral Analysis: Bayesian Hierarchical Drift Diffusion Models) separately for each condition. In other words, the predicted drift rate reflects the relative presence or absence of behaviorally relevant stimulus features. A second drift rate estimate was then obtained in the same way except that all bubbles features other than those associated with the pitch of the target (female) talker were “masked” (set to all zero in the design matrix) from the analysis. We refer to this as the “pitch-restricted” drift rate, which reflects only those components of the drift rate that were modulated by the relative presence or absence of acoustic cues related to the vocal pitch of the female talker.

Trials were split by response category (resp; correct/incorrect) because it was expected that the relation between pitch-restricted drift rate and DP would be moderated by response category. Namely, we expected difficulty to increase with increasing pitch-restricted drift rate for incorrect trials and decrease with increasing pitch-restricted drift rate for correct trials, particularly in the Competing condition. The logic here is that speech on incorrect trials was likely to be heavily distorted, so absence of pitch (low pitch-restricted drift rate) on these trials would result in random guessing (low RT and thus low DP) because of a failure to segregate the already-distorted talkers, whereas presence of pitch (high pitch-restricted drift rate) would result in informed guessing (higher RT and thus higher DP). On the other hand, on correct trials for which speech was likely less distorted, presence of pitch would enable easier identification of the target talker (lower RT and thus lower DP), whereas absence of pitch would make it more difficult to identify the target talker (higher RT and thus higher DP). A similar effect might be expected in the Unison condition wherein presence of pitch would modulate DP when speech is heavily distorted (incorrect trials) but not when speech is clear (correct trials).

As such, if response category moderates the effect of pitch-restricted drift rate on DP, and if DP mediates the effect of pitch-restricted drift rate on fMRI response amplitude, then we expect a moderated mediation effect. The posterior distribution of the moderated mediation effect was obtained following Wang and Preacher (2015; their Equation 4). Briefly, the indirect (moderated mediation) effect of pitch-restricted drift rate on fMRI activation (i.e., that mediated by DP) is obtained as the product of the resp * dr effect on DP and the resp * DP effect on y in Equation 5. The direct effect of pitch-restricted drift rate on fMRI activation is obtained as the resp * dr effect on y in Equation 5. The moderated mediation was considered significant if the posterior 95% credible interval of the indirect effect did not contain zero. In the case of a significant mediation, a point estimate (posterior median) of the proportion of the effect mediated (indirect effect divided by the sum of the direct and indirect effects) was also obtained.

Briefly, the primary goal of HDDM analysis was to determine which drift diffusion model parameters, if any, covaried with trial-by-trial distortion in the acoustic speech signal as characterized by the shape of spectrotemporal modulation filters generated quasirandomly on each trial (see section Behavioral Analysis: Bayesian Hierarchical Drift Diffusion Models). Table 1 shows the results of model comparisons for HDDMs in which the drift rate and threshold parameters (full model), drift rate parameter only (drift rate model), threshold parameter only (threshold model), or no parameters (null model) were allowed to be modulated by the acoustic distortion predictors. If the drift rate and/or threshold parameters covary significantly with acoustic distortion, then at least one of the full, drift rate, or threshold models should outperform the null model. Ultimately, the full model had the highest elpd_loo, which means that it was best able to predict trial-by-trial, accuracy-coded RTs in an LOO cross-validation procedure (i.e., on unseen data). In Table 1, the elpd_loo of the best (full) model is listed as zero, and the elpd_loo of the remaining models is described relative to the best model (elpd_diff; best model minus comparison model). For a given comparison model, the full model is taken to generate significantly better predictions on unseen data when the elpd_diff is more than 4 times its standard error. Thus, the full model made significantly better predictions than the remaining three models. However, inspection of model-predictive performance by condition (Unison, Competing) revealed that the difference between the full model and the drift rate model was driven almost entirely by the Unison condition (elpd_diff ∼ 140) as opposed to the Competing condition (elpd_diff ∼ 10). Thus, we proceeded with the simpler drift rate model whose predictions were significantly better than the threshold and null models in both conditions (all condition-wise elpd_diff > 900, all total elpd_diff > 20 SE). The trial-wise posterior mean estimates of drift rate were either perfectly correlated (Competing, r = 1) or nearly perfectly correlated (Unison, r = .9997) between the full and drift rate models, indicating that our subsequent analyses would be unaffected by choosing the drift rate model over the full model.

To examine precisely which acoustic speech features modulated the drift rate parameter, we examine the fixed-effects regression coefficients for the acoustic distortion predictors (Equation 2, δ: Cond:Bub). Recall, these predictors were defined in the domain of the two-dimensional MPS (Figure 1), with 48 predictors spanning temporal modulations from 0 to 20 Hz in 5-Hz steps and spectral modulations from 0 to 12 cyc/kHz in 1 cyc/kHz steps. The regression coefficients thus define a “classification image” showing the relative direction and magnitude of the effect that filtering these discrete segments of the MPS exerted on the drift rate parameter. Large positive drift rates correspond to fast, accurate responses, and small positive drift rates correspond to slow, accurate responses. Correspondingly, large negative drift rates correspond to fast, inaccurate responses and small negative drift rates correspond to slow, inaccurate responses. Therefore, a positive regression coefficient suggests that the presence of a given MPS segment leads to more positive drift rates (better performance) and the absence of that segment leads to more negative drift rates (worse performance). Correspondingly, a negative regression coefficient suggests that the presence (absence) of a given MPS segment leads to worse (better) performance. For visualization in the figures that follow, the drift rate classification images were upsampled from 4 × 12 in the acoustic predictor space to 40 × 121, the original dimensions of the MPS spanning 0–20 Hz and 0–12 cyc/kHz.

Figure 3 plots the posterior predictive distribution of the drift rate parameter (A) along with drift rate classification images in the Competing (B) and Unison (C) conditions, and finally a “difference classification image” showing the contrast “Competing minus Unison” (D). Each of the classification images is shown unthresholded on the left and thresholded on the right such that pixels are colored white when the 95% highest density interval of the regression coefficient contains zero; otherwise, the color scale reflects the posterior mean of the regression coefficient. In Figure 3A, the drift rate distributions reflect all trials across all participants, but the participant-specific means have been subtracted and the grand mean added back into the data; thus, the distributions reflect trends in within-participant spread around the grand mean. From Figure 3A, we see that the distribution of overall drift rate (blue) was broad in both conditions (Unison, Competing), reflecting a range of performance levels across trials, but centered on more positive drift rates overall in the Unison condition, which is the expected pattern given the Unison task was easier overall. Furthermore, we see that the distribution of pitch-restricted drift rate (red) was narrower in both conditions, reflecting the fact that pitch-restricted drift rate is only a subcomponent of the overall drift rate. However, the distribution of pitch-restricted drift rate was especially narrow in the Unison condition, which reflects the fact that pitch contributed relatively little to performance in the Unison condition, as will become apparent from Figure 3B–D.

From Figure 3B, we see in the Competing condition that MPS regions associated with shared phonetic content (blue bar, ordinate) and vocal pitch of the target talker (green bar, ordinate) made strong, significant positive contributions to drift rate. The vocal pitch of the competing talker (red bar, ordinate) and a subset of the phonetic content region at higher spectral modulation rates (> 1 cyc/kHz) made significant, but relatively small, negative contributions to drift rate, meaning that presence of this acoustic information in the signal tended to disrupt performance. From Figure 3C, we see in the Unison condition that, again, MPS regions associated with shared phonetic content and vocal pitch of the target talker made significant positive contributions to drift rate, but the magnitude of the pitch-related contributions was far lower than in the Competing condition; no MPS regions made significant negative contributions to drift rate. Figure 3D, the direct contrast between conditions, confirms that the vocal pitch of the target talker made a significantly larger, positive contribution to drift rate in the Competing condition than the Unison condition, reflected by a significant positive contrast for Competing versus Unison. Conversely, shared phonetic content made a significantly larger, positive contribution to drift rate in the Unison condition than the Competing condition, reflected by a significant negative contrast for Competing versus Unison. Finally, the MPS regions that made significant negative contributions to drift rate in the Competing condition but not the Unison condition—vocal pitch of the competing talker and shared phonetic content at higher spectral modulation rates—were also significantly negative in the direct contrast.

Together, these results suggest: (i) vocal pitch of the target talker contributed relatively more to performance in Competing than Unison; (ii) shared phonetic content contributed relatively more to performance in Unison than Competing; and (iii) interference from the competing talker was observed in Competing but not Unison. Crucially, posterior predictions of pitch-restricted drift rate were generated from the within-condition classification images using only MPS regions associated with the vocal pitch of the target talker that were significantly larger in the Competing condition than the Unison condition—that is, those regions for which the presence of target-talker vocal pitch allowed participants to perform better in the Competing condition by perceptually segregating the two talkers, whereas such segregation was not necessary to perform well in the Unison condition. Therefore, the range of predicted pitch-restricted drift rates was relatively narrow across trials in the Unison condition (Figure 3A). On the other hand, posterior predictions of overall drift rate were generated from the entire within-condition classification images, thus reflecting the totality of the influence that different MPS regions exerted on performance in each condition. It is also crucial to bear in mind that predicted drift rates reflect information about the stimulus features available to the listener on a given trial, subject to the constraint that such features are modified by a “behavioral relevance” filter—in concrete terms, predicted drift rates are the product of trial-wise bubbles filters with the drift rate classification images plotted in Figure 3B–C (left column).

The key aim of this study was to map out brain regions for which posterior predictions of trial-wise drift rate (overall and pitch-restricted) were correlated with trial-wise fMRI activation, separately for each condition. This was accomplished via a Bayesian hierarchical linear model relating trial-wise predicted drift rate to trial-wise activation. We refer to the model as a brain-behavior-stimulus model because it maps the relation of behaviorally relevant stimulus features (predicted drift rates) to fMRI activation. The brain-behavior-stimulus model was fitted to all trials from all participants, separately for each cortical surface node, and included fixed and random effect terms. Recalling from the Introduction, we predicted that: (i) overall drift rate would be correlated with fMRI activation in dPM and temporal lobe speech regions in the Competing and Unison conditions and (ii) pitch-restricted drift rate would be correlated with fMRI activation in dPM and temporal regions sensitive to pitch and/or voice only in the Competing condition.

Figure 4A shows the results of the brain-behavior-stimulus model using the overall drift rate as the predictor. In both the Unison and Competing conditions, there was a significant (BF₁₀ > 100) positive correlation between overall drift rate and the brain response in a bilateral network including early supratemporal auditory regions (Heschl's gyrus and surrounds); superior temporal gyrus; anterior temporal lobe; parieto-occipital junction; midline frontal, parietal, and occipital regions; and peri-central (pre-)motor and somatosensory regions extending into the parietal cortex (Table 2 gives the Montreal Neurological Institute [MNI] coordinates of cluster peak correlations with drift rate). Positive correlations in midline areas were generally stronger and more widespread in the Unison condition, as were positive correlations in the parieto-occipital junction including the angular gyrus. Positive correlations in AC were generally stronger and more widespread in the Competing condition, with significant correlations spreading into the posterior and anterior superior temporal gyrus/sulcus and the posterior middle temporal gyrus. Significant positive correlations also spanned more ventrally in the left speech-premotor cortex in the Competing condition, and, notably, positive correlations in the midline occipital cortex extended into the calcarine sulcus in the Competing condition but not the Unison condition. Finally, significant positive correlations were observed in left dPM and temporal lobe speech regions in both conditions, confirming our Hypothesis (i). In the Competing condition, the peak positive correlation in the motor cortex was located in left dPM.

A large network of areas with significant negative correlations between overall drift rate and the brain response was observed in both conditions. This included bilateral middle and inferior frontal regions, inferior fronto-opercular regions, inferior parietal cortex, planum temporale, and supplementary motor area (Table 2). Negative correlations within this network were generally stronger and more widespread in the Unison condition. We have previously referred to this network broadly as the “distortion network” because it tends to respond in situations that require greater listening effort (Herrera et al., 2021), consistent with the negative drift rate correlation observed here.

Figure 4B shows the results of the brain-behavior-stimulus model using the pitch-restricted drift rate as the predictor. In both conditions, there was a significant positive correlation between pitch-restricted drift rate and the brain response in anterolateral Heschl's gyrus and the immediately surrounding superior temporal gyrus. This effect tended to be stronger and more widespread in the Competing condition. A significant positive correlation was also observed in bilateral calcarine sulcus, but only in the Competing condition. Significant negative correlations were observed within components of the “distortion network,” but only in the Competing condition. Crucially, a significant positive correlation between pitch-restricted drift rate and the brain response was observed in left dPM, but only in the Competing condition. Thus, our Hypothesis (ii) was confirmed for left dPM but not temporal lobe areas sensitive to pitch, which demonstrated significant positive correlations with pitch-restricted drift rates in both conditions. Although we did not perform a direct contrast between correlations in the Competing and Unison conditions, the geometric mean of BF₁₀ in the region of left dPM for which a significant correlation with pitch-restricted drift rate was observed (Figure 4B, Competing) was 254 in the Competing condition, suggesting strong evidence for the presence of a correlation, and 0.22 in the Unison condition, suggesting modest evidence against the presence of a correlation (i.e., a Bayes Factor of 4.5 in favor of a null effect).

As we have described, the brain-behavior-stimulus model is designed to identify brain regions that respond to behaviorally relevant stimulus features. Indeed, the predictor in the model, trial-wise drift rate, reflects primarily those stimulus features that are heavily weighted in the drift rate classification images produced by the HDDM (Figure 3B/C). This is perhaps especially true for pitch-restricted drift rate, which further ignores stimulus features related to overall speech intelligibility (shared phonetic content) and focuses only on the relative presence or absence of vocal pitch on a given trial. However, it remains possible that pitch-restricted drift rate is correlated with task difficulty across trials, particularly in the Competing condition where pitch makes a strong contribution to task performance (Figure 3D). To determine whether task difficulty mediated the effects of pitch-restricted drift rate in the brain-behavior-stimulus model (Figure 4B), we conducted a post hoc mediation analysis as described in section Post-Hoc Analysis: Effect of Trial-Wise Difficulty. The analysis focused on three ROIs identified by the brain-behavior-stimulus model of pitch-restricted drift rate in the Competing condition: (i) bilateral AC, (ii) left dPM, and (iii) left middle pFC. These ROIs were selected because they appeared in our a priori hypotheses (i/ii) or were shown in our previous work (Herrera et al., 2021) to be modulated by task difficulty (iii). ROIs were selected from the results in the Competing condition because, as expected, this yielded more robust and extensive responses to pitch. Trial-wise activation time series were extracted from each ROI and averaged across all surface nodes in the ROI.

The mediation analysis proceeded in three steps. First, a trial-wise DP was extracted from the HDDM as described in the section Post-Hoc Analysis: Effect of Trial-Wise Difficulty. Next, the effect of trial-wise pitch-restricted drift rate on trial-wise DP was estimated using a hierarchical Bayesian linear model. Finally, the effects of trial-wise DP and trial-wise pitch restricted drift rate on trial-wise fMRI activation were estimated using a hierarchical Bayesian linear model. This was performed separately for each condition (Competing, Unison) and ROI. The effects of the predictors were allowed to interact with response category (correct vs. incorrect trials) because we expected the relation between pitch-restricted drift rate and trial-wise DP would be modulated by response category. Thus, the crucial question was whether the interactive effect of response category and pitch-restricted drift rate on fMRI activation was mediated by an interactive effect of response category and pitch-restricted drift rate on DP—that is, was there evidence for a moderated mediation effect?

The results of the moderated mediation analysis are presented in Figure 5. We focus first on the effect of pitch-restricted drift rate on DP, which was the same regardless of the ROI to be tested in the mediation analysis. In the Unison condition (Figure 5A), DP tended to increase with increasing pitch-restricted drift rate for incorrect trials, whereas it remained relatively flat (but trended down with increasing pitch-restricted drift rate) for correct trials. As expected, there was also a main effect of response category such that incorrect trials tended to be more difficult than correct trials. In the Competing condition (Figure 5B), the same pattern of effects was observed but the tendency of DP to decrease with increasing pitch-restricted drift rate for correct trials was much more pronounced. This appeared to be driven by an increase in DP for low pitch-restricted drift rates on correct trials, wherein this increase was large enough that DP was essentially equivalent for correct and incorrect trials at the lowest pitch-restricted drift rates. This likely reflects the fact that absence of pitch on correct trials, for which speech tended to be less distorted overall, leads to increased DP because the speech signal is overall more intelligible but the talkers are difficult to segregate, whereas absence of pitch on incorrect trials, for which speech tended to be more distorted, leads to decreased DP because the speech signal is overall less intelligible and the listener tends to guess randomly (decreasing DP) because the talkers are also difficult to segregate. Crucially, the effect of pitch-restricted drift rate on DP interacted with response category for both Unison and Competing, which validates the moderated mediation approach.

The direct effects of pitch-restricted drift rate on fMRI activation—that is, those remaining after accounting for the effects of DP on fMRI activation in the final step of the mediation model—are shown for the three ROIs in Figure 5C–H. We focus first on the Competing condition (D, F, H) because this is the crucial condition relative to our a priori hypotheses regarding AC and left dPM. First, activation in bilateral AC and left dPM tended to increase with increasing pitch-restricted drift rate, regardless of whether the response was correct or incorrect (D, F), consistent with our a priori hypothesis regarding these ROIs. We must exercise extreme caution in interpreting these effects because, indeed, these ROIs were selected based on their significant overall correlation with pitch-restricted drift rate in the brain-behavior-stimulus model. However, the moderated mediation effect, which is unrelated to the brain-behavior-stimulus model, was not significant for AC or left dPM, suggesting that, in the Competing condition, the effect of pitch-restricted drift rate on fMRI activation in these ROIs was not mediated by DP. On the other hand, the direct effect of pitch-restricted drift on fMRI activation in the left middle pFC (H) patterned exactly as the effect of pitch-restricted drift rate on DP (B). Indeed, the moderated mediation effect was significant for pFC (posterior mean = 0.04, 95% CI [0.03, 0.05]; proportion mediated = 0.35). Thus, as expected, the effect of pitch-restricted drift rate on fMRI activation in pFC is strongly mediated by the effect of pitch-restricted drift rate on DP. Notably, there was a main effect of response category on fMRI activation in all three ROIs, but this effect took opposite directions for AC and left dPM (correct > incorrect) versus pFC (incorrect > correct), where the latter again matched the direction of the main effect of pitch-restricted drift rate on DP.

The direct effects of pitch-restricted drift rate on fMRI activation in the Unison condition (C, E, G) were similar overall to the Competing condition, but with some important differences. Like the Competing condition, effects in pFC (G) closely mirrored the effect of pitch-restricted drift rate on DP, whereas this was not the case for AC (C) and left dPM (E). Indeed, the moderated mediation effect was significant only for pFC (posterior mean = 0.01, 95% CI [0.001, 0.02]; proportion mediated = 0.23). Contrary to our a priori hypothesis, fMRI activation again tended to increase with increasing pitch-restricted drift rate regardless of response category, which likely explains why the overall correlation between pitch-restricted drift rate and fMRI activation in AC was significant in the brain-behavior-stimulus model in the Unison condition (Figure 5B, Unison; subject again to the caveat of bias in ROI selection). However, the magnitude of the effect was reduced for correct trials. Consistent with our a priori hypothesis, fMRI activation in left dPM tended to increase with increasing pitch-restricted drift rate only for incorrect trials, which likely explains the absence of a significant overall effect in left DPM in the brain-behavior-stimulus model (Figure 5B, Unison), because the vast majority of Unison trials were correct. Notably, fMRI activation in AC and left dPM again tended to be higher on correct trials (although the effect was drastically reduced for left dPM), whereas the opposite was true for pFC, which again mirrored the direction of the effect of response category on DP.

We hypothesized that listening-related activation in the left dorsal speech-premotor cortex (left dPM) and temporal lobe voice/pitch regions would be driven more reliably by vocal pitch for a speech recognition task that requires extraction of vocal pitch versus one that does not. To test this hypothesis, we compared correlations of behaviorally relevant acoustic speech features with fMRI activation in two multitalker speech recognition tasks—Competing, which required talker segregation using vocal pitch, and Unison, which did not—and with two measures reflecting the relative presence or absence of behaviorally relevant acoustic speech features—overall drift rate, which was allowed to be modulated by vocal pitch and phonetic speech content, and pitch-restricted drift rate, which was allowed to be modulated only by vocal pitch (Figure 3). We predicted that (i) overall drift rate would be correlated with fMRI activation in dPM and classic auditory-speech regions in the Competing and Unison conditions; and (ii) pitch-restricted drift rate would be correlated with fMRI activation in dPM and auditory regions sensitive to pitch and/or voice only in the Competing condition. Hypothesis (i) was confirmed by the brain-behavior-stimulus analysis and Hypothesis (ii) was confirmed for dPM but not AC, where activation in the latter was correlated with pitch-restricted drift rate in the Unison and Competing conditions (Figure 4).

In a recent fMRI study (Venezia et al., 2021), we used Auditory Bubbles to degrade sentences spoken by a single female talker in a quiet background, and listeners were asked to make subjective (yes–no) intelligibility ratings. We found that both left dPM and early auditory regions responded to vocal pitch only within trials rated as unintelligible by the listeners. Crucially, behavioral analysis showed that the intelligibility judgments relied exclusively on phonetic speech content. As such, left dPM and early AC were primarily driven by phonetic speech content as required by the task. We suspect these areas activated to vocal pitch on unintelligible trials because phonetic content was largely absent from the signal, leaving pitch as the only potentially relevant acoustic speech feature within this group of trials. This suggests that pitch-responsive cortical regions including left dPM can operate in multiple modes—an auditory-pitch mode and a phonetic-speech mode. Numerous recent studies support this conclusion, showing responses to vocal pitch (e.g., the F0 contour) and phonetic content (or acoustic features correlated thereof, e.g., the speech envelope) in early auditory regions and left dPM (Hamilton et al., 2021; Berezutskaya et al., 2020; Forseth, Hickok, Rollo, & Tandon, 2020; Venezia, Thurman, Richards, & Hickok, 2019; Brodbeck, Presacco, & Simon, 2018; Dichter et al., 2018).

Here, we hypothesized that left dPM and AC would be pushed into the auditory-pitch mode during the Competing task but not the Unison task. The logic supporting this hypothesis was twofold. First, because the Competing task required extraction of vocal pitch to segregate two talkers, pitch-responsive regions would be activated by pitch regardless of overall speech intelligibility. Second, the Unison task was analogous to the single-talker task in Venezia et al. (2021) but overall speech intelligibility was much higher (∼85% correct performance), leaving very few unintelligible trials on which pitch-sensitive regions might be recruited in the auditory-pitch mode. Our HDDM analysis showed that overall drift rate was more strongly driven by vocal pitch for Competing than Unison (Figure 3). Correspondingly, activation in left dPM was more strongly correlated with overall drift rate for Competing than Unison (Figure 4A). However, this effect could have been driven by a categorical response to intelligible speech as in Venezia et al. (2021). First, overall drift rate is modulated by both vocal pitch and phonetic content (Figure 3B/C). Recall that Auditory Bubbles estimates these relative contributions by stochastically removing (or retaining) components of the speech signal related to vocal pitch and phonetic content (Figures 1/2). Because phonetic content is strongly related to overall signal intelligibility (Venezia, Martin, et al., 2019; Venezia et al., 2016) and makes strong contributions to overall drift rate in both conditions (Figure 3B/C), brain-behavior-stimulus correlations with overall drift rate may reflect a categorical response to intelligible versus unintelligible speech. Second, overall task difficulty was higher for Competing (67% correct) than Unison (∼85% correct). Indeed, overall drift rate was almost entirely positive (i.e., speech was almost always intelligible) in the Unison condition (Figure 3A). Any categorical effects of intelligibility were likely to be larger in the Competing condition, which may explain the more widespread positive correlations with overall drift rate in downstream auditory regions (Competing vs. Unison; Figure 4A) that often respond preferentially to intelligible speech (Stoppelman, Harpaz, & Ben-Shachar, 2013; Okada et al., 2010; Narain et al., 2003).

Therefore, it was crucial to also examine brain-behavior-stimulus correlations with pitch-restricted drift rate, which represents the independent correlation of behaviorally relevant vocal pitch cues with fMRI activation (i.e., regardless of overall intelligibility). In both conditions, pitch-restricted drift rate was positively correlated with activation in bilateral auditory networks including anterolateral Heschl's gyrus and neighboring superior temporal gyrus, consistent with prior work on pitch and voice processing (Hamilton et al., 2021; Pernet et al., 2015; Puschmann, Uppenkamp, Kollmeier, & Thiel, 2010; Patterson, Uppenkamp, Johnsrude, & Griffiths, 2002; Griffiths, Uppenkamp, Johnsrude, Josephs, & Patterson, 2001; Belin, Zatorre, Lafaille, Ahad, & Pike, 2000). Given the relatively small contribution of vocal pitch to drift rate in the Unison condition (Figure 3), as expected given the nature of the task (essentially single-talker) and high overall intelligibility of signal, it was somewhat surprising to see robust correlations with pitch-restricted drift rate in early auditory regions (Figure 4B, Unison). Indeed, this violated our Hypothesis (ii) with respect to AC. The presence of robust correlations with pitch-restricted drift rate, despite low intertrial variation in the measure (Figure 3A, left, red), suggests a more obligatory response to vocal pitch in AC. One possible reason why this was observed here and not by Venezia et al. (2021) is increased power to detect correlations with vocal pitch afforded by the distillation of pitch-related cues to a single drift-rate metric via the HDDM, essentially a form of dimension reduction in the stimulus space, whereas Venezia et al. (2021) employed a high-dimensional multivariate analysis of acoustic features across the entire speech MPS.

In line with our Hypothesis (ii), in left dPM, we observed a task-related modulation of the correlation with pitch-restricted drift rate, such that activation was positively correlated with pitch-restricted drift rate in the Competing condition but not in the Unison condition (Figure 4B; Competing BF₁₀ = 254, Unison BF₁₀ = 0.22). In the Competing condition, this was accompanied by the appearance of negative correlations with pitch-restricted drift rate in a bilateral frontoparietal network (Figure 4B). Given the proposed role for this network in effortful listening (Herrera et al., 2021), this suggests that absence of vocal pitch significantly modulated the difficulty of the Competing task. Together, these findings lend support to the conclusions that (i) the Competing task pushed left dPM into an auditory-pitch mode and (ii) left dPM may play a task-modulated role in pitch extraction for multitalker speech recognition.

Regarding conclusion (i), there are at least two counterarguments. First, left dPM may contain functionally heterogeneous subregions. Figure 6 plots left dPM as defined in our previous study (Venezia et al., 2021) with left dPM as defined presently (Figure 4B, Competing). We also plot a recent estimate (peak coordinate) of the gyral component of the dorsal laryngeal motor cortex (Belyk et al., 2021). As is clear, the dPM regions are largely nonoverlapping, with the previous-study region on the crown of the precentral gyrus and the present-study region on the adjacent posterior wall of the precentral gyrus abutting the gyral laryngeal motor cortex. Interestingly, we noted in our previous study that pitch-related responses tended to increase moving posteriorly within left dPM. Thus, it possible that the present study has identified a pitch-selective subregion of left dPM. In Neurosynth (Yarkoni, Poldrack, Nichols, Van Essen, & Wager, 2011; https://neurosynth.org/ accessed on 10/10/2021), the center of mass of left dPM as defined in the present study ([−50, −4, 44]) was significantly associated with “pitch” (z = 4.16), “vocal” (z = 5.39), and “auditory” (z = 6.57), whereas none of these terms was associated with left dPM as defined by Venezia et al. (2021; [−52, 0, 38]). Second, and related, work by Hamilton and colleagues suggests that representations of different acoustic-phonetic speech features are spatially comingled within local patches of auditory responsive cortex, including motor regions (Hamilton et al., 2021; Hamilton, Edwards, & Chang, 2018). Thus, the “multiple modes” of processing in left dPM may reflect subpopulations of neurons tuned to different speech features.

Regarding conclusion (ii), it is noteworthy that we generally observed positive correlations with speech recognition performance (drift rate) in auditory and lower-level (pre-)motor regions and negative correlations with performance in higher-level sensorimotor regions in inferior frontal and parietal cortices (Figure 4). This suggests the response in lower-level motor regions including left dPM is not purely compensatory, because these regions tend to activate more when speech recognition performance is better. However, because this study is correlational, we should not immediately conclude that motor regions play a role in speech recognition. Specifically, because we model trial-by-trial fluctuations in drift rate purely as a function of trial-by-trial fluctuations in the speech acoustics, those brain regions driven purely by speech acoustics may appear to be correlated with performance and vice versa. We have previously argued that motor activations during listening to speech may reflect a sensorimotor “resonance” between auditory regions that code acoustic-phonetic speech features and motor regions that use those features as “targets” to guide speech production (Venezia & Hickok, 2009). This argument cannot be ruled out here in relation to pitch-driven activations in left dPM. The fact that pitch-driven activations in early auditory regions are more obligatory while such activations in left dPM are more task sensitive supports the notion that core encoding of vocal pitch occurs in the AC, but leaves the door open to the possibility of top–down contributions to speech recognition by left dPM in certain listening scenarios such as in multitalker speech recognition (Stokes, Venezia, & Hickok, 2019). However, we also observed task-modulated activation by vocal pitch in the bilateral calcarine sulcus (Figure 4B), which we have previously suggested to reflect cross-modal processing of emotional or socially significant stimuli (Venezia et al., 2021) such as vocal pitch. Thus, increased attention to vocal pitch—driven by the demands of the speech recognition task but unrelated to speech recognition per se—may modulate pitch responses in some brain regions.

It is also important to rule out the possibility that task-related modulation (Competing vs. Unison) of the response to vocal pitch in left dPM reflects merely the fact that the Competing task was more difficult than the Unison task, whereas such difficulty was, by definition, associated with pitch because of the talker segregation demands of the Competing task. That is, we expect the relative presence or absence of vocal pitch across trials in the Competing task to be associated with fluctuations in difficulty across trials, such that fluctuations in difficulty may be driving the correlation with pitch-restricted drift rate in left dPM. To address this, we ran a post hoc mediation analysis (Figure 5) testing whether the correlation with pitch-restricted drift rate in left dPM was mediated by a trial-wise DP, reflecting essentially the decision-related component of RT (higher RT = higher DP). For comparison, we included bilateral AC, for which we expected no such mediation by DP, and left middle pFC, for which we strongly expected mediation by DP, in the mediation analysis. Mediation effects were split by response type (correct, incorrect) because we expected the direction of the relation between pitch-restricted drift rate and DP to be modulated by response type, which was indeed confirmed by the mediation analysis (Figure 5A/B).

Focusing on the crucial Competing condition, DP tended to increase with increasing pitch-restricted drift rate on incorrect trials and decrease with increasing pitch-restricted drift rate on correct trials (Figure 5B). DP was also greater overall for incorrect trials than correct trials. The effects of pitch-restricted drift rate on activation in pFC followed a nearly identical profile, suggesting mediation of the pitch-restricted drift rate effect by DP. Indeed, per our expectation, the mediation effect was significant for pFC. On the contrary, no such mediation was observed for AC or left dPM (Figure 5D/F). In these regions, activation tended to increase with increasing pitch-restricted drift rate regardless of whether the response was ultimately correct or incorrect. That is, AC and left dPM were activated by behaviorally relevant vocal pitch cues whether the presence of those cues made the task more effortful (incorrect trials) or less effortful (correct trials). Thus, the mediation effect of DP was not significant in AC nor left dPM. Interestingly, AC, but not left dPM, showed the same pattern in the Unison condition (Figure 5C/E), in agreement with the modulation of the pitch-restricted drift rate effect by task in left dPM but not AC in the main brain-behavior-stimulus analysis (Figure 4).

Finally, it should be noted that we cannot rule out unequivocally that pitch-related activations in AC and left dPM were related in some way to intelligibility rather than pitch per se. As noted, the presence of vocal pitch tended to improve performance in the Unison and Competing tasks (although dramatically more so in the Competing task; Figure 3), which suggests that pitch may have contributed to speech intelligibility. Such a contribution is likely in the Competing condition where the presence of pitch would aid in talker segregation and thus increase the availability of potentially intelligible speech cues. Consistent with this explanation, activation in AC and left dPM tended to be highest on correct trials for which pitch cues were present in the signal (high pitch-restricted drift rates; Figure 5D/F). However, this result is essentially “baked in” to the study design because (i) AC and left dPM tend to activate more to intelligible speech and (ii) the presence of vocal pitch increases speech intelligibility, at least when operationalized in terms of drift rate for the Competing task. That is, one would expect a region sensitive to both phonetic (intelligible) speech content and vocal pitch to show exactly the pattern of results observed in AC and left dPM. One source of evidence supporting a specific role for AC and left dPM in processing pitch is the fact that correlations with pitch-restricted drift rates did not extend into regions in the lateral superior temporal lobe that classically respond selectively to intelligible speech. Thus, we suggest the most likely explanation of the data is that AC and left dPM play a role in processing vocal pitch, where, at least during perception, the contribution of AC is more obligatory and the role of left dPM is more modulatory. Although it is possible this modulatory contribution reflects a recoding of pitch in laryngeal articulatory terms, it is not clear why such a transformation would be beneficial given suggestions of perceptual invariance for pitch (McPherson & McDermott, 2022) and a linear acoustic-to-articulatory mapping for pitch (Parrell et al., 2019), but it may be relevant that there is a perceptual “voice disadvantage” (i.e., worse performance for vocal compared to nonvocal stimuli) for even simple discrimination of pitch (Gao & Oxenham, 2022). A possible role for left dPM during speech recognition with multiple talkers may be to regulate the extent to which voice- or speech-related representations in AC are facilitated relative to fine-grained pitch cues.

This material is the result of work supported with resources from and the use of facilities at the VALLHCS, Loma Linda, CA. The contents do not represent the views of the U.S. Department of Veterans Affairs or the U.S. government.

Reprint requests should be sent to Jonathan H. Venezia, VALLHCS, Loma Linda, CA 92357; Loma Linda University School of Medicine, Loma Linda, CA 92350, or via e-mail: [email protected].

All code used for stimulus generation/presentation and fMRI preprocessing, as well as a full set of example stimuli, can be found at https://osf.io/ftmpa/. Code for the HDDM, Bayesian brain-behavior-stimulus, and moderated mediation analyses can be found at https://osf.io/ne2x4/. Results of hierarchical Bayesian fMRI analyses in cortical surface format (text files that can be mapped to the cortical surface using SUMA) as well as the results of HDDM and moderated mediation analyses (R data files) can be found at https://osf.io/ne2x4/. Code for generating the final plots is available in the repository where applicable. In accordance with VA policy, interested parties may obtain the raw MRI and behavioral data only once a Data Use Agreement has been finalized between the requesting institution and the VALLHCS, U.S. Department of Veterans Affairs.

Jonathan H. Venezia: Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Resources; Software; Supervision; Validation; Visualization; Writing–Original draft; Writing–Review & editing. Christian Herrera: Data curation; Investigation; Methodology; Project administration; Validation; Writing–Review & editing. Nicole Whittle: Data curation; Investigation; Project administration; Resources; Writing–Review & editing. Marjorie R. Leek: Conceptualization; Supervision; Writing–Review & editing. Samuel Barnes: Data curation; Methodology; Resources; Writing–Review & editing. Barbara Holshouser: Resources; Supervision; Writing–Review & editing. Alex Yi: Methodology; Resources; Writing–Review & editing.

This work was supported by the U.S. Department of Veterans Affairs, Veterans Health Administration, Rehabilitation Research & Development Service Award IK2RX002702 to J. H. V.

Retrospective analysis of the citations in every article published in this journal from 2010 to 2021 reveals a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .407, W(oman)/M = .32, M/W = .115, and W/W = .159, the comparable proportions for the articles that these authorship teams cited were M/M = .549, W/M = .257, M/W = .109, and W/W = .085 (Postle and Fulvio, JoCN, 34:1, pp. 1–3). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance. The authors of this article report its proportions of citations by gender category to be as follows: M/M = .632; W/M = .162; M/W = .132; W/W = .074.