Real-world environments are typically dynamic, complex, and multisensory in nature and require the support of top–down attention and memory mechanisms for us to be able to drive a car, make a shopping list, or pour a cup of coffee. Fundamental principles of perception and functional brain organization have been established by research utilizing well-controlled but simplified paradigms with basic stimuli. The last 30 years ushered a revolution in computational power, brain mapping, and signal processing techniques. Drawing on those theoretical and methodological advances, over the years, research has departed more and more from traditional, rigorous, and well-understood paradigms to directly investigate cognitive functions and their underlying brain mechanisms in real-world environments. These investigations typically address the role of one or, more recently, multiple attributes of real-world environments. Fundamental assumptions about perception, attention, or brain functional organization have been challenged—by studies adapting the traditional paradigms to emulate, for example, the multisensory nature or varying relevance of stimulation or dynamically changing task demands. Here, we present the state of the field within the emerging heterogeneous domain of real-world neuroscience. To be precise, the aim of this Special Focus is to bring together a variety of the emerging “real-world neuroscientific” approaches. These approaches differ in their principal aims, assumptions, or even definitions of “real-world neuroscience” research. Here, we showcase the commonalities and distinctive features of the different “real-world neuroscience” approaches. To do so, four early-career researchers and the speakers of the Cognitive Neuroscience Society 2017 Meeting symposium under the same title answer questions pertaining to the added value of such approaches in bringing us closer to accurate models of functional brain organization and cognitive functions.
Many fundamental principles of brain organization, such as hierarchical processing or segregation, have been established by research utilizing well-controlled, but simplified, paradigms with basic, artificial stimuli. In contrast, real-world situations, which most experiments intend to model, are typically dynamic, complex, multisensory in nature and, as such, rely heavily on a variety of top–down (attentional and otherwise) mechanisms for us to be able to function effectively in everyday life: drive a car, remember our shopping list, and so forth. Recently, a number of approaches have been developed, largely independently from each other, that aim to bridge traditional neurocognitive paradigms and the demands posed by naturalistic environments. This Special Focus is the result of an invited symposium under the same title chaired by Pawel Matusz at the Annual Meeting of Cognitive Neuroscience Meeting in March 2017 in San Francisco, California. The collection of invited manuscripts represents the state of the field within the emerging heterogeneous domain of real-world neuroscience, showcasing the wide range of approaches developed to better understand how the brain works in naturalistic situations.
The ultimate aim of cognitive neuroscience research is to create accurate models of how information processing occurs in everyday situations and how this processing is orchestrated by the brain. The use of simplified paradigms and stimuli has been an incredibly fruitful way to chart the functional organization of the brain, especially in times when little was known about response properties of neurons in different brain areas (Evans & Whitfield, 1964; Hubel & Wiesel, 1962; Mountcastle, 1957). These early investigations helped establish some of the fundamental principles of how information is processed by the brain. With time, whereas such “classic” studies have continued proving their worth throughout the years, new lines of research have been departing further and further away from the traditional paradigms to pursue more complex research questions: discrimination of socially relevant, highly similar objects (e.g., Haxby et al., 1996), facilitation of the processing of task-relevant stimuli in multistimulus settings (e.g., Moran & Desimone, 1985), or neurocognitive processes gauged by stimuli engaging multiple senses at once instead of a single one (e.g., Stein, Huneycutt, & Meredith, 1988; Meredith & Stein, 1983).
Fundamental assumptions regarding both functional brain organization and cognitive processes have been substantiated by these early neuroscientific studies. However, further important novel insights have been afforded by careful, parametric adaptations of the abovementioned traditional paradigms, which varied task relevance, social value, or the (multi)sensory nature of stimulation. Nowadays, a variety of approaches have emerged, with some departing far from well-controlled experimental paradigms in the pursuit of better understanding of neurocognitive processes in naturalistic settings. A pressing question motivated by studies conducted within these lines of research is: Are we now at the stage where we can safely abandon laboratory-based investigations?
The different approaches falling under the umbrella of “real-world neuroscience” have developed drawing on theoretical advances but also, to differing degrees, on the ongoing advances in signal processing techniques, computational power, and/or brain mapping tools. This fact is important inasmuch as there is a heated debate regarding the extent to which scrutinizing neurocognitive functions with multiple imaging methods and/or sophisticated techniques of signal analysis alone can provide us with any clue as to the underlying mechanisms. Reservations in the neuroscientific community regarding many of the novel, technology-heavy approaches are clearly visible in voices of prominent researchers (e.g., Krakauer, Ghazanfar, Gomez-Marin, MacIver, & Poeppel, 2017; see also a provocative study by Jonas & Kording, 2017).
The aim of this Special Focus is to bring together, contrast, and start creating synergies across the variety of the existing “real-world neuroscientific” approaches. These approaches follow distinct definitions of “real-world neuroscience research” and address different shortcomings of existing, “traditional” neurocognitive research. In the “The Issue” section below, we describe the main findings and innovations reported in the invited articles. In the subsequent “Q&A” section, four early-career researchers and the speakers of the CNS 2017 symposium—Suzanne Dikker, Alex Huth, Pawel Matusz, and Catherine Perrodin—answer questions, such as those pertaining to most exciting insights offered by studying brain/mind functioning in more naturalistic settings, the readiness of the field for a paradigm shift, or future directions of the field as a whole. Their responses aim to showcase both distinctive features of the different “real-world neuroscience” approaches as well as their commonalities.
One such commonality may be conceptualization of neuroscientific investigation as a three-stage cycle akin to the “exploration–confirmation” cycle traditionally proposed for the scientific method (see Figure 1). In this cyclical model and contrasting with real-world research, “classic” neuroscientific research (Figure 1A) offers piecemeal insights into naturalistic perception and action, by testing detailed hypotheses regarding specific aspects of a task or situation, with maximized statistical power (enabled by simplified experimental designs). For this purpose, it manipulates a minimal number of factors in paradigms that isolate the process of interest (process-specific tasks), for example, one type of stimuli or task, with participants tested with nonmobile brain imaging tools (in the case of nonhuman animals, using head-fixed or anesthetized subjects). The highly detailed hypotheses generated and falsified by these approaches have been critical for establishing many fundamental principles orchestrating neurocognitive functions underlying perception and action.
At an intermediate point on the spectrum ranging from approaches with maximal parametric control to those with maximal behavioral relevance, real-world-like laboratory approaches (Figure 1B) test models of neurocognitive functions in settings resembling everyday situations. Specifically, they constrain the range of potential processes present during the experiment by also using process-specific tasks, but tasks are adapted so that the “noise” characterizing real-world situations is still present, albeit in a reduced, that is, more controlled, fashion. That is, such real-world-like approaches emulate typically two or multiple important known variabilities present in natural situations, that is, those in stimulus task relevance, complexity, familiarity, and/or format (visual, auditory, tactile, and their multisensory pairings); those in task demands; and/or those in capabilities and experience of the observers (see the Q&A section for more details). Here, hypothesis-driven analyses help to falsify predictions regarding involvement of specific neural mechanisms in given naturalistic functions, whereas data-driven approaches help reveal new insights into the governing mechanisms and/or moderating factors. The presence and/or importance of these factors in naturalistic environments is then verified in studies conducted in fully naturalistic real-world studies (Figure 1C), which in turn generate new questions. These can be then either examined individually in traditional neuroscientific studies (Figure 1A) and/or tested more holistically in naturalistic studies, in settings more closely approximating the real world (Figure 1B). In this model, research conducted in veridical real-world situations (e.g., EEG studies in public places, a large group of mice interacting freely in a natural ecosystem) and/or with unconstrained natural stimuli (e.g., genuine films or narrated stories; Figure 1C) tests the extent to which laboratory-generated hypotheses and models of naturalistic perception and (social) action generalize outside traditional laboratory investigations. As these real-world approaches often heavily rely on data-driven analyses, they have a particular potential to reveal novel factors and mechanisms orchestrating cognition and behavior in the real world.
“REAL-WORLD NEUROSCIENCE” STATE OF THE FIELD: THE ISSUE
The articles included in this Special Focus represent—in a necessarily limited fashion—the wide range of exciting approaches and ideas within the area of real-world neuroscientific investigations. Some of these studies provide novel insights into the role of most stimulus- and context-related factors in influencing object detection and discrimination as well as the associated functional brain organization in naturalistic environments. Murray, Thelen, Ionta, and Wallace (2019) report that moment-to-moment detection of dynamic and stationary stimuli is less variable in multisensory than unisensory contexts. They used reference-independent analyses of global features of the electrical scalp field (electrical neuroimaging; e.g., Murray, Brunet, & Michel, 2008) to reveal that this performance improvement is related to distinct patterns of functional connectivity between nodes of a distributed occipital–parietal–frontal network. In turn, purely visual interactive processes across real-world scenes and objects are scrutinized in a multivariate pattern analysis fMRI study from Brandman and Peelen (2019). The authors show that indoor and outdoor scenes are both better recognized when accompanied by an intact object. Notably, the benefits from these interactions across categorical visual information involved activity solely within the left, but not right, parahippocampal place and occipital place areas, suggesting separation between contextual and featural scene representation across the two hemisphere. This study is complemented by a contribution from Matusz, Turoman, Tivadar, Retsa, and Murray (2019) who highlight how traditional visual selective-attention paradigms can be adapted to better understand the brain mechanisms underlying the attentional selection of visual objects controlled by top–down multisensory task-determined templates. The N2pc component, a widely accepted ERP marker of attentional object selection, triggered by visual stimuli was attenuated across purely visual versus multisensory search tasks. Electrical neuroimaging analyses (see above) revealed that, in contexts of partial match with the multisensory template, the visually induced N2pc can be generated by a different brain network rather than by gain control modulations of activity of the same brain network (as typically suggested for N2pc). Together, these findings showcase the improved neural processing efficiency as a hallmark of information processing in naturalistic environments, where integrative processes (both multisensory and unisensory) are common; they also highlight the importance of top–down, semantic, and task-dependent processes that likely simultaneously influence information processing (i.e., object recognition and attentional selection) in naturalistic settings.
Other studies in this Special Focus showcase the importance of development and individuals' experience in shaping cognitive functions and their brain underpinnings. Wu, Shimi, Solis, and Scerif (2018) demonstrate how the already mentioned N2pc component can be used to study the development of top–down attentional control based on object templates defined by abstract object categories (see Matusz et al., 2019, for use of N2pc to study top–down object templates defined by perceptual features). Specifically, the authors present combined behavioral and ERP results that N2pc can be used to verify whether a given individual groups certain items as belonging into the same abstract category. Building on other well-established ERP correlates of visual attentional control processes, Föcker, Mortazavi, Khoe, Hillyard, and Bavelier (2019) show that extensive experience with action video games is linked to changes in early attentional processes reflected by the anterior N1 component, which suggests that gaming experience equips individuals with superior abilities to direct to and maintain the focus of spatial attention in the attended location(s). These results highlight one of the possible mechanisms whereby an individual's experience, such as that in gaming, improves higher-level cognitive functions, such as visuospatial attention.
Taking neuroscientific investigations into veridical naturalistic environments, Bevilacqua et al. (2019) use portable EEG headsets to record brain activity from a group of high school students and their teacher during their regular classes. They demonstrate how teacher likability and test scores vary as a function of interbrain synchrony (coherence) among the group of students and between students and their teacher. Finally, going fully beyond the canon of cognitive neuroscience research but very much within the aims of this special issue, Altikulaç et al. (2019) shed light on the importance of information available in the media regarding “the adolescent brain” for shaping cognitive and emotional control processes in adolescents. The authors provide novel data pointing to the wide-ranging impact of framing such neurocognitive findings by the media—in a positive (focus on creativity, flexibility, etc.) versus negative fashion (focus on increased risk taking, low capacity to plan, etc.)—on adolescents' behaviors ranging from risk taking to response to failure.
“REAL-WORLD NEUROSCIENCE” STATE OF THE FIELD: Q&A
What does real-world neuroscience mean to you?
Alexander Huth (A. H.): I think it is difficult to answer this question without some historical perspective. Throughout the 20th century, psychologists sought to make their field more scientific and quantitative by constructing elaborate, tightly controlled laboratory experiments. Scientifically, this was a good move! But this reductionism also has a clear downside. That is, restricting one's scope of inquiry to tightly controlled experiments makes it difficult or impossible to study most human behavior. The modern alternative to this 20th century mindset is to perform less-controlled experiments using ethologically relevant, natural stimuli. I believe that this type of experiment can go much further at probing the neurobiological bases of complex behaviors like language and social interaction. And I believe that this type of experiment has only become possible recently, thanks to advances in computational power, data acquisition, and data storage. Using these factors, we have less need to control for confounding factors (like our 20th century forebears) because we can model them computationally. This revolution is what I consider real-world neuroscience.
Suzanne Dikker (S. D.): My collaborators and I take the notion of “real-world neuroscience” quite literally, namely, conducting neuroscience outside of laboratory context, in the real world. We do not, however, predict that real-world neuroscience will ever render laboratory-based research obsolete. My answer to the question posed in the Introduction, “Are we now at the stage where we can safely abandon laboratory-based investigations?”, is an emphatic and decisive “no.” Instead, we think of real-world research as a complementary approach that can inform, enrich, and inspire laboratory research, and vice versa, as illustrated in Figure 1. The advance of real-world neuroscience research enables researchers to test the long-standing assumption that the laboratory serves as a valid model for real-world human behavior, to test hypotheses that cannot be addressed in the absence of face-to-face interaction, to address hardware challenges that can help advance technological innovation, and to gather data that can inspire and inform subsequent laboratory experimentation.
Pawel Matusz (P. M.): In my opinion, the two approaches particularly useful for clarifying the neurocognitive architecture supporting object recognition and social interactions in everyday situations are naturalistic laboratory studies (Figure 1B) and studies conducted in veridical natural situations (Figure 1C). “Classic” approaches (Figure 1A)—with their focus on developing and testing detailed hypotheses about processes operating in specific contexts—have clarified the relative importance and the neural mechanisms governing space-, feature-, and object-related top–down goal-driven processes controlling our attention to visual and auditory stimuli. However, many cognitive functions and behaviors are likely emergent; that is, they cannot be understood by analyzing the underlying neural circuits alone (bird flocking is a great example here). Indeed, the neurocognitive architecture governing attentional control might be dramatically different while performing even the most trivial everyday actions. For example, how we are attending while watching a movie might be fundamentally organized by the emotional value of the movie events, what our viewing companions are paying attention to, and/or the strength of our habit of constantly switching attention between the movie, chatting/social media, and eating or taking out the washing to dry. Fully naturalistic studies (Figure 1C) can help verify the relative importance of such everyday demands on our information processing as well as reveal others. In turn, naturalistic laboratory experiments (Figure 1B) can falsify hypotheses and models regarding the neurocognitive underpinnings of attending selectively in such situations by systematically manipulating the spatial location, emotional value, sensory modality, emotional value, and social cues related to the targets and distractors. Research has primarily focused on manipulating systematically one, perhaps two of these demands. More recently, approaches adapting traditional paradigms to systematically manipulate stimulus task relevance, task demands, and/or skills of observers in multisensory settings have provided important novel insights into the extant models of brain organization, perception, language, or cognitive development (Parise & Ernst, 2017; Matusz, Broadbent, et al., 2015; Alsius & Munhall, 2013; Matusz & Eimer, 2011, 2013; Reich, Maidenbaum, & Amedi, 2012; Cappe, Thut, Romei, & Murray, 2010; Lewkowicz & Ghazanfar, 2009; Iordanescu, Guzman-Martinez, Grabowecky, & Suzuki, 2008; Laurienti, Burdette, Maldjian, & Wallace, 2006; Alsius, Navarra, Campbell, & Soto-Faraco, 2005). Thus, neuroscientific studies that test hypotheses and models by combining process-specific tasks together with the paradigm adaptations that emulate the demands of natural environments and new technologies could provide one useful way of falsifying models of brain mechanisms orchestrating real-world cognition and behavior (e.g., filial imprinting; Vallortigara, Regolin, & Marconato, 2005). A particularly promising avenue for combining unconstrained behavior and well-controlled contexts might be the use of virtual or augmented reality setups (e.g., Vedamurthy et al., 2016; Murray, Matusz, & Amedi, 2015; Rizzo et al., 2000).
Catherine Perrodin (C. P.): To me, “real-world neuroscience” means adopting a neuroethological approach to the study of brain function—in the laboratory; that is, investigating neuronal mechanisms in awake behaving animals, in the context of ethologically relevant situations and/or using complex natural stimuli. Experimental settings that preserve at least the key environmental features a given species' brain has evolved to use allow tapping into an animal's natural behavior. Such a design is in turn instrumental in revealing the neuronal substrates of ecologically valid perception and action. The challenge lies in managing a balance between the demands of a controlled, parametric environment necessary for the neuronal level interrogation of brain circuits and the degree of behavioral relevance of a laboratory-based approximation of the real world (e.g., Juavinett, Erlich, & Churchland, 2018).
What should people know about current paradigms and their limits, and how does your work fit into this “new future”?
A. H.: The fundamental issue is that the brain is not a linear system. If the brain were linear, then we could reasonably expect that results obtained from simplified, controlled stimuli would generalize to natural stimuli. Let's suppose we record V1 neuron responses to every possible image that has one single active pixel. If the brain were linear, these responses should predict how the neurons respond to real images. However, already in V1, some responses are nonlinear, as they depend on motion (i.e., complex cells). Outside V1, neuron responses to single pixels cannot explain the activity of neurons that respond specifically to faces (Chang & Tsao, 2017; Kanwisher, McDermott, & Chun, 1997). Natural stimuli have been demonstrated to be advantageous here (e.g., David, Vinje, & Gallant, 2004).
Furthermore, in many fields, there is little effort to test whether results from controlled studies generalize to real-world situations. Every experimentalist should ask themselves: Does the result of this experiment imply anything about cognition/behavior of this organism in the real world? Findings from controlled studies are naturally limited to the hypotheses specified in the experimental design. Studying how the brain responds to natural, ethological stimuli solves these problems. However, it creates other problems: Many things are correlated in natural stimuli, so hypotheses can be difficult to distinguish. This necessitates collecting more data and using a more careful statistical methodology.
S. D.: To date, the study of the human mind/brain through behavioral, neurobiological, and computational techniques has relied on one very fundamental assumption: Laboratory-based research provides foundational insights into how our brains process information on an everyday basis. However, we rarely test this assumption by conducting studies in the actual real world. Some research questions, such as those involving face-to-face interaction, are virtually impossible to investigate in the laboratory: The study of real-world social exchanges has even been dubbed the “dark matter of social neuroscience” (Schilbach et al., 2013). How similar are the brain processes of a 20-year-old college student who is pressing buttons during a visual search task while wearing noise-canceling headphones inside an experiment booth and those of a 45-year-old working parent who is late for work and looking for her glasses on a cluttered breakfast table, while her 5-year-old child is pulling on her trousers, her 2-year-old is smearing egg yolk all over the table while dangerously balancing on his high chair, and the woman's husband is trying to discuss the day with her?
At the same time, this example illustrates why neuroscience research has been mostly confined to the laboratory. Naturalistic environments are messy, noisy places in which it is virtually impossible to obtain good experimental control. Also, there are obvious practical limitations: You cannot place cutting-edge lab equipment such as MRI or MEG machines into classrooms (or kitchens). While simpler, lower-grade equipment has become increasingly accessible (e.g., Debener, Minow, Emkes, Gandras, & de Vos, 2012), most of this technology is still limited in several respects (e.g., data quality and lack of localization options). As such, it is unrealistic to expect the same level of data quality and experimental control from real-world neuroscience research as we demand from laboratory experiments, and research questions and expectations should be adapted accordingly.
P. M.: The current initiatives involving collecting data from larger samples and sharing data across different groups are applaudable and certainly can help to make cognitive neuroscience more replicable (Poldrack et al., 2017). However, as I have already mentioned, the mechanisms governing cognitive functions and functional brain organization in natural situations will remain elusive if the employed paradigms continue not to emulate the demands typical for these situations. The existing models are based on research that adapted the early simplistic paradigms to emulate one or another attribute of naturalistic environments, for example, their multistimulus nature or the role of memory in object processing (e.g., Hickey, Di Lollo, & McDonald, 2009; Gobbini & Haxby, 2007; Horwitz, Rumsey, & Donohue, 1998; Folk, Remington, & Johnston, 1992). However, it remains largely unclear whether/how well these unisensory (visual and auditory) models generalize to everyday situations that are profoundly multisensory in nature. It is increasingly recognized that neural representations of objects are inherently multisensory (e.g., Murray, Thelen, et al., 2016; Reich et al., 2012). Yet, influences of multisensory processes on brain and cognitive processes are often involuntary, and due to the nonlinear nature of the underlying integrative mechanisms, these influences cannot be predicted from the unisensory responses alone (reviewed in, e.g., De Meo, Murray, Clarke, & Matusz, 2015; Murray & Wallace, 2012).
Multisensory paradigms, especially those carefully manipulating stimulus formats (unisensory vs. multisensory, to help assess the presence/strength of multisensory processes), task relevance, and/or demands, are more emblematic of object processing in the real world. My collaborators and I have shown that traditional adult selective attention and memory paradigms readily lend themselves to adaptations to multisensory settings. Drawing on the rigor of these paradigms, the well-established behavioral measures of specific processes, and the breadth of associated literature, we have been making progress in bridging the traditional models of perception, attentional control, and learning/memory with research on multisensory processing. As a result, we have clarified the role of multisensory processes as bottom–up and top–down processes controlling object attention and memory and how these influences depend on the individual's age and experiences. Where possible, we have also utilized the rich spatiotemporal resolution afforded by high-density EEG and electrical neuroimaging to refine the existing models of functional organization of object processing and top–down attentional control (e.g., Matusz et al., 2019; Matusz, Thelen, et al., 2015; Thelen, Matusz, & Murray, 2014; Matusz & Eimer, 2011, 2013). We have shown recently that EEG might be particularly useful to image brain activity in veridical real-world environments, especially where task compliance is difficult or impossible (e.g., Maitre et al., 2017; see below).
C. P.: Historically, insights into the neuronal level underpinnings of perception have been provided by studying how cortical neurons in anesthetized animals process and encode sensory features present in simple artificial stimuli. This approach has successfully uncovered fundamental principles governing the functional organization of sensory systems, including the diversity of topographic maps in the visual and auditory cortices (Ohki et al., 2006; Merzenich, Knight, & Roth, 1975). The brain, however, typically functions as a nonlinear system, with complex stimuli processed and perceived holistically. Similarly, it has proved challenging to predict neuronal responses to natural stimuli from those to basic synthetic components in isolation: Activity patterns elicited by faces or communication sounds are more successfully estimated from responses to high-dimensional features directly extracted from the relevant natural stimuli (Chang & Tsao, 2017; Machens, Wehr, & Zador, 2004; Theunissen, Sen, & Doupe, 2000). In addition, traditional laboratory-based task contexts, such as those involving stimuli presented to animals under the influence of anesthetics, using artificial stimulus categories, or imposing arbitrary associations between certain stimuli and a reward, likely engage different mechanisms than when the animal responds to a naturalistic stimulus in an ethologically relevant situation.
The goal of my research is to uncover the neuronal substrates enabling mammalian sensory communication. Specifically, I study how groups of neurons in the brain of the listener encode social information found in complex sound patterns. To probe brain processes involved in analyzing real-world, meaningful social signals, my colleagues and I have, by necessity, moved away from simple auditory and visual stimuli toward conspecific vocalizations and their corresponding dynamic facial expressions. Our work has helped clarify the neuronal representation of communication signals in the temporal lobe of awake behaving nonhuman primates; in an illustration of studying ethologically relevant sensory processing in a laboratory setting, we collected single-unit and oscillatory neuronal responses to conspecific vocalizations and their associated facial expressions (Perrodin, Kayser, Logothetis, & Petkov, 2014, 2015). This type of approach is a key first step toward directly probing how the brain encodes social communication signals at the neuronal level when perceiving other individuals in a naturalistic, multisensory context (Perrodin, Kayser, Abel, Logothetis, & Petkov, 2015).
What do we now understand about the brain via real-world studies that we did not previously?
A. H.: One of the most dominant ideas in the neuroscience of language (and perhaps in the entirety of human neuroscience) is that language processing occurs mainly in the left cerebral hemisphere (in most people). That idea is supported by a wealth of neuropsychological and neuroimaging research, most of which utilized isolated words or other nonnatural language tasks. However, more recent neuroimaging studies that used natural, continuous language (de Heer, Huth, Griffiths, Gallant, & Theunissen, 2017; Huth, de Heer, Griffiths, Theunissen, & Gallant, 2016; Silbert, Honey, Simony, Poeppel, & Hasson, 2014; Regev, Honey, Simony, & Hasson, 2013; Honey, Thompson, Lerner, & Hasson, 2012; Lerner, Honey, Silbert, & Hasson, 2011) seem to suggest that the right hemisphere is as involved in language processing (and production) as the left. Areas in the right hemisphere seem to play an equal (or, at least, substantial) role in the processing of every aspect of language, including phonetics, syntax, and semantics. It is not yet known why these effects are only seen (or are seen more readily) using natural language stimuli, but that is an area for future exploration. This up-ending of the traditional theory of left lateralization for language clearly shows the need for and importance of studying how the human brain responds to natural, ethologically relevant stimuli.
S. D.: I can speak most specifically on conducting neuroscience research in the classroom. As real-world neuroscience “laboratories,” classrooms are highly social environments but they still afford some experimental control: Kids are used to engaging in short sessions of different “conditions” (listening to a lecture, watching a video, or engaging in a group discussion) while sitting still and being focused. This semistructured nature of the classroom makes it possible to explore how social, contextual, and individual variables influence the extent to which the brain activity of a class of students becomes synchronized. In our work specifically (Bevilacqua et al., 2019; Dikker et al., 2017), students' brain-to-brain synchrony during classroom activities was correlated with how much they liked a given class and each other/the teacher. This suggested that neural entrainment/synchrony can stem from different sources: the nature of what we see or hear, how much we like it, how focused we generally are, and, perhaps most interestingly, the social dynamics of the situation we are in. These findings corroborate other evidence suggesting that social interaction matters to how we experience the world, even if what we do after engaging with someone is not immediately social in nature (e.g., watching a movie). Some other findings would have been difficult, if not impossible, to obtain in a laboratory environment. For example, brain activity of pairs of students who had directly interacted with each other before class was more in sync during class, and this brain synchrony was correlated with students' social closeness. In other words, friends' brainwaves were more synchronized but only if they interacted with each other. In another project, this time in collaboration with artists, we managed to record EEG data from a large sample of people from a wide age range of ages and backgrounds, in museums across the world. This allowed us to investigate how the brain supports unconstrained face-to-face interaction, building on laboratory verbal communication studies that have used either more constrained tasks or lack direct face-to-face interaction (Cohen & Parra, 2016; Dikker, Silbert, Hasson, & Zevin, 2014; Sänger, Müller, & Lindenberger, 2012; Dumas, Nadel, Soussignan, Martinerie, & Garnero, 2010; Stephens, Silbert, & Hasson, 2010; see Babiloni & Astolfi, 2014; Hasson, Ghazanfar, Galantucci, Garrod, & Keysers, 2012, for reviews; see Nozawa, Sasaki, Sakaki, Yokoyama, & Kawashima, 2016; Liu et al., 2016; Jiang et al., 2015, for face-to-face verbal communication studies).
P. M.: To paraphrase the provocative title of Asif Ghazanfar and Charles Schroeder's (2006) article, large swaths of the neocortex (and subcortex) have the ability to integrate information stimulating multiple senses. Crossmodal convergence as well as, increasingly, multisensory processes are assessed in terms of their utility in the clinics for improving sensory and cognitive deficits (e.g., Bottari et al., 2018; Vercillo, Tonelli, & Gori, 2017; Tinga et al., 2016; Murray et al., 2015; Striem-Amit, Cohen, Dehaene, & Amedi, 2012; Amedi et al., 2007). I believe that the real potential lies here in scrutinizing the interplay between top–down attentional control, object processes, and brain development/plasticity, as it has been done in the purely visual domain (e.g., Gazzaley & Nobre, 2012; Astle & Scerif, 2011). The recent efforts to integrate the breadth of multisensory research spanning different populations, paradigms, and imaging methods, including those by my collaborators and I, have been quite fruitful.
First, integrative processes can occur at “feedforward” (<100 msec postonset) stages of brain processing, within “sensory-specific” cortices, such as V1 (reviewed in, e.g., Deroy et al., 2016; Murray, Thelen, et al., 2016; De Meo et al., 2015; van Atteveldt, Murray, Thut, & Schroeder, 2014). As a result, co-occurring crossmodal stimuli are often easier to detect, perceive, and/or attend to, compared with unisensory stimuli, irrespective of the observer's goals (e.g., Sarmiento, Matusz, Sanabria, & Murray, 2016; Matusz & Eimer, 2011; Cappe et al., 2010; Giard & Peronnet, 1999). Second, in real-world-like, multistimulus settings, semantic multisensory processes might be especially important for implicit benefits in recognition of and attention allocation to naturalistic objects (e.g., Matusz, Broadbent, et al., 2015; Matusz, Thelen, et al., 2015; Alsius & Munhall, 2013; Iordanescu et al., 2008; Laurienti, Kraft, Maldjian, Burdette, & Wallace, 2004; Murray et al., 2004; reviewed in Matusz, Wallace, & Murray, 2017; Murray, Lewkowicz, Amedi, & Wallace, 2016; ten Oever et al., 2016). Third, as already hinted, multisensory research reveals new insights about attentional control and its functional organization at the level of the brain. Multisensory objects are selected involuntarily, independently of unisensory task demands, with multisensory top–down templates controlling responses in “sensory-specific” cortices (Matusz et al., 2019; Matusz, Broadbent, et al., 2015; Matusz & Eimer, 2011, 2013). Fourth, multisensory research extends some of the exciting findings from the visual domain that point to the central role of object familiarity/expertise in determining the efficacy of top–down attentional control (e.g., Wu, Pruitt, Runkle, Scerif, & Aslin, 2016; Wu et al., 2015), by showing paradoxical, potentially protective effects of age on attentional control in real-world situations (e.g., Matusz, Broadbent, et al., 2015).
What to me seems to hold the most promise in education and clinics—two highly pertinent real-world settings—is identifying robust behavioral measures of cognitive processes of interest and use them to understand changes therein associated with not only age and neurodegeneration but also rehabilitation. Here, easy-to-use and low-cost methods of brain imaging, such as EEG, can be particularly useful in understanding the underlying brain mechanisms.
C. P.: In a series of studies, my colleagues and I targeted neurophysiological recordings to a higher-order voice-sensitive area in the superior temporal lobe of awake behaving rhesus macaques. We identified a population of neurons in the anterior temporal plane characterized by a categorical response preference to conspecific vocal signals over other complex natural sounds. These functionally specialized “voice cells” were highly stimulus selective, and each responded to a different subset of the presented voices (Perrodin, Kayser, Logothetis, & Petkov, 2011). Interestingly, neurons in the voice area were sensitive to specific communicative features in the vocal sounds, such as either call type or caller identity, suggesting a form of functional segregation in the neuronal level encoding of different aspects of vocal processing (Perrodin et al., 2014). To understand the processing of acoustic communication in its frequently multisensory context, we then combined conspecific vocalizations with dynamic facial expressions. This approach revealed that an auditory-only characterization of response properties only provides part of the picture: Almost half of neurons in the voice area and in the superior temporal sulcus showed spiking responses to voices that were modulated by the simultaneous presentation of faces (Perrodin et al., 2014). Visual input influenced auditory responses along the temporal lobe with varying degrees of sensitivity to the congruency in the voice–face pair. In addition, the direction of visual modulation was correlated with the temporal relationship between the onset of the facial expression and the vocal sound (Perrodin, Kayser, Logothetis, et al., 2015).
Together, these findings emphasize how key response properties of many cortical neurons, especially in hierarchically higher-level areas, are best captured when using behaviorally relevant, complex stimuli and cannot be easily predicted using low-level stimuli in isolation or when considering only the dominant sensory modality. More generally, our research also corroborates a growing body of evidence indicating that much of cortical processing is multisensory and that information streams from different senses are not processed in isolation from each other.
What questions do you most want to see answered by real-world studies?
A. H.: Every question about sensation or cognition could benefit from examining the brain under natural circumstances. However, there are certainly some areas where the tools of classical neuroscience are less appropriate, and real-world neuroscience could be of more use. In particular, social interaction (including language!) is one cognitive domain where natural paradigms seem critical for understanding.
S. D.: I have answered this question more elaborately above and below. Some of these include: How does the brain support real-world dynamic social interactions? Is laboratory neuroscience a valid model based on which inferences can be drawn regarding neural mechanisms underlying our everyday interactions with our environment?
P. M.: Neuroscientific research conducted in veridical real world and in naturalistic settings has the potential to address such pressing questions as how the brain copes with noisy environments as well as the circumstances and mechanisms that enable objects/events to be processed efficiently. Here, particularly needed are careful investigations into maturational and experience-based changes in processes controlling attending toward multisensory objects in healthy and atypically developing populations. Paradigms emulating information processing demands characteristic of the real world, combined with robust methods of measuring brain activity (e.g., electrical neuroimaging; Murray et al., 2008), are increasingly shown to verify and enrich the existing models of sensory processing and its interactions with top–down control as they occur in everyday situations. Immersive environments, such as those involving tablet games and/or virtual realities, could be particularly viable to emulate the demands of these situations, including varying task demands and stimulus relevance, with much of the stimulation being multisensory in nature. There is a notable disparity between findings in both healthy and atypical populations with respect to the so-called far transfer, namely, improvements of untrained cognitive functions, across the “cognitive training” regimes and certain types of video games (cf. Merkley, Matusz, & Scerif, 2018; Melby-Lervåg, Redick, & Hulme, 2016; Vedamurthy et al., 2016; Astle, Barnes, Baker, Colclough, & Woolrich, 2015; Franceschini et al., 2013). Notably, immersive environments could address the problem of the constrained nature of studied behaviors that is characteristic of in-lab experiments. There is real potential in creating synergies across advances in neurorehabilitation and technology and those pertaining to the theoretical and methodological achievements of both experimental psychology and cognitive neuroscience. Much of the current work conducted by my collaborators and I focuses on building these synergies.
C. P.: How does our brain enable us to communicate with each other? What are the neuronal mechanisms for extracting, encoding, perceiving, and using social cues from the auditory or multisensory signals emitted by another individual?
Do you think classical lab studies will be a thing of the past? How best to bridge the new methods?
A. H.: A well-controlled study with clear null and alternate hypotheses is always going to be the best way to test a specific hypothesis. The problem is that, except in very few cases, we don't know which hypotheses to test. The space of hypotheses is too big, and we know too little about the system. This is why we need real-world/natural experiments: to show us the general shape of the hypothesis space efficiently, instead of shooting in the dark with controlled experiments.
S. D.: As I already pointed out above, and as is illustrated in Figure 1, we do not believe that laboratory studies are going to be a thing of the past. Instead, we think of real-world research as a complementary approach that can inform, enrich, and inspire laboratory research, and vice versa. The advance of real-world neuroscience research enables researchers to test the long-standing assumption that the laboratory serves as a valid model for real-world human behavior, to test hypotheses that cannot be addressed in the absence of face-to-face interaction, to address hardware challenges that can help advance technological innovation, and to gather data that can inspire and inform subsequent laboratory experimentation.
P. M.: To reiterate, rigorous paradigms emulating information processing demands characteristic of everyday situations, combined with robust brain imaging methods, are particularly well suited in testing models of neurocognitive functions as they occur in the real world. One way in which to combine them with those more unconstrained neuroscientific investigations in veridical naturalistic settings is in a form of the classic “exploration–confirmation” scientific investigation cycle, a version adapted toward real-world neuroscientific investigations I and other symposium's speakers provided here in Figure 1. Overall, I believe that the most promising approaches will be those that will combine the main advantages of the two directions: the use of tested behavioral indices of cognitive processes of interest and weaker constraints on available behaviors.
C. P.: Laboratory-based work is more relevant than ever, considering the democratization of recent technological developments that allow more powerful monitoring and perturbation of neuronal circuits in awake, freely moving animals (e.g., ultralight and/or wireless hardware for recording neuronal ensembles, high-density silicon probes for monitoring hundreds of neurons simultaneously, optogenetic and minimally invasive, molecular genetic tools for manipulating neuronal activity). These techniques can now be combined with natural stimuli in ecologically valid behavioral assays to powerfully answer fundamental questions about how brains process the real world. However, this abundance of novel methods should not occlude the importance of neuroethological approaches to understanding brain function (Krakauer et al., 2017).
What do you want to be the take-home message of the symposium for the public?
A. H.: The time has come for real-world neuroscience. We know how to design natural experiments, and we have the technology to process the data. Perhaps, the most difficult element of the natural experiment is how to interpret the results. That is where I believe we will see the most advancement in coming years.
S. D.: Conducting real-world neuroscience research “in the wild” is possible and can provide rich data sets that help elucidate how the brain supports real-world dynamic social interactions. While we are clearly at the dawn of such efforts, the proofs of concept that we now have are important, if only in light of the fact that the neuroscience research community remains (rightfully) skeptical. As a side note, I want to have mentioned that directly involving the general public in research is mutually beneficial: As researchers, we get access to highly motivated participants, while our subjects become acquainted, in a hands-on and intuitive way, with neuroscience research and the scientific process.
P. M.: The current initiatives involving collecting larger amounts of data certainly can help make cognitive neuroscience more replicable (Poldrack et al., 2017). However, without appropriate paradigms, that is, those emulating the information processing demands characteristic of natural environments, the mechanisms governing cognitive functions and functional brain organization in such settings will remain at best incomplete. Particularly promising, especially for education and clinics, are those approaches to testing models of cognitive functions that carefully chart possible synergies across advances in experimental psychology and cognitive neuroscience and those in technology.
C. P.: Brains look the way they do for a reason—they have been optimized over the course of evolution to best solve the specific set of environmental/social challenges that each species faces in their ecological niche. In order to ultimately understand the brain, we need to complement the existing knowledge, which has often been acquired using highly artificial, simple stimuli and situations, with real-world approaches that study brains in the ethologically relevant situations they are designed to handle. We are now ideally placed to harness recent technological and analytical developments in multidisciplinary research that can embrace the complexity of the real world.
The authors thank Micah Murray for helpful comments on the article. They also thank Nora Turoman, Alex Huth, Diane Quinn (© 2015 Trevor Day School), and Bridgette Archer (in the order of picture appearance, top left to bottom right) for providing images of different brain imaging and mapping methods and testing environments included in Figure 1. P. J. M. received support from Swiss National Science Foundation (grant PZ00P1_174150) as well as from the Pierre Mercier Foundation and the Fondation Asile des Aveugles. S. D.'s research is supported by The Netherlands Organization for Scientific Research Veni program (grant 275-89-018), the National Science Foundation INSPIRE Track 1 (grant 1344285), and NSF ECR-STEM (grant 1661016). C. P. is supported by the Sir Henry Wellcome Postdoctoral Fellowship from the Wellcome Trust (grant 110238/Z/15/Z); and A. G. H., by the Career Award at the Scientific Interface from the Burroughs-Wellcome Foundation.
Reprint requests should be sent to Pawel J. Matusz, Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO Valais), Sierre 3960, Switzerland, or via e-mail: firstname.lastname@example.org.
This paper appeared as part of a Special Focus deriving from a symposium at the 2017 annual meeting of Cognitive Neuroscience Society, entitled, “Real World Neuroscience.”