Abstract

Recent developments in neurotechnology raise the possibility of directly reading out—or sending input into—perceptual awareness. Using Marshall McLuhan's statement “the ‘content’ of any medium is always another medium” as a starting point, the author explores the potential for neural decoding and brain-computer interfaces to support a medium of awareness. This article intends to open a set of questions that reconsider ongoing issues in phenomenology and the arts. If art addresses the human condition, then it is arguably essential for art to address our growing integration with external—and increasingly internal—technology.

Transforming Art and Neuroscience

As an artist, I have always had a particular interest in the concept of transformation. I prefer exploring media that can lead to the fantastic, the absurd, the contradictory and the impossible. Artistic interventions—transformations through technology—hold a particular appeal for me. And I learned that there are fewer impossible transformations than I had thought. For me, a useful way to understand such transformations comes from Marshall McLuhan's Understanding Media. He posits that “the ‘content’ of any medium is always another medium” [1]. He uses the example of verbal communication: “The content of writing is speech, just as the written word is the content of print, and print is the content of the telegraph” [2]. This is the possibility of transformation—in scope, in dimensionality, of something that is measurable in the external world and has the power to transform society.

What then happens when the content of media is the phenomenological? What if media technology, the external, a matter of the world, impinges—not on our perceptual apparatus—but rather upon the scale or scope of the container of perception, our consciousness? To put this into the terminology of McLuhan, if the content of any medium is a new medium, then what happens if the medium is awareness itself, for which all experience whatsoever takes the form of content? McLuhan follows up his example of verbal communication with this question: “If it is asked, ‘What is the content of speech?,’ it is necessary to say, ‘It is an actual process of thought’ ” [3]. Thought itself, internal awareness, can be considered a form of media.

This idea of internal awareness as a form of media is not just a theoretical possibility anymore; neurotechnologies have taken aim at externalizing our internal awareness. In the past decade, scientists have used machine learning to “decode” internal perceptual states, such as constructing visual perception from neuroimaging data. The pattern of measurements of brain activity can be used to accurately predict what a subject is perceiving. This is an established technique that has been successfully applied to statically presented stimuli: color, shape or even categories of images [4]. However, neural decoding has recently advanced substantially beyond simple statically presented stimuli, raising questions about how this technology might be used to explore our inner world of experience.

As one example, a team led by Shinji Nishimoto at University of California, Berkeley, published a breakthrough paper, “Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies” [5]. For the first time, functional magnetic resonance imaging (fMRI) decoding techniques could be applied—not just to static images—but to visual experience through time, to the “movie” of one's internal visual experience. Long methodological story short, the results of the experiment resemble a hazy, blurry but artistically compelling retelling of an internal experience [6] (Fig. 1). The researchers reconstructed videos that subjects viewed in real time but leave open the possibility (if not the probability) that this technique could be used to reconstruct visual memories or purely internal visual imagery.

A still image from Nishimoto et al.'s reconstruction.

Fig. 1.
A still image from Nishimoto et al.'s reconstruction.
Fig. 1.
A still image from Nishimoto et al.'s reconstruction.

This is surely a transformation—and a complex one. To take only the most pertinent parts: Video clips (taken from YouTube) are input by the human eye, the related brain activity is extracted using MRI, the MRI-measured patterns of brain activity are statistically compared to the patterns from previously viewed videos and lastly, the top few best-matching videos are averaged together in a digitally reconstructed best guess at the internal visual experience of the viewer. An internal experience, something invisible to the world, which is only accessible to the internally experiencing self, is made into a video that others can see. The content of our awareness is a new medium, a breathing of life into the inanimate.

I would like to suggest that Nishimoto et al.'s work was born of an artistic impulse. The output of the work, that hazy retelling of watching a video, is half data, half artistic choices about how to reconstruct them. In fact, it may be that the artistic impulse and the scientific impulse have merged in this work. After all, this “reconstruction” attempts to manifest our internal awareness—our imagination—into media, arguably the domain of the artist.

And this may be a sign of things to come. Looking at the output of the “video” decoding, one is struck by its ephemeral, contingent and evocative nature—the kinds of interpretative and intuitive evocations of space and time for which artists often aim. Compare the output of this experiment with another experimental use of decoding algorithms—one that was presented at the Whitney Museum of American Art in 2017. Terence Broad and Mick Grierson's Blade Runner—Autoencoded [7] (Fig. 2) used a decoding algorithm to learn, frame by frame, the movie Blade Runner and then used the same algorithm iteratively to make a best guess of the movie based on this learning, in something like a feedback loop that exposed the “thinking” of the algorithm in a way that evoked the concept of memory itself, in all its hazy glory, and asked us to consider the distinction between the machine's “learning and memory” and our own [8].

Blade Runner—Autoencoded, still from the encoding.

Fig. 2.
Blade Runner—Autoencoded, still from the encoding.
Fig. 2.
Blade Runner—Autoencoded, still from the encoding.

However, unlike Blade Runner—Autoencoded, Nishimoto et al.'s work explores direct human awareness rather than the algorithm itself. Consider the comparisons between Nishimoto et al.'s temporal reconstruction of brain activity and the temporal reconstruction of the external world in film: One can think of an MRI machine, broadly speaking, like a camera. The specific details of measurement and mechanics are different, but like the camera, the MRI machine relies on indexical frames—in the parlance of imaging science: frames, slices or volumes discretely measured at specific time intervals. Like the camera, the MRI machine makes visible what the eye cannot on its own perceive and, like the camera, the MRI machine can also be thought of as a technological extension of our sense apparatus.

But unlike the camera, in neural decoding from the MRI scanner, time is no longer discrete or continuous; it is an extratemporal reconstruction of purely abstracted data. This new medium is not about the role of the index against the continuum of true experience. It is a question of internal experience against its abstracted mirror image on a computer screen.

The Contestable and Contingent Future of Perception as Media

This fMRI decoding reconstruction seems unlikely to ever evolve into any form of practical nonscientific use. It requires multimillion-dollar imaging equipment, a team of highly specialized scientists, a subject willing to lie quietly in an MRI machine for hours and months of work to produce results that are imperfect at best. As a practical technology, it is lacking. But artists, on the other hand, need no such compelling motivation for their imaginations. As Stelarc noted, “Artists are in a position to explore contestable, contingent futures which may never be” [9].

Are there possible future techniques for exploring perception as a medium? Researchers are indeed developing noninvasive methods for “reading out” and “inputting” information into the human brain. These technologies are often referred to as brain-computer interfaces (BCI) in that they connect the workings of the brain to external input and output from digital media. While still in their infancy of low-resolution and limited applicability, these technologies are becoming more sophisticated all the time. They are—properly—being developed for clinical cases: paralysis, epilepsy and degenerative motor conditions. But let us, as artists, explore them as contingent and contestable future technologies that may never be.

One such speculative technique is neural dust, millimeter-scale neural transmitter/receivers that can be implanted into the brain [10] (Fig. 3). The technology relies on ultrasound transmission both to communicate and to provide energy for these wireless, battery-less systems. Given the small size of each individual neural dust unit, a mass of “dust” particles could be applied in large quantities to achieve signal covering multiple cortical areas in a probabilistic fashion. The methods rely on a one-time permanent implant of neural dust in connection with ultrasound relays nearby outside the cortex and a remote computer interface. Theoretically, given current technology, neural dust “motes” could be made as small as tens of microns, within the scale of single-neuron measurements [11]. If neural dust could be developed at this scale, the resolution and functionality of these nanodevice “swarms” could be extremely powerful—substantially greater than what Nishimoto used in experiments. An individual with implanted dust could wirelessly communicate with a remote computer system indefinitely. The computer system could conceivably have access to a very large set of neural data input and output—something akin to “big data,” although perhaps better described here as lots of very small data.

Neural dust “mote.”

Fig. 3.
Neural dust “mote.”
Fig. 3.
Neural dust “mote.”

While neural dust gets us close to the idea of totally noninvasive, wireless brain-machine interfaces that could be used as instruments for our new phenomenological medium, it does require a one-time implantation of “dust.” This kind of invasive treatment is sensible when it comes to clinical applications. However, nanotechnology may provide an alternative technology capable of providing true noninvasive wireless communication suitable for everyday use: magnetic nanoparticles (MNPs) [12,13] (Fig. 4). MNPs, biocompatible magnetically sensitive nanoparticles, can be injected and lodged in the cortex to provide a site of activation. MNPs operate through the process of hysteresis: Under the influence of alternating magnetic fields, MNPs literally heat up. When MNPs heat up, it triggers the firing of neurons associated with the MNPs. MNPs can be targeted chemically to particular neural substrates, and different types of MNPs can be activated by applying different frequencies of external alternating magnetic fields in a process known as multiplexing [14]. MNPs have already been shown capable of activating deep brain structures in mice [15].

Magnetic nanoparticle.

Fig. 4.
Magnetic nanoparticle.
Fig. 4.
Magnetic nanoparticle.

Practical applications for nanotechnologies of neural interface may seem like a far-fetched idea, but the Defense Advanced Research Projects Agency (DARPA) has already initiated a program to develop “a safe, portable neural interface system capable of reading from and writing to multiple points in the brain at once” within the scale of four years (starting March 2018) using technologies such as MNP [16].

As an artistic intervention, these new ideas in brain-computer interfaces offer the possibility of entirely novel forms of media. Once improved, they could have tens of times more resolution than fMRI, while at the same time requiring very little infrastructure: no specialized environment, no massive machinery, completely mobile, wireless and self-sufficient. Along with powerful analytic techniques for neural decoding, it is clearly within the realm of possibility, if not probability, that common brain-computer interfaces could become a reality, and with them, the possibility for these technologies to explore the medium of direct awareness.

The Phonautograph of the Mind

Most perceptual neurotechnology research focuses on our visual system—historically the most studied of the perceptual modalities. However, as a sound artist and composer, I am especially interested in how these technologies might play out for a sound practice. Sound is inherently more difficult than vision for this work since the auditory system is simply less well understood, but it is this exact quality that makes it ripe for exploration. By thinking about auditory perception as media we can challenge our understanding of sound as a phenomenological experience. As a technology its potential is simply unknown in its current infancy. Nonetheless, a look into the past might yield useful ways to think about potential futures.

Researchers in the nineteenth century studied sound perception by reading out internal states of the human ear. Like contemporary neurotechnology studies, these earlier studies used scientific devices only intended to understand how sound worked—in this case, by measuring and visualizing sound waves. These explorations were never intended as a sound reproduction technology or a form of media.

Such explorations led to Édouard-Léon Scott de Martinville's 1857 invention of the phonautograph [17], which became the precursor of Thomas Edison's phonograph. The phonautograph worked by the vibration of sound waves onto a diaphragm that held a stylus. The stylus, as it vibrated, etched onto glass plates covered in fine soot, revealing 2D tracings of sound waves (Fig. 5). The resulting traces were among the earliest direct recordings of sound waves ever made.

An 1857 phonautogram by Édouard-Léon Scott.

Fig. 5.
An 1857 phonautogram by Édouard-Léon Scott.
Fig. 5.
An 1857 phonautogram by Édouard-Léon Scott.

In 2009, researchers were able to reverse engineer the sound recorded onto the earliest of Scott's plates [18]. The resulting recovered recording has a fidelity of sound that could generously be described as marginal—full of distortion, warbling and noise that renders the singing on it nearly impossible to decipher.

Over 150 years after Scott's experiments, scientists are exploring internal states of sound perception using a different technology, fMRI [19]. These contemporary studies of our internal experience of sound, like Scott's experiments with a stylus, aim to elucidate auditory perception, and just like Scott's work, they codify the experience of sound and allow for the possibility of playback, for a medium in sound. For example, in 2017, Santoro et al. [20] played short spoken phrases to subjects in an MRI machine and then reconstructed the spectrogram of perceived sound by decoding patterns of brain activity—a general technique termed auditory stimulus reconstruction in the BCI community [21]. Instead of a tracing on soot, we get computer-generated spectrograms (Fig. 6). Nonetheless, both methods, so remote in technological time, share so much—the use of visualization to analyze and define sound. The question remains: What's next, now that we have this technique?

Spectrogram original (left) and reconstructed from fMRI (right). Adapted from Santoro et al. [31]

Fig. 6.
Spectrogram original (left) and reconstructed from fMRI (right). Adapted from Santoro et al. [31]
Fig. 6.
Spectrogram original (left) and reconstructed from fMRI (right). Adapted from Santoro et al. [31]

It is worth listening to the reconstruction of Scott's phonautogram [22] with Santoro et al.'s auditory reconstruction [23]. Which sounds worse? Neither is properly intelligible as its source material unless one knows the latter beforehand. (Scott's phonautogram measures Scott singing a French folk song; the contemporary analysis measures subjects hearing the phrase “Beyond all things.”) They are equally unintelligible—and equally promising in their time of a future of improved technique, of fidelity, of transformation from one dimension to another, leading us to new media unimaginable in the age when they were first formed: The notion of a future medium of perception is as possible now as the notion of a phonograph was over 100 years ago.

This interweaving of scientific and musical instruments, methods and technologies has a long and vital history, from keyboards and computers [24] to sonification using electroencephalogram (EEG) systems [25]. Neurotechnologies for “perception as media” (e.g. MRI, neural dust, MNPs) may become another chapter in this story [26]. What's distinctive about applying neurotechnologies to auditory perception as media is their direct exploration—and potential direct manipulation—of our internal phenomenological world, not our exterior environment.

It is not clear if perception as media will become commonplace. What is certain is that perception as media is possible. As such, these technologies open up questions specific to such a novel form of media. Consider music: What is the fundamental currency of musical experience? On one extreme, by using neurotechnology it might be possible to evoke the emotional and aesthetic feelings one gets from music without any of its content; on the other, it might be possible to evoke the literal sound of audio as one might from any other audio medium. Wherein lies the distinction between these two experiences? Ultimately, this technology asks us to reconsider larger existing conversations about phenomenology [27], specifically of sound [28], and art in relation to neuroscience [29]. In doing so, it raises the possibility of phenomenology as a practical science, of art as scientific phenomenology.

As a first effort in exploring auditory perception as media, consider the possibilities for auditory stimulus reconstruction of music, a blurry line drawn between scientific investigation and externalization of our expressive internal sound experience like Nishimoto et al.'s reconstruction of visual experience. For sound artists and musicians, new technologies always offer opportunities for subversion, for questioning and investigating the meaning of a technology, for turning technology human [30]. What opportunities would arise from such artistic explorations using auditory perception as a form of media? Where does it lead in terms of our theories of media? How does the line between the artist and technologist merge or change?

This article is only a beginning point, a start to a conversation. Neurotechnology has the potential to cross that final barrier between our inner world of awareness and what it means, on the most fundamental level, to create and experience. A universe of questions can then connect these possibilities to the boundaries between scientific exploration and artistic expression and, fundamentally, what it means to be an experiencing creative person engaged with the world.

Acknowledgments

The author thanks Adrian Freed, inventor and scholar, for helpful suggestions and feedback on this manuscript.

References and Notes

1
M.
McLuhan
,
Understanding Media: The Extensions of Man
(
Cambridge, MA
:
MIT Press
,
1994
) p.
8
.
2
McLuhan
[1].
3
McLuhan
[1].
4
J.-D.
Haynes
and
G.
Rees
,
“Decoding Mental States from Brain Activity in Humans,”
Nature Reviews Neuroscience
7
(
2006
) pp.
523
534
.
5
S.
Nishimoto
et al
,
“Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies,”
Current Biology
21
,
No. 19
,
1641
1646
(
2011
).
6
J.
Gallant
,
“Movie reconstruction from human brain activity”
: www.youtube.com/watch?v=nsjDnYxJ0bo.
8
T.
Broad
and
M.
Grierson
,
“Autoencoding Blade Runner: Reconstructing Films with Artificial Neural Networks,”
Leonardo
50
,
No. 4
,
376
383
(
2017
).
9
Stelarc
,
“The Comatose, the Cadaver, and the Chimera: Alternate Anatomical Architectures,”
Art, Technology, and Culture Colloquium lecture, UC Berkeley Center for New Media
(
2
April
2012
).
10
R.M.
Neely
et al
,
“Recent Advances in Neural Dust: Towards a Neural Interface Platform,”
Current Opinion in Neurobiology
50
(
2018
) pp.
64
71
.
11
D.
Seo
et al
,
“Model Validation of Untethered, Ultrasonic Neural Dust Motes for Cortical Recording,”
Journal of Neuroscience Methods
244
(
2015
) pp.
114
122
.
12
R.
Chen
et al
,
“Wireless Magnetothermal Deep Brain Stimulation,”
Science
347
,
No. 6229
,
1477
1480
(
2015
).
13
M.
Roet
et al
,
“Progress in Neuromodulation of the Brain: A Role for Magnetic Nanoparticles?”
Progress in Neurobiology
177
(
2019
) pp.
1
14
.
14
M.G.
Christiansen
et al
,
“Magnetically Multiplexed Heating of Single Domain Nanoparticles,”
Applied Physics Letters
104
,
213103
(
2014
).
15
Chen
et al
[12].
16
“Nonsurgical Neural Interfaces Could Significantly Expand Use of Neurotechnology,”
Defense Advanced Research Projects Agency News & Events website
(
15
March
2018
): www.darpa.mil/news-events/2018-03-16 (
accessed 27 June 2019
).
17
J.
Sterne
,
The Audible Past: Cultural Origins of Sound Reproduction
(
Durham, NC, and London
:
Duke Univ. Press
,
2003
).
18
J.
Rosen
,
“Researchers Play Tune Recorded before Edison,”
The New York Times
(
27
March
2008
).
19
For example,
S.
Hoefle
et al
,
“Identifying Musical Pieces from fMRI Data Using Encoding and Decoding Models,”
Scientific Reports
8
, article number 2266 (
2018
).
20
R.
Santoro
et al
,
“Reconstructing the Spectrotemporal Modulations of Real-Life Sounds from fMRI Response Patterns,”
PNAS
114
,
No. 18
,
4799
4804
(
2017
).
21
H.
Akbari
et al
,
“Towards Reconstructing Intelligible Speech from the Human Auditory Cortex,”
Scientific Reports
9
, article number 874 (
2019
).
23
Listenable here: www.pnas.org/content/114/18/4799/tab-figures-data
(accessed March 28, 2021). At the page bottom, “Audio_S02” plays the stimulus subjects heard followed by its reconstruction from brain activity.
24
J.
Tresch
and
E.I.
Dolan
,
“Toward a New Organology: Instruments of Music and Science,”
Osiris
28
,
No. 1
: Music, Sound, and the Laboratory from 1750–1980, pp.
278
298
(
2013
).
25
B.
Lutters
and
P.J.
Koehler
,
“Brainwaves in Concert: The 20th Century Sonification of the Electroencephalogram,”
Brain
139
,
No. 10
,
2809
2814
(
2016
).
26
As an exercise, try applying perceptual media neurotechnologies to the four “ethics of instruments” categories introduced in Tresch and Dolan [24].
27
For example,
M.
Merleau-Ponty
,
Phenomenology of Perception
, Colin Smith, trans. (
London
:
Routledge & Kegan Paul
,
1965
).
28
For example,
D.
Ihde
,
Listening and Voice: Phenomenologies of Sound
(
Albany, NY
:
SUNY Press
,
1976
).
29
For example,
A.
Noë
,
Strange Tools: Art and Human Nature
(
New York
:
Farrar, Straus and Giroux
,
2015
).
30
C.
Kelly
,
Cracked Media: The Sound of Malfunction
(
Cambridge, MA
:
MIT Press
,
2009
).
31
Santoro
et al
[20].